Select your cookie preferences

We use cookies and similar tools to enhance your experience, provide our services, deliver relevant advertising, and make improvements. Approved third parties also use these tools to help us deliver advertising and provide certain site features.

File transfer configuration

When uploading, downloading, or copying a file or S3 object, the AWS SDK for Python automatically manages retries and multipart and non-multipart transfers.

The management operations are performed by using reasonable default settings that are well-suited for most scenarios. To handle a special case, the default settings can be configured to meet requirements.

Configuration settings are stored in a boto3.s3.transfer.TransferConfig object. The object is passed to a transfer method (upload_file, download_file, etc.) in the Config= parameter.

The remaining sections demonstrate how to configure various transfer operations with the TransferConfig object.

Multipart transfers

Multipart transfers occur when the file size exceeds the value of the multipart_threshold attribute.

The following example configures an upload_file transfer to be multipart if the file size is larger than the threshold specified in the TransferConfig object.

import boto3
from boto3.s3.transfer import TransferConfig

# Set the desired multipart threshold value (5GB)
GB = 1024 ** 3
config = TransferConfig(multipart_threshold=5*GB)

# Perform the transfer
s3 = boto3.client('s3')
s3.upload_file('FILE_NAME', 'BUCKET_NAME', 'OBJECT_NAME', Config=config)

Concurrent transfer operations

The maximum number of concurrent S3 API transfer operations can be tuned to adjust for the connection speed. Set the max_concurrency attribute to increase or decrease bandwidth usage.

The attribute's default setting is 10. To reduce bandwidth usage, reduce the value; to increase usage, increase it.

# To consume less downstream bandwidth, decrease the maximum concurrency
config = TransferConfig(max_concurrency=5)

# Download an S3 object
s3 = boto3.client('s3')
s3.download_file('BUCKET_NAME', 'OBJECT_NAME', 'FILE_NAME', Config=config)

Threads

Transfer operations use threads to implement concurrency. Thread use can be disabled by setting the use_threads attribute to False.

If thread use is disabled, transfer concurrency does not occur. Accordingly, the value of the max_concurrency attribute is ignored.

# Disable thread use/transfer concurrency
config = TransferConfig(use_threads=False)

s3 = boto3.client('s3')
s3.download_file('BUCKET_NAME', 'OBJECT_NAME', 'FILE_NAME', Config=config)