File transfer configuration#
When uploading, downloading, or copying a file or S3 object, the AWS SDK for Python automatically manages retries and multipart and non-multipart transfers.
The management operations are performed by using reasonable default settings that are well-suited for most scenarios. To handle a special case, the default settings can be configured to meet requirements.
Configuration settings are stored in a
boto3.s3.transfer.TransferConfig object. The object is passed to
a transfer method (
download_file, etc.) in the
The remaining sections demonstrate how to configure various transfer operations
Multipart transfers occur when the file size exceeds the value of the
The following example configures an
upload_file transfer to be multipart
if the file size is larger than the threshold specified in the
import boto3 from boto3.s3.transfer import TransferConfig # Set the desired multipart threshold value (5GB) GB = 1024 ** 3 config = TransferConfig(multipart_threshold=5*GB) # Perform the transfer s3 = boto3.client('s3') s3.upload_file('FILE_NAME', 'BUCKET_NAME', 'OBJECT_NAME', Config=config)
Concurrent transfer operations#
The maximum number of concurrent S3 API transfer operations can be tuned to
adjust for the connection speed. Set the
max_concurrency attribute to
increase or decrease bandwidth usage.
The attribute’s default setting is 10. To reduce bandwidth usage, reduce the value; to increase usage, increase it.
# To consume less downstream bandwidth, decrease the maximum concurrency config = TransferConfig(max_concurrency=5) # Download an S3 object s3 = boto3.client('s3') s3.download_file('BUCKET_NAME', 'OBJECT_NAME', 'FILE_NAME', Config=config)
Transfer operations use threads to implement concurrency. Thread use can be
disabled by setting the
use_threads attribute to
If thread use is disabled, transfer concurrency does not occur. Accordingly,
the value of the
max_concurrency attribute is ignored.
# Disable thread use/transfer concurrency config = TransferConfig(use_threads=False) s3 = boto3.client('s3') s3.download_file('BUCKET_NAME', 'OBJECT_NAME', 'FILE_NAME', Config=config)