CRT Enhancements #285
Merged
CRT Enhancements #285
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is a backport of this pull request: aws/aws-cli#8291
Support file-like objects in CRT transfer manager
For uploads, the CRT S3 client will add CRC32 trailing checksums. For downloads, the CRT S3 client will validate checksums associated to the object when possible.
Turn on checksum validation for CRT S3 transfer manager
The CRT S3 client performs best when there is only one instance of it running on a host. This lock allows an application to signal across processes whether there is another process of the same application using the CRT S3 client and prevent spawning more than one CRT S3 clients running on the system for that application.
Add CRT process lock utility
Instead of defaulting to five gigabits per second when no target througput is configured, s3transfer will use the CRT to determine if there is a recommended target throughput to use. If it is unable, to determine a recommended target throughput, it will default to ten gigabits per second, which is the current throughput default for the CRT S3 client. This behavior was updated to help minimize the amount of configuration required to get the fastest throughputs from the CRT transfer client. The fallback throughput was updated from five to ten gigabits so that the CRT S3 integration is consistent with the underlying CRT S3 client's defaults.
Update throughput target defaults
The two main changes are: * Update the CRT S3 client factory to accept a CRT credential provider instead of a botocore credential provider. Under the hood, the S3 client only accepts a CRT credential provider. So, this provides more flexibility in being able to provide other CRT credenital providers directly instead of being forced to the botocore credential provider interface * Update the botocore to CRT credentials adapter interface to only accept botocore credential objects instead of credential providers. In general, the credentials object is more accessible than the provider; it can be retrieved at the session level and is what is passed into clients. Also, this change avoids a limitation where the load_credentials() method on the credential provider cannot be called more than twice for some configurations (e.g. assume role from profile), which can be an issue if you create both a botocore client and CRT S3 client.
Update interface for providing credentials to CRT
Codecov ReportAttention:
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. Additional details and impacted files@@ Coverage Diff @@
## develop #285 +/- ##
===========================================
- Coverage 85.36% 83.15% -2.21%
===========================================
Files 16 16
Lines 2719 2796 +77
===========================================
+ Hits 2321 2325 +4
- Misses 398 471 +73 ☔ View full report in Codecov by Sentry. |
nateprewitt
force-pushed
the
crt-enhancements
branch
from
403184d to
87445dd
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR merges a collection of recent enhancements to the CRTTransferManager to improve both durability and throughput in the CLI/Boto3 when opted into this feature.