- Added
--s3-max-retries
for the s3 client. Default is 4
- Added
--part-size-multiplier
, which allows you to set the size of each uploaded chunk
- Fixed bug when deleting files
- Clean up old threads after tar'ing is complete
- Close s3 connections after tar'ing is complete
- Option to preserve folder structure. Also add files to a custom folder
- Fixed connection pool bug to s3
- Added
--remove
which when set will delete all files that were added to the tar file - Refactored the S3Tar class to be easier to work with/test
- Added tests
- Fixed bug where the base folder is added to the tar with no filename
- Added
--save-metadata
. Will save any metadata a file has to a json file of the same name with the suffix.metadata.json
- Added threads
- adds the cli flag
--cache-size
which is the number of files to download and store in memory
- adds the cli flag
- Fixed memory bug where files that were download were not getting cleaned up properly
- Fix bug when looping over a folder with many files
- More output when setting the env var
S3_TAR_LOG_LEVEL=debug
- Added support for a different target bucket
- Cli args changes
--bucket
-->--source-bucket
- Added
--target-bucket
- Cli args changes
- Added support for
.tar.bz2
archives
- Converted to a pypi package and added cli command