All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
- updated dependencies
- #214 Fix issue where s3 backend didn't reset
readEOFSeenflag when resetting the file cursor during Seek operations.
- Unit Test improvements: report underlying unit tests errors, always run test cases in a sub-test, always use test suite functions, use more specific assert functions where possible.
- General spelling and grammar corrections everywhere.
- Normalize whitespace: Removed trailing whitespace and superfluous new lines.
- Ensure all mocked types are included in mockery configuration and regenerate.
- Add IncludeInsecureCiphers option to FTP backend. Fixes #204.
- Add ability to set file permissions after writing an SFTP file. Resolves #202.
- Updated mocks to use mockery Expecter. Resolves #200.
- Added additionally-supported HostKeyAlgorithms for SFTP backend. Resolves #198.
- Fixes #196 - Update to latest go and dependencies
- Update .gvmrc to 1.23.0.
- Update go.mod min version to Go 1.23.
- Update actions to test only supported Go versions (1.22, 1.23).
- Updated dependencies.
- Update golangci-lint.yml for latest golang-ci-lint version.
- Fixed #193 - Correct method signature for EncodeAuthority to not return an error.
- Updated dependencies.
- Fixed #191 - Add a util functions to encode authority strictly following RFC 3986.
- Fixed #189 - Update utils authority package to handle proper encoding/decoding of uri with reserved characters.
- Fixed #187 - Update to latest jlaffaye/ftp library to fix issue where FTPS connections were failing due to a bug in the library. Also updated dataconn to continue even if it fails to MakeDir.
- Fixed #185 - location.Exists was checking if a list entry was a directory but it was only checking the first entry.
- updated dependencies
- Fixed #182 - location.Exists was checking the wrong dir for existence. Now it checks the file's parent dir for the subdir's existence.
- Corrected where file.Close() occurs then file.Location().NewFile("somefile.txt") but was not respecting the connection reset. Now it does.
- Fixed #180 - SFTP backend now only opens files for W (instead of RW) except after read/seek. This should fix issues where server permission are write only.
- Fixed #177 - SFTP MoveToFile fails if destination file already exists
- Updated all backends to use error wrapping for Read, Write, Close, Seek. #174
- Update GS backend to comply with new io integration test suite. #172
- Fixed #170 - Fix bug where mem backend doesn't remove fs entries when location.DeleteFile is called
- Fixed #167 - Update CopyToFile to defer file close so cursor is reset.
- Fixes #146 - Update S3 backend to comply with new io integration test suite. Update to no longer write to memory.
- Fixes #156 - Update os backend to comply with new io integration test suite
- Fixes #160 - Update mem backend to comply with new io integration test suite
- Fixes #165 - Update to latest go and dependencies
- Update .gvmrc to 1.22.0.
- Update go.mod min version to Go 1.22.
- Update actions to test only supported Go versions (1.21, 1.22)
- Updated dependencies (ie, Google Cloud, Azure, AWS sdks)
- Fixed #177 - SFTP MoveToFile fails if destination file already exists
- Fixed #158 bug. Updated sftp backend to fix issue where some servers return a generic error message when a file is opened for RW on Read().
- Fixed #154 bug. Updated sftp backend to overwrite except after read/seek.
- Fixed #152 bug where s3 backend failed to read empty files
- Added support for hmac-sha1 and hmac-sha1-96 and removed hmac-ripemd160
- fixed #145 - Remove use of local temp file when reading/seeking from s3 files. This should improve performance by allowing streaming reads from s3 files.
- fixes #149 - Add overridable default HostKeyAlgorithms, Ciphers, MACs, and KeyExchanges
- fixed sftp backend to consider configured ports other than 22
- Update .gvmrc to 1.21.1. Fixes #143.
- Update go.mod min version to Go 1.20.
- Update actions to test only supported Go versions (1.20, 1.21)
- Updated dependencies (ie, Google Cloud, Azure, AWS sdks)
- ftp backend to support connection via FTP, FTPS (implicit TLS), and FTPES (explicit TLS).
- Update go.mod min version to Go 1.18. Fixes #139.
- Update actions to test only supported Go versions (1.19, 1.20)
- Updated dependencies (ie, Google Cloud, Azure, AWS sdks)
- fixed #129(cdd4143) Resolve failing golangci-lint - elliotwutingfeng
- fixed #133 Replaced usage of deprecated ioutil functions with os and io equivalents.
- Update go.mod min version to Go 1.18. Fixes #131.
- Update actions to test only supported Go versions (1.18, 1.19)
- Updated dependencies (ie, Google Cloud, Azure, AWS sdks)
- fixed edge cases where in-memory files either weren't found when they should be, or didn't reset state on new writes
- Replaced pre-generated ssh keypair in sftp options tests with auto-generated pair to remediate false-positive security alerts for hard-coded credentials.
- Fixed the azure deleteAllVersions to fetch versions using file path instead of location path
- Add DeleteAllVersions option support for Azure to remove all versions of a file
- Fixed golangci-lint issues
- Fixed #90 - integration were tests failing on move for mem-to-mem move and bad mutex unlock call when doing mem to non-mem move
- Add DeleteAllVersions option support for S3 to remove all versions of a file
- Add DeleteAllVersions option support for GS to remove all versions of a file
- Update dependencies
- Update .gvmrc
- Update go.mod min version to Go 1.17
- Update actions/setup-go action to v3
- Update actions/checkout action to v3
- Update golangci/golangci-lint-action action to v3
- Update actions to test only supported Go versions (1.17, 1.18)
- Updated dependencies (ie, Google Cloud, Azure, AWS sdks)
- Add S3 option ForcePathStyle. Fixes #96
- Add S3 option DisableServerSideEncryption. Fixes #98
- Added utils.PathToURI() func to change path to OS URI. resolves #112
- fixed linting issues with missing godoc on exported functions and new build tag formatting.
- fixed #92 (broken by #72) where calling ListByPrefix() was fail from non-root locations when calling file-level prefixes.
- fixed azure helper func for vfssimple, ensuring it works on File URIs in addition to Location URIs
- fixed #97 by updating vfssimple logic to ensure the most specific registered backend that matches a url is used, not just the first one it comes across. Updated vfssimple docs.
- Added vfssimple tests. Zero to 100% coverage.
- Fix codecov validation which got lost when converting to github actions. Removed .travis.yml.
- fixed #103 Ensures that sftp backend closed underlying SSH connection on autodisconnect, not just SFTP subsystem.
- Modified sftp Key Exchange option to accept an array instead of a string, allowing multiple kex algorithms
With v6.0.0, sftp.Options struct changed to accept an array of Key Exchange algorithms rather than a string. To update, change the syntax of the auth commands.
"keyExchanges":"diffie-hellman-group-a256"
becomes
"keyExchanges":["diffie-hellman-group-a256"]
- Modified S3 file.go so that on the initial read when the remote file is downloaded, a temporary file is locally created using concurrent go routines to download parts of the file vs. a single request to download the whole object
- Fixed #100 Rolled back Seek validation before Copy or Move in SFTP backend due to bug on some SFTP servers and how we cache open "file handles"
- Add support (and tests) for Go 1.17. Updated dependencies.
- Updated golangci.yml config.
- fixed broken backend integration test.
- fixed #82 Return error when CopyTo/MoveTo functions are called when Seek offset is not (0,0) for all backends, not just GCS.
- fixed #84 where sftp connections were never disconnected in a long-lived app. Added Close to sftp backend Client interface. Close client automatically 10 seconds (configurable in Options) after connecting unless reset by calling some server request action.
- Deprecating utils.TouchCopy (#89).
- Added utils.TouchCopyBuffered(#89). TouchCopyBuffered uses a min buffer size of 256MB and allows for a custom buffer size to be set with filesystem options. Improves large file transfer with negligible impact on smaller file transfers.
- Add support of keyexchanges algorithm as a sftp option
- fixed #71 Listing a Google Cloud Storage bucket does not return things in the root of the bucket
- fixed #78 Return error when CopyTo/MoveTo functions are called when Seek offset is not (0,0) for GS
- fixed #74 utils.TouchCopy() unnecessarily calls file.Size()
- fixed: do not offset number of bytes read for memfs (#70) Jeffrey Larson [email protected]
- fixed S3 VFS backend throws an error when you open then close a file. shivanigaurh-c2fo
- fixed minor markdown doc formatting of type headers
- Add Azure Blob Storage backend. Dusty McIlvain [email protected]
- Add github actions for testing and linting (golangci-lint)
- fixed failing test due to CI asking for mod tidy.
- fixed ineffectual assignment failures keeping vfs from achieving 100% in goreportcard.
- fixed #48 bug where 'invalid cross-device link' error occurs when renaming os files that are found on different devices/volumes including temp dirs.
- fixed #53 bug where s3 file tests were failing due to a deferred removal of a temp file.
- Upgraded dependencies. Required regen of S3 API mock due to a couple new methods. SSH returns a different error text since Go 1.14 (only affects 1 test).
- fixes #49 - Update sftp.Options struct tags
- fixes #21 - OS file.Location().ChangeDir(something/) should not affect location
- fixes #44 - make S3 waitUntilFileExists Exists error more descriptive of root error
- fixes #45 - fix ASCII diagram alignment for SFTP doc
- Improved test coverage for os.FileSystem
- Addresses an issue where writes/touch calls on the vfs.File backend did not work properly on AWS-hosted SFTP environments. (See https://github.com/pkg/sftp/pull/310/files)
- Ensure that writing truncates existing file. Fixes #40
- Ensure that spaces (url encoded or not) in filenames and paths work as expected when copying (particularly for s3 to s3 native copyObject). Fixes #36.
- SFTP backend vfs implementation
- Added fallback to TouchCopy for S3/GCS implementations for CopyTo and MoveTo operations between files and locations which use different authentication and region configurations. When possible, the directory copy/move operations will be used by the underlying clients.
- ACL can now be passed in as an s3.Option value. See https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl for values.
- The GS implementation of location.List() returned an empty string for files found inside a persistent "folder" object created in the UI or by creating a zero-length object ending with "/". These objects are now ignored.
- Fixed a gs-to-gs copy bug that became visible with integration tests added in v5.2.2.
- The OS implementation of file.MoveToLocation() now ensures that the target directory exists before attempting to move the file
- Missing error check in utils.UpdateLastModifiedByMoving()
- Minor goreportcard checks failed. Now 100%
- In-Memory backend vfs implementation
- Included the in-memory backend to the list of registered backends used in backend_integration_test
- Checking for existence at the start of various File functions (size, seek, etc) and returning and error if the file does not exist on the OS backend
- Tests in backend_integration_test that ensure that operations done on non-existent files throw an error
- mem.md in vfs/docs, updated link to it in the README.md
- Relative path validation in utils.go now disallows empty names
- utils_test.go now expects an empty string NOT to validate
- updated README to include "Touch()" definition under the File interface rather than the Location interface
- updated README to exclude "in-memory-backend" from the list of ideas
- Added Touch() method to File interface and implemented in each backend.
- Increased test coverage in both os and s3 backends.
- Defined stricter definitions in interfaces, largely related to input requirements being more URI compliant.
- FileSystem's NewFile and NewLocation required absolute paths. Locations NewFile and NewLocation require relative paths.
- Added/Updated several new utils functions for URI validation across implementations.
- Ensure native methods are being used for atomic functions like copy or move/rename where possible
- Document expected behavior with empty readers in all copy and move function as well as io.Copy
- ListByPrefix now allows relative paths.
- Integration test suite for backend developers that ensures expected interface behaviors and tests interactions between implementations.
- Examples of io.Copy usage in docs.
- Fixed #10
- Fixed vfssimple bug that would return os file system if the uri had the word 'file' anywhere in it even if was s3, like s3://bucket/path/to/file.txt
- Fixed bug where MoveToLocation was returning the old file with updated location rather than a brand new file with new location. Since File may represent a non-existent file, the original file should still be valid.
- Added the Retry() method to the FileSystem interface, along with a no-op default retryer.
- Integrated the newly defined retryer interface into the GCS implementation of VFS and now allow the retry method to be injected as a FileSystem option.
- Integrated the request.Retryer from the s3 client as an injectable implementation for the S3 vfs.Option.
With v4.0.0, we introduced the ability to add a 'retry' option to each registered file system. This version bump will require an update to any existing FileSystem implementations. If a custom retryer is not required by the implemented file system, the default retryer can be returned.
func (fs *FileSystem) Retry() vfs.Retry {
return vfs.DefaultRetryer()
}
Initial release for go modules compliant project.
- Remove go modules from v2 since we don't prefix path with
/v2. See github.com//issues/14.