Package s3 - AWS S3 VFS implementation.
Rely on github.com/c2fo/vfs/v6/backend
import(
"github.com/c2fo/vfs/v6/backend"
"github.com/c2fo/vfs/v6/backend/s3"
)
func UseFs() error {
fs := backend.Backend(s3.Scheme)
...
}Or call directly:
import "github.com/c2fo/vfs/v6/backend/s3"
func DoSomething() {
fs := s3.NewFileSystem()
...
}s3 can be augmented with the following implementation-specific methods. Backend returns vfs.FileSystem interface so it would have to be cast as s3.FileSystem to use the following:
func DoSomething() {
...
// cast if fs was created using backend.Backend(). Not necessary if created directly from s3.NewFileSystem().
fs = fs.(s3.FileSystem)
// to pass in client options
fs = fs.WithOptions(
s3.Options{
AccessKeyID: "AKIAIOSFODNN7EXAMPLE",
SecretAccessKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
Region: "us-west-2",
ACL: "bucket-owner-full-control",
},
)
// to pass specific client, for instance a mock client
s3apiMock := &mocks.S3API{}
s3apiMock.On("GetObject", mock.AnythingOfType("*s3.GetObjectInput")).
Return(&s3.GetObjectOutput{
Body: nopCloser{bytes.NewBufferString("Hello world!")},
}, nil)
fs = fs.WithClient(s3apiMock)
}Canned ACL's can be passed in as an Option. This string will be applied to all writes, moves, and copies. See https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl for values.
Authentication, by default, occurs automatically when Client() is called. It looks for credentials in the following places, preferring the first location found:
-
StaticProvider - set of credentials which are set programmatically, and will never expire.
-
EnvProvider - credentials from the environment variables of the running process. Environment credentials never expire. Environment variables used:
- Access Key ID:
AWS_ACCESS_KEY_IDorAWS_ACCESS_KEY - Secret Access Key:
AWS_SECRET_ACCESS_KEYorAWS_SECRET_KEY
- Access Key ID:
-
SharedCredentialsProvider - looks for
AWS_SHARED_CREDENTIALS_FILEenv variable. If the env value is empty will default to current user's home directory.- Linux/OSX:
$HOME/.aws/credentials - Windows:
%USERPROFILE%\.aws\credentials
- Linux/OSX:
-
RemoteCredProvider - default remote endpoints such as EC2 or ECS IAM Roles
-
EC2RoleProvider - credentials from the EC2 service, and keeps track if those credentials are expired
See the following for more auth info: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html and https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html
See: https://github.com/aws/aws-sdk-go/tree/master/service/s3
const Scheme = "s3"Scheme defines the file system type.
type File struct {
}File implements vfs.File interface for S3 fs.
func (f *File) Close() errorClose cleans up underlying mechanisms for reading from and writing to the file. Closes and removes the local temp file, and triggers a write to s3 of anything in the f.writeBuffer if it has been created.
func (f *File) CopyToFile(targetFile vfs.File) errorCopyToFile puts the contents of File into the targetFile passed. Uses the S3 CopyObject method if the target file is also on S3, otherwise uses io.Copy.
func (f *File) CopyToLocation(location vfs.Location) (vfs.File, error)CopyToLocation creates a copy of *File, using the file's current name as the new file's name at the given location. If the given location is also s3, the AWS API for copying files will be utilized, otherwise, standard io.Copy will be done to the new file.
func (f *File) Delete(opts ...options.DeleteOption) errorDelete clears any local temp file, or write buffer from read/writes to the file, then makes a DeleteObject call to s3 for the file. If opts is of type DeleteAllVersions, DeleteObject call is made to s3 for each version of the file. Returns any error returned by the API.
func (f *File) Exists() (bool, error)Exists returns a boolean of whether or not the object exists on s3, based on a call for the object's HEAD through the s3 API.
func (f *File) LastModified() (*time.Time, error)LastModified returns the LastModified property of a HEAD request to the s3 object.
func (f *File) Location() vfs.LocationLocation returns a vfs.Location at the location of the object. IE: if file is at s3://bucket/here/is/the/file.txt the location points to s3://bucket/here/is/the/
func (f *File) MoveToFile(targetFile vfs.File) errorMoveToFile puts the contents of File into the targetFile passed using File.CopyToFile. If the copy succeeds, the source file is deleted. Any errors from the copy or delete are returned.
func (f *File) MoveToLocation(location vfs.Location) (vfs.File, error)MoveToLocation works by first calling File.CopyToLocation(vfs.Location) then, if that succeeds, it deletes the original file, returning the new file. If the copy process fails the error is returned, and the Delete isn't called. If the call to Delete fails, the error and the file generated by the copy are both returned.
func (f *File) Name() stringName returns the name portion of the file's key property. IE: "file.txt" of "s3://some/path/to/file.txt
func (f *File) Path() stringPath return the directory portion of the file's key. IE: "path/to" of "s3://some/path/to/file.txt
func (f *File) Read(p []byte) (n int, err error)Read implements the standard for io.Reader. For this to work with an s3 file, a temporary local copy of the file is created, and reads work on that. This file is closed and removed upon calling f.Close()
func (f *File) Seek(offset int64, whence int) (int64, error)Seek implements the standard for io.Seeker. A temporary local copy of the s3 file is created (the same one used for Reads) which Seek() acts on. This file is closed and removed upon calling f.Close()
func (f *File) Size() (uint64, error)Size returns the ContentLength value from an s3 HEAD request on the file's object.
func (f *File) String() stringString implement fmt.Stringer, returning the file's URI as the default string.
func (f *File) URI() stringURI returns the File's URI as a string.
func (f *File) Write(data []byte) (res int, err error)Write implements the standard for io.Writer. A buffer is added to with each subsequent write. When f.Close() is called, the contents of the buffer are used to initiate the PutObject to s3. The underlying implementation uses s3manager which will determine whether it is appropriate to call PutObject, or initiate a multi-part upload.
type FileSystem struct {
}FileSystem implements vfs.FileSystem for the S3 file system.
func NewFileSystem() *FileSystemNewFileSystem initializer for FileSystem struct accepts s3.Client, a local subset of aws-sdk s3.S3API client and returns FileSystem or error.
func (fs *FileSystem) Client() (s3.S3API, error)Client returns the underlying aws s3 client, creating it, if necessary See Authentication for authentication resolution
func (fs *FileSystem) Name() stringName returns "AWS S3"
func (fs *FileSystem) NewFile(volume string, name string) (vfs.File, error)NewFile function returns the s3 implementation of vfs.File.
func (fs *FileSystem) NewLocation(volume string, name string) (vfs.Location, error)NewLocation function returns the s3 implementation of vfs.Location.
func (fs *FileSystem) Scheme() stringScheme return "s3" as the initial part of a file URI ie: s3://
func (fs *FileSystem) WithClient(client interface{}) *FileSystemWithClient passes in an s3 client and returns the filesystem (chainable)
func (fs *FileSystem) WithOptions(opts vfs.Options) *FileSystemWithOptions sets options for client and returns the filesystem (chainable)
type Location struct {
}Location implements the vfs.Location interface specific to S3 fs.
func (l *Location) ChangeDir(relativePath string) errorChangeDir takes a relative path, and modifies the underlying Location's path. The caller is modified by this so the only return is any error. For this implementation there are no errors.
func (l *Location) DeleteFile(fileName string, opts ...options.DeleteOption) errorDeleteFile removes the file at fileName path using given options.
func (l *Location) Exists() (bool, error)Exists returns true if the bucket exists, and the user in the underlying s3.FileSystem.Client() has the appropriate permissions. Will receive false without an error if the bucket simply doesn't exist. Otherwise could receive false and any errors passed back from the API.
func (l *Location) FileSystem() vfs.FileSystemFileSystem returns a vfs.FileSystem interface of the location's underlying file system.
func (l *Location) List() ([]string, error)List calls the s3 API to list all objects in the location's bucket, with a prefix automatically set to the location's path. This will make a call to the s3 API for every 1000 keys to return. If you have many thousands of keys at the given location, this could become quite expensive.
func (l *Location) ListByPrefix(prefix string) ([]string, error)ListByPrefix calls the s3 API with the location's prefix modified relatively by the prefix arg passed to the function. The resource considerations of List() apply to this function as well.
func (l *Location) ListByRegex(regex *regexp.Regexp) ([]string, error)ListByRegex retrieves the keys of all the files at the location's current path, then filters out all those that don't match the given regex. The resource considerations of List() apply here as well.
func (l *Location) NewFile(filePath string) (vfs.File, error)NewFile uses the properties of the calling location to generate a vfs.File (backed by an s3.File). The filePath argument is expected to be a relative path to the location's current path.
func (l *Location) NewLocation(relativePath string) (vfs.Location, error)NewLocation makes a copy of the underlying Location, then modifies its path by calling ChangeDir with the relativePath argument, returning the resulting location. The only possible errors come from the call to ChangeDir, which, for the s3 implementation doesn't ever result in an error.
func (l *Location) Path() stringPath returns the prefix the location references in most s3 calls.
func (l *Location) String() stringString implement fmt.Stringer, returning the location's URI as the default string.
func (l *Location) URI() stringURI returns the Location's URI as a string.
func (l *Location) Volume() stringVolume returns the bucket the location is contained in.
type Options struct {
AccessKeyID string `json:"accessKeyId,omitempty"`
SecretAccessKey string `json:"secretAccessKey,omitempty"`
SessionToken string `json:"sessionToken,omitempty"`
Region string `json:"region,omitempty"`
Endpoint string `json:"endpoint,omitempty"`
}Options holds s3-specific options. Currently only client options are used.