Skip to content

Commit

Permalink
more context-specific command-line documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
tomcz committed Oct 9, 2020
1 parent a102bff commit 99002f4
Show file tree
Hide file tree
Showing 2 changed files with 50 additions and 52 deletions.
54 changes: 17 additions & 37 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,34 +2,25 @@

# S3 backup script in a single binary

Provides a standard way of backing up an archive to a S3 bucket, and restoring the backed up
archive from its S3 bucket. No more custom backup scripts please ...
Provides a standard way of backing up an archive to a S3 bucket, and restoring the backed up archive from its S3 bucket. No more custom backup scripts please ...

You can download the latest release from [here](https://github.com/tomcz/s3backup/releases).

## Upload process

1. Encrypt the file to be backed up (optional but highly recommended). `s3backup` uses AES-256
encryption via a password or a Base64-encoded secret key, or a PEM-encoded RSA public key. If
a public key is provided, `s3backup` will generate a random 256-bit symmetric key which will
be encrypted using the public key and stored with the encrypted file. To make key creation
easier, you can use the `keygen` commands as outlined [below](#backup-key-generation).
1. Encrypt the file to be backed up (optional but highly recommended). `s3backup` uses AES-256 encryption via a password of your choice, a Base64-encoded secret key, or a PEM-encoded RSA public key. If a public key is provided, `s3backup` will generate a random 256-bit symmetric key which will be encrypted using the public key and stored with the encrypted file. To make key creation easier, you can use the `keygen` commands as outlined [below](#backup-key-generation).

2. Calculate SHA-256 checksum for the file to be uploaded. For encrypted uploads the checksum
is calculated on the encrypted file.
2. Calculate SHA-256 checksum for the file to be uploaded. For encrypted uploads the checksum is calculated on the encrypted file.

3. Upload to AWS S3 using concurrent uploads to handle large files and store the checksum with
the uploaded file.
3. Upload to AWS S3 using concurrent uploads to handle large files and store the checksum with the uploaded file.

## Download process

1. Download file from AWS S3 using concurrent downloads to handle large files and retrieve the
stored checksum of the uploaded file.
1. Download file from AWS S3 using concurrent downloads to handle large files and retrieve the stored checksum of the uploaded file.

2. Verify that the stored checksum matches the downloaded file.

3. Optionally decrypt the downloaded file using either the same symmetric key that was used
to encrypt it, or the RSA private key matching the RSA public key that was used for encryption.
3. Optionally decrypt the downloaded file using either the same symmetric key that was used to encrypt it, or the RSA private key matching the RSA public key that was used for encryption.

## Usage

Expand Down Expand Up @@ -58,9 +49,7 @@ GLOBAL OPTIONS:

### AWS S3 Credentials

AWS S3 integration in `s3backup` can be configured from the command line, and using AWS' environment
variables and config files. [Click here](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html)
for details on using default AWS credentials.
AWS S3 integration in `s3backup` can be configured from the command line, and using AWS environment variables and config files. [Click here](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html) for details on using default AWS credentials.

#### s3backup put

Expand All @@ -73,14 +62,13 @@ USAGE:
OPTIONS:
--symKey value Password to use for symmetric AES encryption (optional)
--pemKey FILE Path to PEM-encoded public or private key FILE (optional)
--pemKey FILE Path to PEM-encoded public key FILE (optional)
--accessKey value AWS Access Key ID (if not using default AWS credentials)
--secretKey value AWS Secret Key (required when accessKey is provided)
--token value AWS Token (effective only when accessKey is provided, depends on your AWS setup)
--region value AWS Region, override when necessary (default: "us-east-1")
--endpoint URL Custom AWS Endpoint URL (optional)
--nocheck Do not create or verify backup checksums (default: false)
--help, -h show help (default: false)
--nocheck Do not create backup checksums (default: false)
```

#### s3backup get
Expand All @@ -93,26 +81,21 @@ USAGE:
s3backup get [command options] s3://bucket/objectkey local_file_path
OPTIONS:
--symKey value Password to use for symmetric AES encryption (optional)
--pemKey FILE Path to PEM-encoded public or private key FILE (optional)
--symKey value Password to use for symmetric AES decryption (optional)
--pemKey FILE Path to PEM-encoded private key FILE (optional)
--accessKey value AWS Access Key ID (if not using default AWS credentials)
--secretKey value AWS Secret Key (required when accessKey is provided)
--token value AWS Token (effective only when accessKey is provided, depends on your AWS setup)
--region value AWS Region, override when necessary (default: "us-east-1")
--endpoint URL Custom AWS Endpoint URL (optional)
--nocheck Do not create or verify backup checksums (default: false)
--help, -h show help (default: false)
--nocheck Do not verify backup checksums (default: false)
```

### HashiCorp Vault

`s3backup` provides `vault-put` and `vault-get` commands that allow it to be configured using secrets
held by a [vault](https://www.vaultproject.io/) instance so that you can store encryption keys and AWS
credentials in a secure manner. The secrets that you need to hold in vault for `s3backup` are described
[here](https://github.com/tomcz/s3backup/blob/master/config/config.go).
`s3backup` provides `vault-put` and `vault-get` commands that allow it to be configured using secrets held by a [vault](https://www.vaultproject.io/) instance so that you can store encryption keys and AWS credentials in a secure manner. The secrets that you need to hold in vault for `s3backup` are described [here](https://github.com/tomcz/s3backup/blob/master/config/config.go).

Vault integration in `s3backup` can be configured from the command line, and using vault's own
[environment variables](https://www.vaultproject.io/docs/commands/environment.html).
Vault integration in `s3backup` can be configured from the command line, and using vault's own [environment variables](https://www.vaultproject.io/docs/commands/environment.html).

#### s3backup vault-put

Expand All @@ -130,8 +113,7 @@ OPTIONS:
--path value Vault secret path containing backup credentials (required)
--caCert FILE Vault root certificate FILE (optional)
--vault URL Vault service URL (required)
--nocheck Do not create or verify backup checksums (default: false)
--help, -h show help (default: false)
--nocheck Do not create backup checksums (default: false)
```

#### s3backup vault-get
Expand All @@ -150,14 +132,12 @@ OPTIONS:
--path value Vault secret path containing backup credentials (required)
--caCert FILE Vault root certificate FILE (optional)
--vault URL Vault service URL (required)
--nocheck Do not create or verify backup checksums (default: false)
--help, -h show help (default: false)
--nocheck Do not verify backup checksums (default: false)
```

## Backup key generation

To make things easier, `s3backup` also provides `keygen` commands to create 256-bit symmetric keys
and 2048-bit RSA private/public key pairs suitable for use by `s3backup`.
To make things easier, `s3backup` also provides `keygen` commands to create 256-bit symmetric keys and 2048-bit RSA private/public key pairs suitable for use by `s3backup`.

```
NAME:
Expand Down
48 changes: 33 additions & 15 deletions cmd/s3backup/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -44,28 +44,28 @@ func main() {
Usage: "Upload file to S3 bucket using local credentials",
ArgsUsage: "s3://bucket/objectkey local_file_path",
Action: basicPut,
Flags: basicFlags(),
Flags: basicFlags(true),
}
cmdBasicGet := &cli.Command{
Name: "get",
Usage: "Download file from S3 bucket using local credentials",
ArgsUsage: "s3://bucket/objectkey local_file_path",
Action: basicGet,
Flags: basicFlags(),
Flags: basicFlags(false),
}
cmdVaultPut := &cli.Command{
Name: "vault-put",
Usage: "Upload file to S3 bucket using credentials from vault",
ArgsUsage: "s3://bucket/objectkey local_file_path",
Action: vaultPut,
Flags: vaultFlags(),
Flags: vaultFlags(true),
}
cmdVaultGet := &cli.Command{
Name: "vault-get",
Usage: "Download file from S3 bucket using credentials from vault",
ArgsUsage: "s3://bucket/objectkey local_file_path",
Action: vaultGet,
Flags: vaultFlags(),
Flags: vaultFlags(false),
}
cmdGenAES := &cli.Command{
Name: "aes",
Expand All @@ -88,14 +88,14 @@ func main() {
Usage: "Just encrypt a local file",
ArgsUsage: "inFile outFile",
Action: encryptLocalFile,
Flags: cipherFlags(),
Flags: cipherFlags(true),
}
cmdDecrypt := &cli.Command{
Name: "decrypt",
Usage: "Just decrypt a local file",
ArgsUsage: "inFile outFile",
Action: decryptLocalFile,
Flags: cipherFlags(),
Flags: cipherFlags(false),
}
app := &cli.App{
Name: "s3backup",
Expand All @@ -117,16 +117,24 @@ func main() {
}
}

func basicFlags() []cli.Flag {
func basicFlags(encrypt bool) []cli.Flag {
sym := "decryption"
asym := "private"
check := "verify"
if encrypt {
sym = "encryption"
asym = "public"
check = "create"
}
return []cli.Flag{
&cli.StringFlag{
Name: "symKey",
Usage: "Password to use for symmetric AES encryption (optional)",
Usage: fmt.Sprintf("Password to use for symmetric AES %s (optional)", sym),
Destination: &symKey,
},
&cli.StringFlag{
Name: "pemKey",
Usage: "Path to PEM-encoded public or private key `FILE` (optional)",
Usage: fmt.Sprintf("Path to PEM-encoded %s key `FILE` (optional)", asym),
Destination: &pemKeyFile,
},
&cli.StringFlag{
Expand Down Expand Up @@ -158,28 +166,38 @@ func basicFlags() []cli.Flag {
},
&cli.BoolFlag{
Name: "nocheck",
Usage: "Do not create or verify backup checksums",
Usage: fmt.Sprintf("Do not %s backup checksums", check),
Destination: &skipHash,
},
}
}

func cipherFlags() []cli.Flag {
func cipherFlags(encrypt bool) []cli.Flag {
sym := "decryption"
asym := "private"
if encrypt {
sym = "encryption"
asym = "public"
}
return []cli.Flag{
&cli.StringFlag{
Name: "symKey",
Usage: "Password to use for symmetric AES encryption",
Usage: fmt.Sprintf("Password to use for symmetric AES %s", sym),
Destination: &symKey,
},
&cli.StringFlag{
Name: "pemKey",
Usage: "Path to PEM-encoded public or private key `FILE`",
Usage: fmt.Sprintf("Path to PEM-encoded %s key `FILE`", asym),
Destination: &pemKeyFile,
},
}
}

func vaultFlags() []cli.Flag {
func vaultFlags(encrypt bool) []cli.Flag {
check := "verify"
if encrypt {
check = "create"
}
return []cli.Flag{
&cli.StringFlag{
Name: "role",
Expand Down Expand Up @@ -215,7 +233,7 @@ func vaultFlags() []cli.Flag {
},
&cli.BoolFlag{
Name: "nocheck",
Usage: "Do not create or verify backup checksums",
Usage: fmt.Sprintf("Do not %s backup checksums", check),
Destination: &skipHash,
},
}
Expand Down

0 comments on commit 99002f4

Please sign in to comment.