Storage: Add support for storage bucket backup (from Incus) #13924

boltmark · 2024-08-12T16:43:58Z

This PR adds support for storage bucket backups. It includes cherry-picks from lxc/incus#365, and the below description comes in part from the associated incus PR.

Description

API

The following endpoints are added, along with the storage_bucket_backup extension:

GET /1.0/storage-pools/<pool>/buckets/<bucket>/backups

POST /1.0/storage-pools/<pool>/buckets/<bucket>/backups

GET /1.0/storage-pools/<pool>/buckets/<bucket>/backups/<name>

POST /1.0/storage-pools/<pool>/buckets/<bucket>/backups/<name>

DELETE /1.0/storage-pools/<pool>/buckets/<bucket>/backups/<name>

GET /1.0/storage-pools/<pool>/buckets/<bucket>/backups/<name>/export

Export

CLI

$ lxc storage bucket export
Description:
  Export storage buckets as tarball.

Usage:
  lxc storage bucket export [<remote>:]<pool> <bucket> [<path>] [flags]

Examples:
  lxc storage bucket default b1
      Download a backup tarball of the b1 storage bucket.

Flags:
      --compression   Define a compression algorithm: for backup or none
      --target        Cluster member name

Global Flags:
      --debug          Show all debug messages
      --force-local    Force using the local unix socket
  -h, --help           Print help
      --project        Override the source project
  -q, --quiet          Don't show progress information
      --sub-commands   Use with help or --help to view sub-commands
  -v, --verbose        Show all information messages
      --version        Print version number

$ lxc storage bucket export default bucket0
Backup exported successfully!

Import

CLI

$ lxc storage bucket import
Description:
  Import backups of storage buckets.

Usage:
  lxc storage bucket import [<remote>:]<pool> <backup file> [<bucket>] [flags]

Examples:
  lxc storage bucket import default backup0.tar.gz
                Create a new storage bucket using backup0.tar.gz as the source.

Flags:
      --target   Cluster member name

Global Flags:
      --debug          Show all debug messages
      --force-local    Force using the local unix socket
  -h, --help           Print help
      --project        Override the source project
  -q, --quiet          Don't show progress information
      --sub-commands   Use with help or --help to view sub-commands
  -v, --verbose        Show all information messages
      --version        Print version number

$ lxc storage bucket import default backup.tar.gz bucket0
Backup exported successfully!

Design decisions of note

One of the existing bucket keys is used by default, as everyone can at least read the bucket.
A TransferManager struct is implemented, which utilizes MinIO to handle downloading files from a bucket to create the backup, and uploading files to a bucket when creating a bucket from a backup.

Overall, what's been changed

The command storage bucket export and storage bucket import has been added to the command line interface (CLI).
New APIs added
Added new functionality for performing storage bucket backups and cleaning up expired backups
Added new functionality for importing storage bucket backups
Updated API and CLI documentation

lxd/backup.go

lxd/backup/backup_bucket.go

lxd/storage/s3/transfer_manager.go

github-actions · 2024-08-15T16:15:38Z

Heads up @mionaalex - the "Documentation" label was applied to this issue.

lxd/backup/backup_bucket.go

tomponline · 2025-01-10T11:03:54Z

please can you rebase

boltmark · 2025-02-03T21:27:47Z

@tomponline I've isolated the storage bucket backups functionality as discussed, though I'm seeing the pipeline fail due to large lxc binary size:
FAIL: lxc binary size exceeds 15MiB

I'm not entirely sure what to do about this.

tomponline · 2025-02-03T21:41:05Z

@simondeziel can point u in right direction to update this (if the growth is expected, but it sounds like you may have increasex the lxc command unexpectedly via import changes)

simondeziel · 2025-02-03T21:44:23Z

client/lxd_storage_buckets.go

+		return nil, err
+	}
+
+	op, _, err := r.queryOperation("POST", fmt.Sprintf("/storage-pools/%s/buckets/%s/backups", url.PathEscape(poolName), url.PathEscape(bucketName)), backup, "", true)


Please use string concatenation ("/storage-pools"+url.PathEscape(poolName)+"/buckets/"+...) instead of fmt.Sprintf().

Same applies to each of the new funcs.

simondeziel · 2025-02-03T21:48:47Z

lxd/project/project.go

@@ -123,6 +123,11 @@ func StorageVolumeProjectFromRecord(p *api.Project, volumeType int) string {
 	return api.ProjectDefaultName
 }

+// StorageBucket adds the "<project>_prefix" to the storage bucket name. Even if the project name is "default".
+func StorageBucket(projectName string, storageBucketName string) string {
+	return fmt.Sprintf("%s%s%s", projectName, separator, storageBucketName)


Suggested change

return fmt.Sprintf("%s%s%s", projectName, separator, storageBucketName)

return projectName + separator + storageBucketName

simondeziel · 2025-02-03T21:50:28Z

lxd/storage/s3/transfer_manager.go

+		}
+
+		fi := instancewriter.FileInfo{
+			FileName:    fmt.Sprintf("backup/bucket/%s", objectInfo.Key),


Suggested change

FileName: fmt.Sprintf("backup/bucket/%s", objectInfo.Key),

FileName: "backup/bucket/" + objectInfo.Key,

simondeziel · 2025-02-03T21:53:47Z

lxd/storage/s3/transfer_manager.go

+func (t TransferManager) getEndpoint() string {
+	hostname := t.s3URL.Hostname()
+	if validate.IsNetworkAddressV6(hostname) == nil {
+		hostname = fmt.Sprintf("[%s]", hostname)


Suggested change

hostname = fmt.Sprintf("[%s]", hostname)

hostname = "[" + hostname + "]"

simondeziel · 2025-02-03T21:54:05Z

lxd/storage/s3/transfer_manager.go

+		hostname = fmt.Sprintf("[%s]", hostname)
+	}
+
+	return fmt.Sprintf("%s:%s", hostname, t.s3URL.Port())


Suggested change

return fmt.Sprintf("%s:%s", hostname, t.s3URL.Port())

return hostname + ":" + t.s3URL.Port()

lxd/storage/s3/transfer_manager.go

simondeziel · 2025-02-03T21:56:00Z

lxd/storage/s3/transfer_manager.go

+		DisableCompression: true,
+		TLSClientConfig: &tls.Config{
+			InsecureSkipVerify: true,
+			MinVersion:         tls.VersionTLS12,


Dunno if that'd be applicable here but we have some helper function to get a TLS/HTTPS enabled connection where it uses some LXD specific tweaks like requiring TLS 1.3+.

simondeziel · 2025-02-03T21:56:22Z

lxd/storage_buckets.go

+	defer revert.Fail()
+
+	// Create temporary file to store uploaded backup data.
+	backupFile, err := os.CreateTemp(shared.VarPath("backups"), fmt.Sprintf("%s_", backup.WorkingDirPrefix))


Suggested change

backupFile, err := os.CreateTemp(shared.VarPath("backups"), fmt.Sprintf("%s_", backup.WorkingDirPrefix))

backupFile, err := os.CreateTemp(shared.VarPath("backups"), backup.WorkingDirPrefix+"_")

simondeziel · 2025-02-03T22:00:13Z

test/suites/storage_buckets.sh

@@ -186,3 +186,107 @@ EOF

  delete_object_storage_pool "${poolName}"
 }
+
+test_storage_bucket_export() {


At first glance, that's a rather extensive test script, thanks for that!!

simondeziel · 2025-02-03T22:01:08Z

test/suites/storage_buckets.sh

@@ -186,3 +186,107 @@ EOF

  delete_object_storage_pool "${poolName}"
 }
+
+test_storage_bucket_export() {
+  # shellcheck disable=2039,3043


Disabling those shouldn't be needed AFAIK.

simondeziel · 2025-02-03T22:05:46Z

@tomponline I've isolated the storage bucket backups functionality as discussed, though I'm seeing the pipeline fail due to large lxc binary size: FAIL: lxc binary size exceeds 15MiB

main has lxc at 15717840 bytes ATM. You PR barely brings it above this with:

+ '[' 15746576 -gt 15728640 ']'
FAIL: lxc binary size exceeds 15MiB

15746576 (your PR) - 15717840 (main) = 28736 => ~28KiB.

I'm not entirely sure what to do about this.

It seems like lxc has slowly been growing and your PR just happen to make it above the 15MiB limit. We can probably bump it to 16MiB for now and think about putting it to diet later on ;)

Signed-off-by: Fabian Mettler <[email protected]> (cherry picked from commit 21ed02ae159abd398ea623406101f877d4092b59) Signed-off-by: Mark Bolton <[email protected]> License: Apache-2.0

Signed-off-by: Fabian Mettler <[email protected]> (cherry picked from commit 95bfa8881566ab6a66135a4709065d97feeaaa6b) Signed-off-by: Mark Bolton <[email protected]> License: Apache-2.0

Signed-off-by: Fabian Mettler <[email protected]> (cherry picked from commit b2dbe44d3447c554d9d7e5d4ee855238a7b27c6e) Signed-off-by: Mark Bolton <[email protected]> License: Apache-2.0

Signed-off-by: Fabian Mettler <[email protected]> (cherry picked from commit 8f0061f699db14ba101378ca4314c3ee33afd93c) Signed-off-by: Mark Bolton <[email protected]> License: Apache-2.0

Signed-off-by: Fabian Mettler <[email protected]> (cherry picked from commit f4b0e4dad87ebc1bd0cc94d9b6b0cdaa0a848020) Signed-off-by: Mark Bolton <[email protected]> License: Apache-2.0

Signed-off-by: Fabian Mettler <[email protected]> (cherry picked from commit db96183b5d2f728bcf92f65b54e9265a8ff7a5f5) Signed-off-by: Mark Bolton <[email protected]> License: Apache-2.0

Signed-off-by: Fabian Mettler <[email protected]> (cherry picked from commit 5f6dbb7bea79b315797533d792ff81c3a0bc8fbb) Signed-off-by: Mark Bolton <[email protected]> License: Apache-2.0

Signed-off-by: Fabian Mettler <[email protected]> (cherry picked from commit d5b4350adc2f318ed9fdfd1078eaf8e7f00e8f88) Signed-off-by: Mark Bolton <[email protected]> License: Apache-2.0

Signed-off-by: Mark Bolton <[email protected]>

Signed-off-by: Fabian Mettler <[email protected]> (cherry picked from commit 679ce9355b1f16cbf2a825eb3f0b901b80f35298) Signed-off-by: Mark Bolton <[email protected]> License: Apache-2.0

Signed-off-by: Fabian Mettler <[email protected]> (cherry picked from commit 2e6756328f2e72b1d2b1704a2b2b4373230e2cee) Signed-off-by: Mark Bolton <[email protected]> License: Apache-2.0

Signed-off-by: Fabian Mettler <[email protected]> (cherry picked from commit 98693c062efc05ec500022e7032e63ca96c291ba) Signed-off-by: Mark Bolton <[email protected]> License: Apache-2.0

Signed-off-by: Mark Bolton <[email protected]>

Signed-off-by: Fabian Mettler <[email protected]> (cherry picked from commit 50f73d42057032403b29e5986763d265cfcb5256) Signed-off-by: Mark Bolton <[email protected]> License: Apache-2.0

Signed-off-by: Mark Bolton <[email protected]>

lxd/storage/s3/transfer_manager.go

+		IdleConnTimeout:    30 * time.Second,
+		DisableCompression: true,
+		TLSClientConfig: &tls.Config{
+			InsecureSkipVerify: true,


github-actions bot added Documentation Documentation needs updating API Changes to the REST API labels Aug 12, 2024

boltmark changed the title ~~Storage: Add support for storage buckets backup~~ Storage: Add support for storage bucket backup Aug 12, 2024

boltmark changed the title ~~Storage: Add support for storage bucket backup~~ Storage: Add support for storage bucket backup (from Incus) Aug 12, 2024

github-advanced-security bot found potential problems Aug 12, 2024

View reviewed changes

boltmark force-pushed the storage-buckets-backup branch from 08a58ba to 8fdf38f Compare August 15, 2024 16:15

boltmark force-pushed the storage-buckets-backup branch 4 times, most recently from 19cae2b to 8b2a40c Compare August 16, 2024 01:30

This comment was marked as outdated.

Sign in to view

boltmark force-pushed the storage-buckets-backup branch from 8b2a40c to 94eb9bf Compare August 22, 2024 00:29

boltmark force-pushed the storage-buckets-backup branch 6 times, most recently from eb04463 to ccbbddd Compare September 4, 2024 05:44

github-advanced-security bot found potential problems Sep 4, 2024

View reviewed changes

lxd/backup/backup_bucket.go Fixed Show fixed Hide fixed

github-advanced-security bot found potential problems Sep 4, 2024

View reviewed changes

lxd/backup/backup_bucket.go Fixed Show fixed Hide fixed

lxd/backup/backup_bucket.go Fixed Show fixed Hide fixed

boltmark force-pushed the storage-buckets-backup branch 9 times, most recently from d70a96f to b1894d6 Compare September 5, 2024 15:35

boltmark force-pushed the storage-buckets-backup branch 3 times, most recently from 4a4b0ad to a58a3e1 Compare February 3, 2025 18:36

simondeziel reviewed Feb 3, 2025

View reviewed changes

boltmark force-pushed the storage-buckets-backup branch from a58a3e1 to 818d31b Compare February 5, 2025 04:19

maveonair and others added 19 commits February 4, 2025 20:20

api: Add storage_bucket_backup extension

f776e38

Signed-off-by: Fabian Mettler <[email protected]> (cherry picked from commit 21ed02ae159abd398ea623406101f877d4092b59) Signed-off-by: Mark Bolton <[email protected]> License: Apache-2.0

shared/api: Add storage bucket backup

18b8450

Signed-off-by: Fabian Mettler <[email protected]> (cherry picked from commit 95bfa8881566ab6a66135a4709065d97feeaaa6b) Signed-off-by: Mark Bolton <[email protected]> License: Apache-2.0

lxd/db: Add storage bucket backup functions

ab12981

Signed-off-by: Fabian Mettler <[email protected]> (cherry picked from commit b2dbe44d3447c554d9d7e5d4ee855238a7b27c6e) Signed-off-by: Mark Bolton <[email protected]> License: Apache-2.0

lxd/db/operation: Add storage volume backup types

bdf171a

Signed-off-by: Fabian Mettler <[email protected]> (cherry picked from commit 8f0061f699db14ba101378ca4314c3ee33afd93c) Signed-off-by: Mark Bolton <[email protected]> License: Apache-2.0

lxd/lifecycle: Add storage bucket backup events

56cd88b

Signed-off-by: Fabian Mettler <[email protected]> (cherry picked from commit f4b0e4dad87ebc1bd0cc94d9b6b0cdaa0a848020) Signed-off-by: Mark Bolton <[email protected]> License: Apache-2.0

lxd/project: Add StorageBucket function

d611d33

Signed-off-by: Fabian Mettler <[email protected]> (cherry picked from commit db96183b5d2f728bcf92f65b54e9265a8ff7a5f5) Signed-off-by: Mark Bolton <[email protected]> License: Apache-2.0

lxd/storage/s3: Add transfer manager

2209878

Signed-off-by: Fabian Mettler <[email protected]> (cherry picked from commit 5f6dbb7bea79b315797533d792ff81c3a0bc8fbb) Signed-off-by: Mark Bolton <[email protected]> License: Apache-2.0

lxd: Add storage bucket backup

d7a1148

Signed-off-by: Fabian Mettler <[email protected]> (cherry picked from commit d5b4350adc2f318ed9fdfd1078eaf8e7f00e8f88) Signed-off-by: Mark Bolton <[email protected]> License: Apache-2.0

lxd/auth/drivers: Add can_manage_backups to type storage_bucket

edfcc58

Signed-off-by: Mark Bolton <[email protected]>

lxd/auth: Run make update-auth

ae2179d

Signed-off-by: Mark Bolton <[email protected]>

lxd/metadata: Run make update-metadata

8eb0c8f

Signed-off-by: Mark Bolton <[email protected]>

doc: Run make update-metadata

0271867

Signed-off-by: Mark Bolton <[email protected]>

client: Add storage bucket backup

3c45ed1

Signed-off-by: Fabian Mettler <[email protected]> (cherry picked from commit 679ce9355b1f16cbf2a825eb3f0b901b80f35298) Signed-off-by: Mark Bolton <[email protected]> License: Apache-2.0

lxc: Add storage bucket import/export

3460da0

Signed-off-by: Fabian Mettler <[email protected]> (cherry picked from commit 2e6756328f2e72b1d2b1704a2b2b4373230e2cee) Signed-off-by: Mark Bolton <[email protected]> License: Apache-2.0

doc/rest-api: Refresh swagger YAML

e71202f

Signed-off-by: Fabian Mettler <[email protected]> (cherry picked from commit 98693c062efc05ec500022e7032e63ca96c291ba) Signed-off-by: Mark Bolton <[email protected]> License: Apache-2.0

lxc: Use errors.New instead of fmt.Errorf

9b9bbe8

Signed-off-by: Mark Bolton <[email protected]>

test: Add storage bucket backup

4ab6aba

Signed-off-by: Fabian Mettler <[email protected]> (cherry picked from commit 50f73d42057032403b29e5986763d265cfcb5256) Signed-off-by: Mark Bolton <[email protected]> License: Apache-2.0

.github/workflows: Update max lxc binary size to 16MiB

16a2742

Signed-off-by: Mark Bolton <[email protected]>

i18n: Update translation templates

1bf4b1a

Signed-off-by: Mark Bolton <[email protected]>

boltmark force-pushed the storage-buckets-backup branch from 818d31b to 1bf4b1a Compare February 5, 2025 04:23

github-advanced-security bot found potential problems Feb 5, 2025

View reviewed changes

lxd/storage/s3/transfer_manager.go

IdleConnTimeout: 30 * time.Second,

DisableCompression: true,

TLSClientConfig: &tls.Config{

InsecureSkipVerify: true,

Check failure

Code scanning / CodeQL

Disabled TLS certificate check High

InsecureSkipVerify should not be used in production code.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Storage: Add support for storage bucket backup (from Incus) #13924

Storage: Add support for storage bucket backup (from Incus) #13924

boltmark commented Aug 12, 2024 •

edited

Loading

github-actions bot commented Aug 15, 2024

This comment was marked as outdated.

tomponline commented Jan 10, 2025

boltmark commented Feb 3, 2025

tomponline commented Feb 3, 2025

simondeziel Feb 3, 2025

simondeziel Feb 3, 2025

simondeziel Feb 3, 2025

simondeziel Feb 3, 2025

simondeziel Feb 3, 2025

simondeziel Feb 3, 2025

simondeziel Feb 3, 2025

simondeziel Feb 3, 2025

simondeziel Feb 3, 2025

simondeziel commented Feb 3, 2025

	return fmt.Sprintf("%s%s%s", projectName, separator, storageBucketName)
	return projectName + separator + storageBucketName

	FileName: fmt.Sprintf("backup/bucket/%s", objectInfo.Key),
	FileName: "backup/bucket/" + objectInfo.Key,

	hostname = fmt.Sprintf("[%s]", hostname)
	hostname = "[" + hostname + "]"

	return fmt.Sprintf("%s:%s", hostname, t.s3URL.Port())
	return hostname + ":" + t.s3URL.Port()

	backupFile, err := os.CreateTemp(shared.VarPath("backups"), fmt.Sprintf("%s_", backup.WorkingDirPrefix))
	backupFile, err := os.CreateTemp(shared.VarPath("backups"), backup.WorkingDirPrefix+"_")

Storage: Add support for storage bucket backup (from Incus) #13924

Are you sure you want to change the base?

Storage: Add support for storage bucket backup (from Incus) #13924

Conversation

boltmark commented Aug 12, 2024 • edited Loading

Description

API

Export

CLI

Archive

Import

CLI

Design decisions of note

Overall, what's been changed

github-actions bot commented Aug 15, 2024

This comment was marked as outdated.

tomponline commented Jan 10, 2025

boltmark commented Feb 3, 2025

tomponline commented Feb 3, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

simondeziel commented Feb 3, 2025

boltmark commented Aug 12, 2024 •

edited

Loading