Hub Snapshots

[jortel]

Use Cases

Note: all use cases can performed using API or UI.

As a user, I want to backup the inventory.

As a user, I want to restore a specific snapshot of the inventory.

As a user, I can list, create and delete snapshots.

As a user, I want to demo using a (choice of) snapshots of the inventory.

Overview

A Hub snapshot contains the DB and bucket/file data. They can be used to prime an instance for demos, backup/restore and reverting to a snapshot in time.

• Created
• Listed
• Exported
• Imported
• Deployed
• Deleted

Snapshot Anatomy (tarball):

• /snapshot.yaml - manifest YAML file about the snapshot.
  • name: The name of the snapshot.
  • key: (optional) The encryption key (encrypted using a password).
• /hub.db - The sqlite3 DB file.
• /bucket - The bucket tree.

Snapshots stored on the /snapshot PV.

Workflows:

List

route: GET /snapshots
returns: A list of created or uploaded snapshots. (Listing of name: found in manifests).

Export

route: POST /snapshots/export
body: A password used to encrypt the encryption key.
returns: streamed snapshot (tarball).

Create

route: POST /snapshots/:name
actions:

• Create /snapshot/uuid/snapshot.yaml (with name).
• Create /snapshot/uuid/snapshot.tar.gz

Import

route: POST /snapshots/import/:name
body: multi-part file upload of a tarball and the password used to encrypt the encryption key.
action:

• Store the uploaded in snapshot (tarball) in /snapshot/uuid
• Extract /snapshot.yaml and decrypt the key using the password.
• Encrypt key in /snapshot.yaml using the (local) encryption key and update the extracted file.

Deploy

route: POST /snapshot/deploy/:name
action: Create a symlink /snapshot/deploy => the snapshot (directory) to be deployed.

Deploy Flow in Hub main.

1. Detect /snapshot/deploy
2. Extract the tarball
3. Update Settings.Hub.DB.Path = /snapshot/deploy/hub.db.
4. Update Settings.Hub.Bucket.Path = /snapshot/deploy/bucket
5. Run migration
6. Re-Encrypt identities using the key in /snapshot/deploy/snapshot.yaml and the (local) encryption key.
7. Replace the Live hub.db and /bucket (tree) snapshot.
8. Delete (unlink) /snapshot/deploy.
9. Reload Settings.Hub.
10. Start services.

[aufi]

Hey Jeff, sounds good, I'm adding few quick notes on this topic from slightly different point of view (to ideally get the best solution in the end).

Generally

Use the sqlite database file (with buckets) for export/import/backup/restore seems reasonable to me.

Few cons about exporting the raw database file first

• partial imports are not possible (e.g. python script does GETs on all resources from import dump before actual POST calls, so if there is no collision in IDs, data can be imported even to non-empty Hub)
• it will be harder to change exported data manually before import

However pros are winning IMO

• database versioning&migrations => allows import (most of) older export archives (Hub API doesn't have anything like that)
• work with all-in-one archive, not needed to update this feature with each API resources change
• ...

Flow

A logic managing Snapshots might not be that far from some kind of Tenancy (separation of Konveyor data to Projects). Not exactly with the Flow described above, but if such feature would be reasonable to PM/Ramon, implement a switching of data files (sqlite&buckets) instead of have multiple snapshots and deploy those to the working path might be worth discussion (at least to clarify this does or does not make sense).

Identities encryption

Just to be sure I understand correctly. An encryption key is not part of database, but comes fom Hub env (so will not be exported from the Konveyor instance and thats reason an export key is needed to re-encrypt idetities without exporting original key)?

[jortel]

The encryption key is stored in a secret. The original (source) encryption key is needed to decrypt the identities when the snapshot is imported at the destination. The key (in the manifest) is encrypted using a password supplied by the user when the snapshot is created/exported. The password is used to encrypt the (encryption) key so that it is not stored in the clear in the tarball (at rest). The same password is supplied by the user when the snapshot is imported and used to decrypt the key which is then (re)encrypted using the (local) encryption key.

[shawn-hurley]

One thing to consider is how does this work with OADP/velero and if we need/should write instructions for users. I think we would have to use some hook and then teach OADP how to back up the hub.

Just some thoughts but something to consider as I think this will work for a lot of users, and honestly how I would probably use it, but I wonder how easy it would be to use with a tool like velero

[aufi]

Just note, I think this feature should be used from UI as well as from automated tools or scripts (e.g. RHPDS), so we'd need provide curl commands or put these calls to a simple CLI (or kantra).

[shawn-hurley]

Could you explain the user case from a user experience perspective?

I understand, as a user, that I want to backup and restore. But it sounds like there are other use cases that we are discussing and trying to follow them.

Sorry if this is somewhere else and I just have not read it

[jortel]

One thing to consider is how does this work with OADP/velero and if we need/should write instructions for users. I think we would have to use some hook and then teach OADP how to back up the hub.

Just some thoughts but something to consider as I think this will work for a lot of users, and honestly how I would probably use it, but I wonder how easy it would be to use with a tool like velero

@shawn-hurley I don't know anything about how to use OADP . Can you describe the steps for using OADP to satisfy the use cases?

[konveyor-ci-bot]

This issue is currently awaiting triage.
If contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.
The triage/accepted label can be added by org members.

[shawn-hurley]

OADP is an operator that installs velero, I would look into this: https://docs.openshift.com/container-platform/4.14/backup_and_restore/application_backup_and_restore/oadp-intro.html

If you need more help we can reach out to the team internally