Making the duffle store transportable

[youreddy]

In its current architecture, duffle limits the ability to have different users interact with the cli and perform bundle lifecycle actions on the same installation. If I want the ability to have another user or maybe a script in a CI system run subsequent actions on my installation, there's a bit of manual toil I need to do in order to make that happen.

I have to create a tarball of the duffle home directory every time the state changes and uncompress that tarball at the location where I want to run the subsequent command. This new location could be at a different workstation completely or at a later point in the CI pipeline.

I also have to make sure that the format of the duffle home directory works with the version of the duffle cli that I have available since there's no guarantee that the duffle home directory structure will work with all duffle cli versions.

In order to make this process less manual, we could incorporate some of the following changes:

• Consider changes to duffle's home directory structure as a reason to ship new versions of the duffle cli
• Make it easier to export and compress the contents of the duffle home directory.
• Make it easer to import a compressed duffle home file into duffle's home directory.

I'm imagining something like this:

# a command for compressing duffle's home directory. 
# with an optional --home-dir flag to specify the directory to compress

$ duffle carry out --home-dir ~/.duffle-home-staging

# a command for importing a compressed duffle file into the current home directory. 
# with an optional --home-dir flag  to specify the home directory to import into. 
# this command would overwrite the home-dir contents. 

$ duffle carry in --home-file duffle-home-0.34.0.tgz --home-dir ~/.duffle-home-staging

Additionally, we could add safeguards where duffle carry out produces a tarball that contains a version file to track the duffle cli version. If the home-dir path was not empty, duffle carry in could check for a version file before overriding the contents in that directory and throw an error if the versions were mismatched.

I think there's some edge cases to consider for duffle carry in. What should the behavior be if the home dir being imported into is not empty? Should there be a --force option, or should it always overwrite the contents? It seems the simplest approach would be to overwrite.

[glyn]

The duffle home directory contains various subdirectories including bundles, claims, credentials, logs, plugins, and a repositories.json file. Would we want to export all these? Exporting credentials seems a bit risky.

[youreddy]

Exporting everything would make it easier for the user to be up and running with duffle after running carry in in a new location. There will still be some amount of manual reconfiguration. Especially if files were used for parameters and credentials as files won't be guaranteed to exist in the new location.

Given that exporting credentials are risky and they may require reconfiguration anyway, we can make carry out exclude credentials by default. If users want to take credentials with them, they can opt in with a --include-credentials flag.