Enable exporting files which can be combined with OS files via an ove…

…rlay filesystem (#178)

* Add a file exports as an optionally-provided resource type

* Assemble file exports into staged pallet bundles

* Reorganize some code

* Rename "definition" to "manifest" for bundles & stage stores

* Show information about file exports

* Add `stage locate-bun` command

* Add a `--stage-store` flag to override the path of the stage store

* Bump version numbers

CHANGELOG.md

@@ -7,6 +7,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## Unreleased
 
+## 0.7.0-alpha.1 - 2024-04-13
+
+### Added
+
+- (spec) Added a file export resource type as a resource which packages can provide as part of their deployments and/or feature flags.
+- (cli) Added checking of conflicts between file export resources with `plt check`/`stage check`.
+- (cli) When information about a package is shown (e.g. with `cache show-pkg` or `[dev] plt show-pkg`), the source and target paths of file exports are also shown.
+- (cli) When information about a package deployment is shown (e.g. with `stage show-bun-depl` or `[dev] plt show-depl`), the target paths of file exports are also shown.
+- (cli) When information about a staged pallet bundle is shown (e.g. with `stage show-bun`), the target paths of file exports are also shown for each package deployment.
+- (cli) Information about the target paths of file exports for each package deployment in a staged pallet bundle is now recorded in the staged pallet bundle's manifest file, in a new `exports` section.
+- (cli) Added a `stage locate-bun` subcommand to show the absolute file path of the specified staged pallet bundle.
+- (cli) Added a global `--stage-store` string flag which, when not empty, overrides the path of the store of staged pallet bundles to an arbitrary path. When the string is empty (the default behavior), the CLI uses the sstore in the workspace specified by the global `--workspace` flag (i.e. path-of-workspace/.local/share/forklift/stages).
+
 ## 0.7.0-alpha.0 - 2024-04-10
 
 ### Added

README.md

@@ -12,13 +12,13 @@ Note: this is still an experimental prototype and is not yet ready for general u
 
 ## Introduction
 
-Forklift is a software deployment and configuration system providing a simpler, easier, and safer mechanism for updating, reconfiguring, recomposing, and extending browser apps, network services, and system services on single-computer systems (such as a Raspberry Pi or a laptop), especially computers embedded in open-source scientific instruments. While Forklift can also be used in other contexts, it makes tradeoffs specific to the ways in which many open-source scientific instruments need to be deployed and operated (e.g. intermittent internet access, independent administration by individual people, decentralized management & customization).
+Forklift is a software deployment and configuration system providing a simpler, easier, and safer mechanism for updating, reconfiguring, recomposing, and extending browser apps, network services, system services, and operating system configuration files on single-computer systems (such as a Raspberry Pi or a laptop), especially computers embedded in open-source scientific instruments. While Forklift can also be used in other contexts, it makes tradeoffs specific to the ways in which many open-source scientific instruments need to be deployed and operated (e.g. intermittent internet access, independent administration by individual people, decentralized management & customization).
 
-For end-users operating open-source instruments with application services (e.g. network APIs or browser-based interfaces) and/or system services (for e.g. data backups/transfer, hardware support, computer networking, monitoring, etc.), Forklift aims to provide an experience for installing and uninstalling software similar what is achieved by app stores for mobile phones - but with more user control. Forklift also simplifies the process of keeping software up-to-date and the process of rolling software back to older versions if needed; this reduces the need to (for example) re-flash a Raspberry Pi's SD card with a new OS image just to update the application software running on the instrument while still ensuring that the resulting state of the system will be valid.
+For end-users operating open-source instruments with application services (e.g. network APIs or browser-based interfaces) and/or system services (for e.g. data backups/transfer, hardware support, computer networking, monitoring, etc.), Forklift aims to provide an experience for installing and uninstalling software similar what is achieved by app stores for mobile phones - but with more user control. Forklift also simplifies the process of keeping software up-to-date and the process of rolling software back to older versions if needed; this reduces the need to (for example) re-flash a Raspberry Pi's SD card with a new OS image just to update the application software running on the instrument while ensuring the validity of the resulting state of the system.
 
-For open-hardware project developers, Forklift enables Linux-based devices and [appliances](https://en.wikipedia.org/wiki/Computer_appliance) to be retrofitted and extended with an open ecosystem of containerized software - device-specific or general-purpose, project-maintained or third-party. Forklift also provides an incremental path for migrating project-specific application/system services into management by Forklift so that they can be configured, distributed, installed, and replaced by users just like any other app managed by Forklift - i.e. with version control and easy upgrades/rollbacks. The [PlanktoScope](https://www.planktoscope.org/), an open-source microscope for quantitative imaging of plankton, uses Forklift as foundational infrastructure for software releases, deployment, and extensibility in the PlanktoScope's custom Linux distro based on the Raspberry Pi OS; and Forklift was designed specifically to solve the software maintenance and operations challenges experienced in the PlanktoScope project.
+For open-hardware project developers, Forklift enables Linux-based devices and [appliances](https://en.wikipedia.org/wiki/Computer_appliance) to be retrofitted and extended with an open ecosystem of containerized software - device-specific or general-purpose, project-maintained or third-party. Forklift also provides an incremental path for migrating project-specific application/system services and OS configs into management by Forklift so that they can be configured, distributed, installed, and replaced by users just like any other app managed by Forklift - i.e. with version control and easy upgrades/rollbacks. The [PlanktoScope](https://www.planktoscope.org/), an open-source microscope for quantitative imaging of plankton, uses Forklift as foundational infrastructure for software releases, deployment, and extensibility in the [PlanktoScope OS](https://docs-edge.planktoscope.community/reference/software/architecture/os/), a hardware-specific operating system based on the Raspberry Pi OS; and Forklift was designed specifically to solve the software maintenance and operations challenges experienced in the PlanktoScope project.
 
-For indie software developers and sysadmins familiar with DevOps and cloud-native patterns, Forklift is just a GitOps-inspired system which is small and simple enough to work beyond the cloud - using Docker Compose to avoid the architectural complexity, operational overhead, and memory usage of even minimal Kubernetes distributions like k0s; and (in a future version of Forklift) enabling deployment of system files, binaries, and systemd units from configuration files version-controlled in Git repositories. Thus, Forklift allows hassle-free management of software configurations on one or more machines with only occasional internet access and no specialized ops or platform team.
+For indie software developers and sysadmins familiar with DevOps and cloud-native patterns, Forklift is just a GitOps-inspired system which is small and simple enough to work beyond the cloud - using Docker Compose to avoid the architectural complexity, operational overhead, and memory usage of even minimal Kubernetes distributions like k0s; and bundling app deployment with the deployment of system files, executables, and systemd units from configuration files version-controlled in Git repositories. Thus, Forklift allows hassle-free management of software configurations on one or more machines with only occasional internet access (or, in the future, no internet access at all!) and no specialized ops or platform team.
 
 For information about the design of Forklift, please refer to the [design document](./docs/design.md).
 
@@ -121,6 +121,7 @@ The following projects solve related problems with the base OS, though they make
 - systemd's Portable Services pattern and `portablectl` tool provide a way to atomically add system services: <https://systemd.io/PORTABLE_SERVICES/>
 - ostree enables atomic updates of the base OS, but [it is not supported by Raspberry Pi OS](https://github.com/ostreedev/ostree/issues/2223): <https://ostreedev.github.io/ostree/>
 - The bootc project enables the entire operating system to be delivered as a bootable OCI container image, but currently it relies on ostree: <https://containers.github.io/bootc/>
+- gokrazy enables atomic deployment of Go programs (and also of software containers!), but it has a very different architecture compared traditional Linux distros: <https://gokrazy.org/>
 
 Other related OS-level projects can be found at [github.com/castrojo/awesome-immutable](https://github.com/castrojo/awesome-immutable).
 

cmd/forklift/dev/plt/pallets.go

@@ -212,20 +212,20 @@ func stageAction(versions Versions) cli.ActionFunc {
 		if err != nil {
 			return err
 		}
-		stageStore, err := workspace.GetStageStore(versions.NewStageStore)
+		stageStore, err := fcli.GetStageStore(
+			workspace, c.String("stage-store"), versions.NewStageStore,
+		)
 		if err != nil {
 			return err
 		}
-		if _, err = fcli.StagePallet(pallet, stageStore, cache, versions.NewBundle); err != nil {
-			return err
-		}
-		if err = fcli.DownloadImagesForStoreApply(
-			stageStore, versions.Tool, versions.MinSupportedBundle, c.Bool("parallel"),
-			c.Bool("ignore-tool-version"),
+		if _, err = fcli.StagePallet(
+			pallet, stageStore, cache, c.String("exports"),
+			versions.Tool, versions.MinSupportedBundle, versions.NewBundle,
+			c.Bool("parallel"), c.Bool("ignore-tool-version"),
 		); err != nil {
-			return errors.Wrap(err, "couldn't cache Docker container images required by staged pallet")
+			return err
 		}
-		fmt.Println("Done! To apply the staged pallet, you can run `sudo -E forklift stage apply`.")
+		fmt.Println("Done! To apply the staged pallet, run `sudo -E forklift stage apply`.")
 		return nil
 	}
 }
@@ -249,12 +249,29 @@ func applyAction(versions Versions) cli.ActionFunc {
 			return err
 		}
 
-		if err = fcli.ApplyPallet(
-			pallet, repoCache, workspace, versions.NewStageStore, versions.NewBundle, c.Bool("parallel"),
-		); err != nil {
+		stageStore, err := fcli.GetStageStore(
+			workspace, c.String("stage-store"), versions.NewStageStore,
+		)
+		if err != nil {
 			return err
 		}
-		fmt.Println("Done!")
+		index, err := fcli.StagePallet(
+			pallet, stageStore, repoCache, c.String("exports"),
+			versions.Tool, versions.MinSupportedBundle, versions.NewBundle,
+			c.Bool("parallel"), c.Bool("ignore-tool-version"),
+		)
+		if err != nil {
+			return errors.Wrap(err, "couldn't stage pallet to be applied immediately")
+		}
+
+		bundle, err := stageStore.LoadFSBundle(index)
+		if err != nil {
+			return errors.Wrapf(err, "couldn't load staged pallet bundle %d", index)
+		}
+		if err = fcli.ApplyNextOrCurrentBundle(0, stageStore, bundle, c.Bool("parallel")); err != nil {
+			return errors.Wrapf(err, "couldn't apply staged pallet bundle %d", index)
+		}
+		fmt.Println("Done! You may need to reboot for some changes to take effect.")
 		return nil
 	}
 }

cmd/forklift/main.go

@@ -60,6 +60,13 @@ var app = &cli.App{
 			Usage:   "Path of the forklift workspace",
 			EnvVars: []string{"FORKLIFT_WORKSPACE"},
 		},
+		&cli.StringFlag{
+			Name:    "stage-store",
+			Aliases: []string{"ss"},
+			Value:   "",
+			Usage:   "Path of the forklift stage store, overriding the default path in the workspace",
+			EnvVars: []string{"FORKLIFT_STAGE_STORE"},
+		},
 		&cli.BoolFlag{
 			Name:    "ignore-tool-version",
 			Value:   false,
@@ -81,16 +88,16 @@ const (
 	palletMinVersion = "v0.4.0"
 	// bundleMinVersion is the minimum supported Forklift version among staged pallet bundles. A
 	// bundle with a lower Forklift version cannot be used.
-	bundleMinVersion = "v0.7.0-alpha.0"
+	bundleMinVersion = "v0.7.0-alpha.1"
 	// newBundleVersion is the Forklift version reported in new staged pallet bundles made by Forklift.
 	// Older versions of the Forklift tool cannot use such bundles.
-	newBundleVersion = "v0.7.0-alpha.0"
+	newBundleVersion = "v0.7.0-alpha.1"
 	// newStageStoreVersion is the Forklift version reported in a stage store initialized by Forklift.
 	// Older versions of the Forklift tool cannot use the state store.
-	newStageStoreVersion = "v0.7.0-alpha.0"
+	newStageStoreVersion = "v0.7.0-alpha.1"
 	// fallbackVersion is the version reported which the Forklift tool reports itself as if its actual
 	// version is unknown.
-	fallbackVersion = "v0.7.0-alpha.0"
+	fallbackVersion = "v0.7.0-alpha.2-dev"
 )
 
 var (

cmd/forklift/plt/pallets.go

@@ -76,53 +76,39 @@ func switchAction(versions Versions) cli.ActionFunc {
 		if err != nil {
 			return err
 		}
-
-		// clone pallet
-		if err = os.RemoveAll(workspace.GetCurrentPalletPath()); err != nil {
-			return errors.Wrap(err, "couldn't remove local pallet")
-		}
-		if err = fcli.CloneQueriedGitRepoUsingLocalMirror(
-			0, workspace.GetPalletCachePath(), c.Args().First(), workspace.GetCurrentPalletPath(),
-		); err != nil {
+		pallet, repoCache, err := preparePallet(c, workspace, versions)
+		if err != nil {
 			return err
 		}
 		fmt.Println()
-		// TODO: warn if the git repo doesn't appear to be an actual pallet, or if the pallet's forklift
-		// version is incompatible
 
-		// cache everything required by pallet
-		pallet, repoCache, err := processFullBaseArgs(c, false)
+		stageStore, err := fcli.GetStageStore(
+			workspace, c.String("stage-store"), versions.NewStageStore,
+		)
 		if err != nil {
 			return err
 		}
-		if err = fcli.CheckShallowCompatibility(
-			pallet, repoCache, versions.Tool, versions.MinSupportedRepo, versions.MinSupportedPallet,
-			c.Bool("ignore-tool-version"),
-		); err != nil {
-			return err
-		}
-		if _, err = fcli.CacheStagingRequirements(pallet, repoCache.Path()); err != nil {
+		index, err := fcli.StagePallet(
+			pallet, stageStore, repoCache, c.String("exports"),
+			versions.Tool, versions.MinSupportedBundle, versions.NewBundle,
+			c.Bool("parallel"), c.Bool("ignore-tool-version"),
+		)
+		if err != nil {
 			return err
 		}
-		fmt.Println()
-
 		if !c.Bool("apply") {
-			// stage pallet
-			if err = stagePallet(
-				workspace, pallet, repoCache, versions, c.Bool("parallel"), c.Bool("ignore-tool-version"),
-			); err != nil {
-				return err
-			}
 			fmt.Println("Done! To apply the staged pallet, run `sudo -E forklift stage apply`.")
 			return nil
 		}
-		// apply pallet
-		if err = fcli.ApplyPallet(
-			pallet, repoCache, workspace, versions.NewStageStore, versions.NewBundle, c.Bool("parallel"),
-		); err != nil {
-			return err
+
+		bundle, err := stageStore.LoadFSBundle(index)
+		if err != nil {
+			return errors.Wrapf(err, "couldn't load staged pallet bundle %d", index)
 		}
-		fmt.Println("Done!")
+		if err = fcli.ApplyNextOrCurrentBundle(0, stageStore, bundle, c.Bool("parallel")); err != nil {
+			return errors.Wrapf(err, "couldn't apply staged pallet bundle %d", index)
+		}
+		fmt.Println("Done! You may need to reboot for some changes to take effect.")
 		return nil
 	}
 }
@@ -141,25 +127,36 @@ func ensureWorkspace(wpath string) (*forklift.FSWorkspace, error) {
 	return workspace, nil
 }
 
-func stagePallet(
-	workspace *forklift.FSWorkspace, pallet *forklift.FSPallet, repoCache forklift.PathedRepoCache,
-	versions Versions, parallel, ignoreToolVersion bool,
-) error {
-	stageStore, err := workspace.GetStageStore(versions.NewStageStore)
-	if err != nil {
-		return err
+func preparePallet(
+	c *cli.Context, workspace *forklift.FSWorkspace, versions Versions,
+) (pallet *forklift.FSPallet, repoCache forklift.PathedRepoCache, err error) {
+	// clone pallet
+	if err = os.RemoveAll(workspace.GetCurrentPalletPath()); err != nil {
+		return nil, nil, errors.Wrap(err, "couldn't remove local pallet")
 	}
-	if _, err = fcli.StagePallet(pallet, stageStore, repoCache, versions.NewBundle); err != nil {
-		return errors.Wrap(err, "couldn't stage pallet to be applied immediately")
+	if err = fcli.CloneQueriedGitRepoUsingLocalMirror(
+		0, workspace.GetPalletCachePath(), c.Args().First(), workspace.GetCurrentPalletPath(),
+	); err != nil {
+		return nil, nil, err
 	}
-
 	fmt.Println()
-	if err = fcli.DownloadImagesForStoreApply(
-		stageStore, versions.Tool, versions.MinSupportedBundle, parallel, ignoreToolVersion,
+	// TODO: warn if the git repo doesn't appear to be an actual pallet, or if the pallet's forklift
+	// version is incompatible
+
+	// cache everything required by pallet
+	if pallet, repoCache, err = processFullBaseArgs(c, false); err != nil {
+		return nil, nil, err
+	}
+	if err = fcli.CheckShallowCompatibility(
+		pallet, repoCache, versions.Tool, versions.MinSupportedRepo, versions.MinSupportedPallet,
+		c.Bool("ignore-tool-version"),
 	); err != nil {
-		return errors.Wrap(err, "couldn't cache Docker container images required by staged pallet")
+		return pallet, repoCache, err
 	}
-	return nil
+	if _, err = fcli.CacheStagingRequirements(pallet, repoCache.Path()); err != nil {
+		return pallet, repoCache, err
+	}
+	return pallet, repoCache, nil
 }
 
 // clone
@@ -333,11 +330,17 @@ func stageAction(versions Versions) cli.ActionFunc {
 		if err != nil {
 			return err
 		}
-		stageStore, err := workspace.GetStageStore(versions.NewStageStore)
+		stageStore, err := fcli.GetStageStore(
+			workspace, c.String("stage-store"), versions.NewStageStore,
+		)
 		if err != nil {
 			return err
 		}
-		if _, err = fcli.StagePallet(pallet, stageStore, cache, versions.NewBundle); err != nil {
+		if _, err = fcli.StagePallet(
+			pallet, stageStore, cache, c.String("exports"),
+			versions.Tool, versions.MinSupportedBundle, versions.NewBundle,
+			c.Bool("parallel"), c.Bool("ignore-tool-version"),
+		); err != nil {
 			return err
 		}
 		fmt.Println("Done! To apply the staged pallet, run `sudo -E forklift stage apply`.")
@@ -364,12 +367,29 @@ func applyAction(versions Versions) cli.ActionFunc {
 			return err
 		}
 
-		if err = fcli.ApplyPallet(
-			pallet, repoCache, workspace, versions.NewStageStore, versions.NewBundle, c.Bool("parallel"),
-		); err != nil {
+		stageStore, err := fcli.GetStageStore(
+			workspace, c.String("stage-store"), versions.NewStageStore,
+		)
+		if err != nil {
 			return err
 		}
-		fmt.Println("Done!")
+		index, err := fcli.StagePallet(
+			pallet, stageStore, repoCache, c.String("exports"),
+			versions.Tool, versions.MinSupportedBundle, versions.NewBundle,
+			c.Bool("parallel"), c.Bool("ignore-tool-version"),
+		)
+		if err != nil {
+			return errors.Wrap(err, "couldn't stage pallet to be applied immediately")
+		}
+
+		bundle, err := stageStore.LoadFSBundle(index)
+		if err != nil {
+			return errors.Wrapf(err, "couldn't load staged pallet bundle %d", index)
+		}
+		if err = fcli.ApplyNextOrCurrentBundle(0, stageStore, bundle, c.Bool("parallel")); err != nil {
+			return errors.Wrapf(err, "couldn't apply staged pallet bundle %d", index)
+		}
+		fmt.Println("Done! You may need to reboot for some changes to take effect.")
 		return nil
 	}
 }