buildsys: generate kit metadata and create oci image #271

jmt-lab · 2024-06-05T20:39:10Z

Issue number: 255

Closes #255

Description of changes:

Reintroduces rpm2kit this time with a working version
Expands buildsys build kit to pass through the needed information to build the kit metadata and oci archive

Testing done:

Was able to build local-kits test project and verified
- oci archive loads using docker load -i
- metadata is generated and injected correctly
- contents of oci archive are correct

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

jmt-lab · 2024-06-05T21:10:29Z

Cleaned up commits

jmt-lab · 2024-06-05T21:31:37Z

Adjust based off changes in external-kit-metadata.json

webern

Looks like the local kit building test failed.

tests/projects/local-kit/kits/extra-2-kit/Cargo.toml

tools/buildsys/src/args.rs

tools/buildsys/src/builder.rs

webern · 2024-06-05T22:33:06Z

tools/buildsys/src/manifest.rs

+            .context(error::CargoPackageMissingVendorSnafu {
+                name: self.package.name.clone(),


We should create an issue, if we don't have one already, for deserializing these manifest metadata sections with Serde in a validating manner. We should catch these errors earlier.

We should create an issue, if we don't have one already, for deserializing these manifest metadata sections with Serde in a validating manner. We should catch these errors earlier.

#273

twoliter/embedded/rpm2kit

cbgbt · 2024-06-06T20:12:57Z

twoliter/embedded/rpm2kit

+  sdk: .[0].sdk,
+  kit: (
+    [ .[1] | values[] | {name: ., version: "$VERSION_ID", vendor: "$VENDOR"}]
+    + [ .[0].kit[] | {name: .name, version: .version, vendor: .vendor } ]


Should kits also include the digest of their dependencies?

I suppose if they do, it would be difficult to "override" transitive dependencies during development, but it also seems like it would be surprising if the world shifted underneath you in a production setting.

This also wouldn't work for local kits.

Should kits also include the digest of their dependencies?

🤔 I'm not sure, but I worry about making development too painful. Ideally all the digests would be captured in Cargo.lock and we'd prefer to fetch by digest, so that tags could shift around until the moment they were published, and then after that any change would be detected and cause people to notice.

twoliter/embedded/rpm2kit

webern

Test failure

Unable to instantiate the builder: Failed to create build arguments due to an error reading external kit metadata: No such file or directory (os error 2)

jmt-lab · 2024-06-06T20:33:03Z

Test failure should go away once fetch PR is merged and this is rebased

tools/buildsys/src/manifest/error.rs

tools/buildsys/src/builder.rs

twoliter/embedded/build.Dockerfile

tools/buildsys/src/manifest.rs

tools/buildsys/src/manifest/error.rs

tools/buildsys/src/args.rs

twoliter/embedded/rpm2kit

tools/buildsys/src/manifest/error.rs

webern · 2024-06-10T23:50:21Z

tools/buildsys/src/manifest.rs

+            .context(error::CargoPackageMissingVendorSnafu {
+                name: self.package.name.clone(),


We should create an issue, if we don't have one already, for deserializing these manifest metadata sections with Serde in a validating manner. We should catch these errors earlier.

#273

tools/buildsys/src/args.rs

twoliter/embedded/rpm2kit

bcressey

integ test change LGTM, it is basically going from this:

let project_path = project_dir.join("Twoliter.toml");

to this:

let project_path = project_dir.join("Twoliter.toml");
twoliter_update(&project_path).await;
twoliter_fetch(&project_path, arch).await;

tools/buildsys/src/args.rs

cbgbt · 2024-06-11T21:52:40Z

tools/buildsys/src/builder/error.rs

+    ))]
+    ExternalMetadataRead { source: std::io::Error },
+
+    #[snafu(display("Failed to serialize kit metadata: {source}"))]
+    MetadataSerialize { source: serde_json::Error },


nit: seems like these errors are no longer used.

jmt-lab requested review from bcressey, cbgbt and webern June 5, 2024 20:39

jmt-lab self-assigned this Jun 5, 2024

jmt-lab force-pushed the ootb/external-kits/metadata-file branch from 44f89ac to 81786a9 Compare June 5, 2024 21:09

jmt-lab force-pushed the ootb/external-kits/metadata-file branch 2 times, most recently from 19ed67e to c6c7496 Compare June 5, 2024 21:31

webern reviewed Jun 5, 2024

View reviewed changes

cbgbt reviewed Jun 6, 2024

View reviewed changes

twoliter/embedded/rpm2kit Outdated Show resolved Hide resolved

cbgbt reviewed Jun 6, 2024

View reviewed changes

webern mentioned this pull request Jun 6, 2024

buildsys: validate manifest metadata when deserializing #273

Open

webern reviewed Jun 6, 2024

View reviewed changes

bcressey reviewed Jun 7, 2024

View reviewed changes

tools/buildsys/src/manifest/error.rs Outdated Show resolved Hide resolved

tools/buildsys/src/builder.rs Outdated Show resolved Hide resolved

twoliter/embedded/build.Dockerfile Outdated Show resolved Hide resolved

bcressey mentioned this pull request Jun 10, 2024

core kit migration 🚧 tracking issue bottlerocket-os/bottlerocket#4048

Closed

jmt-lab force-pushed the ootb/external-kits/metadata-file branch from c6c7496 to d0eed51 Compare June 10, 2024 21:48

jmt-lab requested review from bcressey, webern and cbgbt June 10, 2024 21:49

bcressey reviewed Jun 10, 2024

View reviewed changes

jmt-lab force-pushed the ootb/external-kits/metadata-file branch 2 times, most recently from 679eb73 to 1304dd4 Compare June 10, 2024 23:33

webern reviewed Jun 10, 2024

View reviewed changes

webern approved these changes Jun 10, 2024

View reviewed changes

tools/buildsys/src/args.rs Outdated Show resolved Hide resolved

bcressey reviewed Jun 11, 2024

View reviewed changes

jmt-lab force-pushed the ootb/external-kits/metadata-file branch from 1304dd4 to 4c8a738 Compare June 11, 2024 18:41

jmt-lab requested a review from bcressey June 11, 2024 18:43

bcressey approved these changes Jun 11, 2024

View reviewed changes

jmt-lab force-pushed the ootb/external-kits/metadata-file branch from 4c8a738 to 0362885 Compare June 11, 2024 21:32

webern approved these changes Jun 11, 2024

View reviewed changes

tools/buildsys/src/args.rs Outdated Show resolved Hide resolved

jmt-lab force-pushed the ootb/external-kits/metadata-file branch 2 times, most recently from c0d21d3 to e248556 Compare June 11, 2024 21:42

jmt-lab added 2 commits June 11, 2024 14:45

buildsys: kit metadata generation and external kit handling

9080b33

twoliter: rpm2kit to build kit oci image

1948133

jmt-lab force-pushed the ootb/external-kits/metadata-file branch from e248556 to 1948133 Compare June 11, 2024 21:45

webern approved these changes Jun 11, 2024

View reviewed changes

cbgbt approved these changes Jun 11, 2024

View reviewed changes

jmt-lab merged commit a450415 into bottlerocket-os:develop Jun 11, 2024
1 check passed

jmt-lab deleted the ootb/external-kits/metadata-file branch June 11, 2024 22:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

buildsys: generate kit metadata and create oci image #271

buildsys: generate kit metadata and create oci image #271

jmt-lab commented Jun 5, 2024

jmt-lab commented Jun 5, 2024

jmt-lab commented Jun 5, 2024

webern left a comment

webern Jun 5, 2024

webern Jun 10, 2024

cbgbt Jun 6, 2024

bcressey Jun 7, 2024

webern left a comment

jmt-lab commented Jun 6, 2024

webern Jun 10, 2024

bcressey left a comment

cbgbt Jun 11, 2024

		.context(error::CargoPackageMissingVendorSnafu {
		name: self.package.name.clone(),

buildsys: generate kit metadata and create oci image #271

buildsys: generate kit metadata and create oci image #271

Conversation

jmt-lab commented Jun 5, 2024

jmt-lab commented Jun 5, 2024

jmt-lab commented Jun 5, 2024

webern left a comment

Choose a reason for hiding this comment

webern Jun 5, 2024

Choose a reason for hiding this comment

webern Jun 10, 2024

Choose a reason for hiding this comment

cbgbt Jun 6, 2024

Choose a reason for hiding this comment

bcressey Jun 7, 2024

Choose a reason for hiding this comment

webern left a comment

Choose a reason for hiding this comment

jmt-lab commented Jun 6, 2024

webern Jun 10, 2024

Choose a reason for hiding this comment

bcressey left a comment

Choose a reason for hiding this comment

cbgbt Jun 11, 2024

Choose a reason for hiding this comment