Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Ind Agent] Update packaging to properly package from manifest if given #4885

Merged
merged 37 commits into from
Jul 2, 2024
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
7a0913e
Download from manifest if version has +1 patch
dwhyrock Jun 6, 2024
e7d6a3e
Try modified globExpr
dwhyrock Jun 7, 2024
41fe9fb
use version package
dwhyrock Jun 7, 2024
9f1af73
catch err
dwhyrock Jun 7, 2024
1fad3b2
use panic
dwhyrock Jun 7, 2024
18456a9
Add the filepath
dwhyrock Jun 7, 2024
ac0f535
better package finding
dwhyrock Jun 12, 2024
5c13998
more intermediate work
dwhyrock Jun 13, 2024
9b0a181
more progress
dwhyrock Jun 17, 2024
bacda1f
it seems to maybe work
dwhyrock Jun 18, 2024
49a9dfc
fixed bug
dwhyrock Jun 18, 2024
8c9e21a
Copying spec files as well
dwhyrock Jun 18, 2024
2f05b52
temp test value
dwhyrock Jun 18, 2024
4f827be
fixing linting errors
dwhyrock Jun 18, 2024
01ead7e
Merge branch 'main' into ind-agent-fix-package-endpoint-next-version
dwhyrock Jun 18, 2024
3d9f4ca
Clean up manifest code
dwhyrock Jun 18, 2024
48c65c3
Cleaning up
dwhyrock Jun 18, 2024
5f1ab42
Cleanup 2
dwhyrock Jun 19, 2024
619512d
removing test variable
dwhyrock Jun 19, 2024
9afa054
addressing PR comments and cleanup
dwhyrock Jun 20, 2024
36c2fac
Adding manifest tests
dwhyrock Jun 20, 2024
5f11a2b
Update magefile.go
dwhyrock Jun 20, 2024
8d9bd37
Merge branch 'main' into ind-agent-fix-package-endpoint-next-version
pierrehilbert Jun 21, 2024
799ed15
Switch to go:embed for tests.
cmacknz Jun 24, 2024
4fd3c50
Build component specs from external binaries.
cmacknz Jun 26, 2024
ff3d84a
Convert component to project in var names
cmacknz Jun 26, 2024
6a8c4ae
Return error when package not found.
cmacknz Jun 26, 2024
9fd0ffa
Filter unsupported platforms.
cmacknz Jun 26, 2024
9eda0ac
Fix darwin/arm64 build.
cmacknz Jun 26, 2024
026ece7
Several renames for consistency.
cmacknz Jun 26, 2024
ab3ecae
A few more renames.
cmacknz Jun 26, 2024
b4ce8c7
Move code out of magefile
cmacknz Jun 26, 2024
11404b1
mage fmt
cmacknz Jun 26, 2024
539d648
Fix log message.
cmacknz Jun 26, 2024
c495c94
Fix lint warnings.
cmacknz Jun 26, 2024
03e64b1
Rename test.
cmacknz Jun 26, 2024
7754ffd
Refactor to share download from manifest logic.
cmacknz Jun 28, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions dev-tools/mage/manifest/manifest.go
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,16 @@ func DownloadComponentsFromManifest(manifest string, platforms []string, platfor
// This eliminates the "+buildYYYYMMDDHHMM" suffix on Independent Agent Release builds
majorMinorPatchVersion := parsedManifestVersion.VersionWithPrerelease()

// For Independent Agent Releases, any opted-in projects will have a version that is
// one Patch version ahead since we are using the new bumped DRA artifacts for those projects.
// This is acceptable since it is being released only as bundled with Elastic Agent.
patchPlusOneString := fmt.Sprintf("%d.%d.%d", parsedManifestVersion.Major(), parsedManifestVersion.Minor(), parsedManifestVersion.Patch()+1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason why the input manifest can't specify the patch+1 version? That would keep the logic for this in the release jobs, with the agent package target just doing what the manifest tells it to.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that may be a decent way to go.

I think the way it works is that it uses the version value in the header, which is something like 8.13.4+build202406101001. And then we strip off the metadata to get the base 3-number version for the previously-released (opted-out) projects.

We could probably somewhat easily add something like an optin_version in the header, but that doesn't tell us exactly which projects should use that version.

As it is now, we'd probably have to go through and parse all of the packages in each of the projects and find any that have a filename that includes the patch+1 version in it.

Perhaps we could also include just a list of opted_in_projects at the top level?

We probably need to ask @DaveSys911 whether adding fields to the Ind Agent manifest would be a big issue or not.

parsedPatchPlusOneVersion, err := version.ParseVersion(patchPlusOneString)
if err != nil {
return fmt.Errorf("failed to parse bumped patch version for Ind Agent Release: [%s]", patchPlusOneString)
}
majorMinorPatchPlusOneVersion := parsedPatchPlusOneVersion.VersionWithPrerelease()

errGrp, downloadsCtx := errgroup.WithContext(context.Background())
for component, pkgs := range componentSpec {
for _, platform := range platforms {
Expand All @@ -123,6 +133,11 @@ func DownloadComponentsFromManifest(manifest string, platforms []string, platfor
for _, pkg := range pkgs {
reqPackage := platformPackages[platform]
pkgURL := resolveManifestPackage(projects[component], pkg, reqPackage, majorMinorPatchVersion)
if pkgURL == nil {
// If the artifact doesn't match the manifest's version, check if it matches
// the Patch + 1 version that Independent Agent opted-in projects use
pkgURL = resolveManifestPackage(projects[component], pkg, reqPackage, majorMinorPatchPlusOneVersion)
}
if pkgURL != nil {
for _, p := range pkgURL {
log.Printf(">>>>>>>>> Downloading [%s] [%s] ", pkg, p)
Expand Down
16 changes: 15 additions & 1 deletion magefile.go
Original file line number Diff line number Diff line change
Expand Up @@ -1174,6 +1174,19 @@ func collectPackageDependencies(platforms []string, packageVersion string, requi
return archivePath, dropPath
}

func getIndAgentGlobExpr(versionedFlatPath string, packageVersion string) string {
parsedPackageVersion, err := version.ParseVersion(packageVersion)
if err != nil {
panic(err)
}

bumpedPatchNumber := parsedPackageVersion.Patch() + 1

globExpr := filepath.Join(versionedFlatPath, fmt.Sprintf("*%d.%d.[%d|%d]*", parsedPackageVersion.Major(), parsedPackageVersion.Minor(), parsedPackageVersion.Patch(), bumpedPatchNumber))

return globExpr
}

// flattenDependencies will extract all the required packages collected in archivePath and dropPath in flatPath and
// regenerate checksums
func flattenDependencies(requiredPackages []string, packageVersion, archivePath, dropPath, flatPath string) {
Expand Down Expand Up @@ -1219,7 +1232,8 @@ func flattenDependencies(requiredPackages []string, packageVersion, archivePath,
}
}

globExpr := filepath.Join(versionedFlatPath, fmt.Sprintf("*%s*", packageVersion))
//globExpr := filepath.Join(versionedFlatPath, fmt.Sprintf("*%s*", packageVersion))
globExpr := getIndAgentGlobExpr(versionedFlatPath, packageVersion)
if mg.Verbose() {
log.Printf("Finding files to copy with %s", globExpr)
}
Expand Down
Loading