Maintenance: Improve and automate release process

[jeromevdl]

Summary

The release process for Powertools Java is mostly manual as of today. We do have some github actions to help (release-drafter for the release note, release-prep to upgrade versions) but they are not used and not complete.

To improve our MTTR, ease our release process and avoid any manual (human) error, we need to make it as automated as possible.

Why is this needed?

Manual releases are:

• source of error (deploying a wrong version to maven central),
• slow ("what's the next step?"),
• infrequent ("too long, will do it another day")

And this is a vicious circle. slow/infrequent releases increase the risk of errors.

Which area does this relate to?

Automation, Governance

Solution

Leverage knowledge and existing process for Python version, especially see:

• A running release: https://github.com/aws-powertools/powertools-lambda-python/actions/runs/5288817403
• github worklow: https://github.com/aws-powertools/powertools-lambda-python/blob/develop/.github/workflows/release.yml

Also get inspiration on release-maven-plugin.

More precisely:

Inputs:

• the release version (ex: 1.16.0)
• the next snapshot version (ex: 1.17.0-SNAPSHOT)

Steps:

1. checkout code
2. Update maven pom files with the new version. Note sure using jacobtomlinson/gha-find-replace action is a good idea, because if we have a version similar to a dependency, it will also update it. Prefer using the versions-maven-plugin:

mvn versions:set -DnewVersion=x.y.z

⚠️ But it won't update the documentation and other projects (samples / e2e tests)...

3. Create a hash of the code, to make sure code is not tampered when releasing
4. Build / verify (unit + e2e tests + quality checks)

mvn -Pbuild-with-spotbugs clean verify -Dspotbugs.failOnError=true && mvn -Pe2e -B verify --file powertools-e2e-tests/pom.xml

5. Commit and create a tag on git (❓can we do that on main branch. Otherwise we need to do a PR, but then the release process stops as we need to validate the PR. In python they release to Pypi before doing the tag/PR which is weird IMHO, release plugin is tagging before deploying... which is what we want/should do):

git commit -am "chore: version bump ${RELEASE_VERSION}"
git tag -a v"${RELEASE_VERSION}" -m "release_version: v${RELEASE_VERSION}"

7. Checkout the tag
8. Verify the hash: if someone committed in between, there will be a difference and release will stop
9. Build & Release to maven central (use GitHub environments)

mvn clean deploy -Pbuild-without-spotbugs

11. Close issues related to the release (see python script)
12. Update CHANGELOG/releases (can we / do we want to automate this ❓), not sure with the dependabot PRs. We have the draft available, we need to have a human cleaning this.
13. Update maven pom files and everything with the next snapshot version (see step 2):

mvn versions:set -DnewVersion=x.y.z-SNAPSHOT

15. Commit/push (same question as step 5, we may not be able to push to main directly)

Acknowledgment

• This request meets Powertools for AWS Lambda (Java) Tenets
• Should this be considered in other Powertools for AWS Lambda (Java) languages? i.e. Python, TypeScript

[scottgerring]

This is something we really need to invest time into.

Do we want to automate this off of the top of github's existing release mechanism? This creates the tag and nominates the release version and fires an event that github actions can be bound to.

Here I don't really know where we'd get the next version from - the only inputs you give are the current release name / tag, plus the release notes. We could autobump the minor as part of the release and then have a separate triggerable job to update the version to something else, but that's a bit janky.

Alternatively, the whole thing could be a manual triggerable process, which takes 1/ next version and 2/ release notes. An initial step then creates the github release using the current maven version info, and at the end we have what we need to bump the version correctly in one go.

Update CHANGELOG/releases (can we / do we want to automate this question), not sure with the dependabot PRs. We have the draft available, we need to have a human cleaning this.

If we used the release UI, when the build runs the release text has already been cultivated by the person doing the release into github. As the release runs we can pull that text back out and add it to the releases file.

There are also some plugins that might be useful here (e.g. this) but I am not sure how we feel about 1/ injecting extra 3rd party code into our release process and 2/ whether or not we'll get the flexibility we need (e.g. in terms of nested projects such as examples).

[scottgerring]

It's probably worth hacking this up on a test project with unrelated artifacts in mvn to get it going, as we are going to make a mess of our commit and release history, and the contents of maven while we work it out!

[scottgerring]

@sthulb is currently working on simplifying this across all the Powertools libraries; once that is done we can leverage his work to complete the Java process. Marking as blocked in the meantime.

[scottgerring]

Assigning to @sthulb to reflect reality ( 😃 ) and ensure no-one else decides to pick this up in the meantime.

[sthulb]

I've finally picked this up :D

[phipag]

@sthulb I created a PR to fix the end-to-end tests #1816. It would be interesting to see if they also work in the GitHub workflows once we add AWS credentials again.