ADR 2. CI/CD Platform

[beechnut]

ADR 2: Use GitHub Actions for CI/CD is now ready for discussion & review! attn: @AdamUSDA @jayrbergjr @Cope1-FS

Reminder that this is an in-progress document, not a final draft. As you review this, please:

• Add research in the form of Pros/Cons to the considered options. Feel free to post them as a comment (tag @beechnut, I'll add them) or commit them to the branch. We want to make a thoroughly considered decision here, so the more information the better.
• Amend the RACI section at the bottom, with extra focus on the Accountable section—I don't know the FS approvals process the way @AdamUSDA, @aaronburk, or other folks might.

Please respond here with any questions, comments, amendments, additional research, approvals, etc., or make proposed changes to the file. Thanks!

---

A little more background:

A few weeks ago Matt Reiss presented a slide to @AdamUSDA and I with approved technologies. For this ADR, I researched all the options under the "Continuous Integration & Build" and "Containerization & Continuous Deployment" columns that could be used as a CI/CD pipeline (which excluded only Kubernetes).

Jenkins, Azure DevOps, and GitHub Actions were the three main candidates based on appropriate languages & technology. I spoke with Chaochung Tsai three times (in great detail) about Jenkins, as well as Jai Jaiprakash, who has a lot of familiarity setting up NRM apps on Jenkins.

I landed on proposing GitHub Actions in this draft despite wanting to be able to recommend Jenkins, the clear standard in the Forest Service. This research uncovered unknowns and risks that could hinder our ability to meet the QASP and deliver working software. Details are in the pros/cons of each option's section.

[AdamUSDA]

@beechnut Thanks for the detailed description of what your working on. That really helps me to understand and to assess.

Let me ask a couple more questions.

1. Can you clarify how you see as the CI/CD process working at a high level. In a mature environment that can mean small and continuous builds to production. This may not be optimal for this project especially at the start when we are working through the process. Continuous testing is often implemented as a set of automated regression, performance, and other tests that are executed in the CI/CD pipeline. Can you clarify what test will be done in the pipeline and at what level GitHub Actions can track those issues? i.e High Level failure or down into the build process to the particular failure point.

2. Because this is a GitHub service I assume that they can generate a GitHub issue throughout the pipeline?

3. When you mentioned QASP I assume that your talking about code quality review and tools like SonarQube. If a build fails because of a poor code rating it would be helpful if the person that had checked in that particular piece of code was notified. This could be the creation of an issue in that sprint. This would allow them to resolve that issue in the sprint instead of adding more technical debt to the project. I'm not certain at what level GitHub Actions is integrated with SonarQube.

4. If a build fails because of a dependency issue or a security issue that has larger technical implications and would normally be escalated to the lead or primary developer to help resolve. They would help to remediate that issue with the developer. In this case an issue of high priority would be created by the GitHub Actions if that's possible. I'm not certain if the tool can identify a Maven build issue down to that level or how we will track security, performance or other types of build issues.

What I'm trying to get at is at each possible failure point we don't want to have something pass through the pipeline without some sort of tracking. Being able to identify and track all issues in the EAP would be very helpful. What I've seen in the past is that you have to look in multiple places to find issues and things can get lost in the minutiae. If those problems can be identified in the Sprint with an issue attached to it it would help from a reporting and management perspective.

[beechnut]

1. At a high level, CI/CD is what ensures that we continuously deliver working software from day one. It runs:

• static analysis: security, linting, code quality, style checks
• dependency checks (e.g. OWASP)
• tests: unit, integration, coverage, accessibility

Code is prevented from merging until all the checks pass. And generally speaking, if CI passes, deployable branches get deployed, especially development/staging (production may or may not involve more steps).

If a build fails because of a poor code rating it would be helpful if the person that had checked in that particular piece of code was notified.

100% agreed, and both GitHub Actions and Jenkins notify upon failure, and if we communicate back to the GitHub API, build statuses are quite public:

---

2. From a technical standpoint, because GitHub, Jira, and other issue trackers tend to have good API clients, I suspect it is relatively straightforward to set up automated issue creation regardless of platform (GH, Jenkins) or issue tracking software.

GitHub Actions, Jenkins, etc. don't automatically create issues, but they do notify upon failure. More on issue creation in (4) below.

---

3. When I referred to the QASP in the original post, I was thinking specifically about being able to perform isolated builds and isolated accessibility tests. With Matt Reiss' recent reassurance that Jenkins can perform those tests, this might become a non-issue.

---

4. Here's where I'm looking to gather information and build some understanding:

If a build fails because of a dependency issue or a security issue that has larger technical implications and would normally be escalated to the lead or primary developer to help resolve.

Opening an issue for things like this, that affect the entire project, makes sense to me: we could create a new ticket, or, if we have the right level of visibility, assign the right person to the existing ticket.

If a build fails because of a poor code rating it would be helpful if the person that had checked in that particular piece of code was notified. This could be the creation of an issue in that sprint.

The convention I'm most familiar with is that an issue (in GitHub, Jira, etc.) represents a unit of work that is only considered finished when it passes all the checks in the CI pipeline. Therefore, the developer is responsible for fixing any checks that didn't pass, such as a poor code quality rating, before the code can be accepted & merged. This would prevent that kind of technical debt, and prevent needing additional issues.

We currently use a review process like this: I can't merge my code until it passes all the CI checks and has been reviewed by someone else on the team.

Is this convention similar to or different from how you usually work? And I'm curious to know, what do you see as the benefits of opening a separate issue for CI failures? (Or perhaps, what do you lose if those issues are not opened?)

[beechnut]

Good news! Matt Reiss said that accessibility tests have been run as part of isolated builds for OpenForest, and is confident that we can containerize infrastructure to run tests. If this all checks out, I'm planning on updating the ADR to recommend using the Forest Service Jenkins. Following up with Matt Reiss to validate this ASAP.

Demos I'll be requesting:

• Demonstration of isolated pa11y build in OpenForest's Jenkins build
• Demonstration or initial setup of test runs within containerized infrastructure

[beechnut]

Alright, got a demo / explanation of both of these. pa11y's running against a localhost Angular app, and Docker Compose runs in Jenkins, so pretty much all we have to do is point them in the direction of setup instructions.

[beechnut]

After reviewing Jenkins' capabilities with Matt Reiss, @AdamUSDA, and others, I've changed the recommendation to propose using the Forest Service Jenkins instance. It comes with plenty of major positive consequences, and just a few minor negative (really, "potential negative") consequences, all laid out in the latest version of the ADR.

There's still a few open questions around:

• automatic issue creation/escalation,
• getting Jenkins to talk to the GitHub Checks API,
• and who the decision-makers are for this ADR.

Welcoming thoughts, questions, and other feedback.

[beechnut]

With the prioritization of #348, it seems as if a de-facto decision has been made to use Forest Service Jenkins.

@AdamUSDA and @jayrbergjr—do you approve this ADR? And are there any other people who should be involved in the approval?

[AdamUSDA]

Hey @beechnut ,

That was a very well thought out document. One of the best I've seen on this subject. There are several concerns that you raised that I would like to discuss.

"Jenkins is not set up to isolate application's configuration from one another—since Jenkins tasks are shared Forest Service-wide, others' work on Jenkins could introduce bugs into our workflow, and vice versa."

"Notifications are not as clear or directed (failures are emailed to the entire team, not the individual who checked in the code)."

I'm not a fan of exposing our builds to an audience outside of the specific application and DevOps team. You also know that I don't like for errors to be lost in the minutia of the Jenkins logs. Can you schedule some time for us to walk through this again. It should just be You, Neil, myself and Lorena.

[beechnut]

@AdamUSDA Thanks, and that sounds like a good idea. Just set up a time for us at 2 pm Eastern today.

[beechnut]

@AdamUSDA The sense I'm getting is that we need answers from the following questions before you're ready to approve this ADR, is that correct?

• Can Jenkins talk back to the GitHub Checks API to properly allow or block merging, based on whether the checks pass or fail?
• Can Jenkins build statues be sent to Teams, either directly Jenkins-->Teams or Jenkins-->GitHub Checks API-->Teams?

[AdamUSDA]

Before we present I would like to have this in the google document so that we can walk the Cloud Foundation team through our questions. I've asked a couple of other questions that we need to discuss before we present. i.e what will have happen in Test, and Production.

I would like to have a review of the serverless frameworks. We may present that a serverless framework will be used but not designate which one until we have completed our due diligence.

Please setup a time so that we can walk through the document and the questions. Before we present I need to approve what will be presented.

Thanks

Adam