From 76edbb52c99ae519a42cb00a8ec8252743749aa3 Mon Sep 17 00:00:00 2001 From: MengT Date: Tue, 12 Dec 2023 14:28:28 +0800 Subject: [PATCH] move validation docs to explanation/ --- .../release_process/canary_pipeline.rst | 1 - .../validation_pipeline_execution.rst | 2 +- docs/how-to/index.rst | 1 - docs/how-to/release_process/canary.rst | 106 ----------------- .../release_process/canary_pipeline.rst | 44 ------- .../validation_job_example.rst | 66 ----------- .../validation_pipeline_execution.rst | 110 ------------------ 7 files changed, 1 insertion(+), 329 deletions(-) delete mode 100644 docs/how-to/release_process/canary.rst delete mode 100644 docs/how-to/release_process/canary_pipeline.rst delete mode 100644 docs/how-to/release_process/validation_job_example.rst delete mode 100644 docs/how-to/release_process/validation_pipeline_execution.rst diff --git a/docs/explanation/release_process/canary_pipeline.rst b/docs/explanation/release_process/canary_pipeline.rst index 7910e085bb..1afe999459 100644 --- a/docs/explanation/release_process/canary_pipeline.rst +++ b/docs/explanation/release_process/canary_pipeline.rst @@ -42,4 +42,3 @@ This is an example on how the whole pipeline can be implemented in Jenkins. } } } - diff --git a/docs/explanation/release_process/validation_pipeline_execution.rst b/docs/explanation/release_process/validation_pipeline_execution.rst index 688b959b84..ac8250c579 100644 --- a/docs/explanation/release_process/validation_pipeline_execution.rst +++ b/docs/explanation/release_process/validation_pipeline_execution.rst @@ -106,5 +106,5 @@ There are multiple entities participating in the chain of validating a Checkbox .. add code format to link text .. |hwcert-jenkins-jobs| replace:: ``hwcert-jenkins-jobs`` -.. _hwcert-jenkins-jobs: https://github.com/canonical/hwcert-jenkins-jobs +.. _hwcert-jenkins-jobs: https://github.com diff --git a/docs/how-to/index.rst b/docs/how-to/index.rst index b3f3d4811e..ad56614eac 100644 --- a/docs/how-to/index.rst +++ b/docs/how-to/index.rst @@ -11,4 +11,3 @@ These how-to guides cover key operations and processes in Checkbox. nested-test-plan side-loading agent-service - \ No newline at end of file diff --git a/docs/how-to/release_process/canary.rst b/docs/how-to/release_process/canary.rst deleted file mode 100644 index cdfc5dc3b8..0000000000 --- a/docs/how-to/release_process/canary.rst +++ /dev/null @@ -1,106 +0,0 @@ -Canary Testing for Checkbox Edge Version: In-depth Process -========================================================== - -Introduction ------------- - -Canary testing applies on Checkbox snaps that are released through the `edge channel `_ in the Snap Store. New versions are built daily if changes are made in the code repository. - -The following sections provide a detailed walk-through of the Canary Testing process for the Checkbox Edge version, starting from snap building to the outcomes of the testing. - -.. toctree:: - :maxdepth: 1 - :hidden: - - canary_pipeline - validation_pipeline_execution - validation_job_example - - -Snap Build and Release ------------------------ - -Trigger Conditions -^^^^^^^^^^^^^^^^^^ -The GitHub action responsible for building the snap runs every day. However, it only triggers when the following condition is met: - -At least one pull request (PR) has been merged since the last edge build. - -Build Workflow -^^^^^^^^^^^^^^^ - -To monitor the build process, or to review the configuration and logs, see `the GitHub workflow `_. - -Post-Build Actions -^^^^^^^^^^^^^^^^^^ - -Once the build is successful, the snap packages are automatically pushed to the Snap Store in the edge channel. Testers and early adopters can access the latest version through edge releases. - -Jenkins Monitoring and Validation ---------------------------------- - -Snap Monitoring -^^^^^^^^^^^^^^^ - -Once the snap is published to the edge channel in the Snap Store, our Jenkins job titled ``checkbox-edge-validation-detect-new-build`` gets into action. - -Monitoring job URL -^^^^^^^^^^^^^^^^^^ - -This Jenkins job monitors the Snap Store for the presence of the new snap using the following URL: - -``https://api.snapcraft.io/v2/snaps/find?q=checkbox22&channel=edge&fields=revision&architecture=amd64`` - -The specific JSON path that's being monitored for changes is: - -``$.results[0].revision.revision`` - -The job checks this path every minute for updates. - -Validation and Wait Logic -^^^^^^^^^^^^^^^^^^^^^^^^^ - -Upon detecting a new snap: - -1. The job verifies the presence of all other related checkbox snaps. -2. If any snap is missing, the job waits for an hour, periodically checking for its availability. -3. If the snaps are available within the waiting period, the next stage of testing is initiated. - -checkbox-edge-canary-validation Pipeline ----------------------------------------- - -Upon successful snap validation, the ``checkbox-edge-canary-validation`` pipeline begins its operation. - -The :doc:`canary_pipeline` contains the groovy script implementing the pipeline. - -Testing Platforms and Specifications -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - - -The pipeline concurrently runs 3 distinct jobs on different platforms: - -1. **amd64 ubuntu core 22**: Utilizing machines that consume Testflinger tasks from the ``dearest team`` queue. -2. **amd64 ubuntu core 16**: Also using machines from the ``dearest team`` queue. -3. **arm64 ubuntu core 22**: Targeting machines in the ``cert-rpi4b4g`` queue, which corresponds to the Raspberry Pi4 4GB model. - -The :doc:`validation_job_example` contains the Jenkins job definition for the amd64 ubuntu core 22 validation. - -For a detailed look on how the job execution is carried out by all the entities in the chain, -refer to the :doc:`validation_pipeline_execution`. - -Canary Test Plan Criteria -^^^^^^^^^^^^^^^^^^^^^^^^^ - -The canary test plan outlines specific tests that are imperative for the new snap's validation. The pipeline's successful conclusion is contingent upon all these tests passing on each of the mentioned platforms. - -Outcome and Commit Tagging ----------------------------- - -The outcome of the process is a tag that's being assigned to the commit from which the snaps were built. -The tag is either ``edge-validation-succeeded`` on success or ``edge-validation-failed`` otherwise. -The tag is pushed to the Checkbox repository on GitHub. - -Conclusion ----------- - -This Canary Testing process, complete from snap building to testing, ensures that every release of Checkbox in the edge channel is thoroughly vetted and stable. \ No newline at end of file diff --git a/docs/how-to/release_process/canary_pipeline.rst b/docs/how-to/release_process/canary_pipeline.rst deleted file mode 100644 index c4760d9065..0000000000 --- a/docs/how-to/release_process/canary_pipeline.rst +++ /dev/null @@ -1,44 +0,0 @@ -.. _canary_pipeline: - -Canary Pipeline -^^^^^^^^^^^^^^^ -This is an example on how the whole pipeline can be implemented in Jenkins. - -.. code-block:: groovy - - pipeline { - agent any - - stages { - stage('Run validation sub jobs') { - steps { - script { - parallel ( - "Checkbox series-22 for amd64": { - echo 'Running Canary on core22 amd64' - build job: 'checkbox-edge-validation-core22-amd64', wait: true, propagate: true - }, - "Checkbox series-22 for arm64": { - echo 'Running Canary on core22 arm64' - build job: 'checkbox-edge-validation-core22-arm64', wait: true, propagate: true - }, - "Checkbox series-16 for amd64": { - echo 'Running Canary on core16 amd64' - build job: 'checkbox-edge-validation-core16-amd64', wait: true, propagate: true - } - - ) - } - } - } - } - post { - always { - script { - def resultParam = currentBuild.resultIsBetterOrEqualTo('SUCCESS') ? 'edge-validation-succeeded' : 'edge-validation-failed' - // Trigger the tagging job - build job: 'checkbox-edge-validation-tag-and-finish', parameters: [string(name: 'RESULT', value: resultParam)] - } - } - } - } \ No newline at end of file diff --git a/docs/how-to/release_process/validation_job_example.rst b/docs/how-to/release_process/validation_job_example.rst deleted file mode 100644 index 0534151be2..0000000000 --- a/docs/how-to/release_process/validation_job_example.rst +++ /dev/null @@ -1,66 +0,0 @@ -.. _validation_job_example: - -Example of a validation job -^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Below is the shell code that can be used as a build step in Jenkins in a job -responsible for validating edge snap of Checkbox. - -.. code-block:: shell - - #!/bin/bash - set -e - set -x - - cat > job.yaml < JOB_ID - testflinger poll $JOB_ID - - TEST_STATUS=$(testflinger results $JOB_ID |jq -r .test_status) - - echo "Test exit status: $TEST_STATUS" - exit $EXITCODE diff --git a/docs/how-to/release_process/validation_pipeline_execution.rst b/docs/how-to/release_process/validation_pipeline_execution.rst deleted file mode 100644 index ac8250c579..0000000000 --- a/docs/how-to/release_process/validation_pipeline_execution.rst +++ /dev/null @@ -1,110 +0,0 @@ -Validation pipeline execution -============================= - -When validating various versions of Checkbox multiple workflows are in play. -Everything starts with automated creation of an Edge version. - -Creating new Edge version -------------------------- -Every day github checks for new commits that landed in the Checkbox repository, -and if anything new landed a new version is created. - - -.. mermaid:: - - graph TB - - A[GitHub Workflow: Detect New Commits] - - B[Run Metabox Tests on New Version] - C[Mark Version as Edge and Build Snaps] - D{Did Metabox Tests Pass?} - E{Did Snap Builds Succeed?} - F[Upload New Edge Version to Store] - - G[End: Build Failed - No New Edge Version] - - - A --> B - B --> D - D -->|Yes| C - C --> E - E -->|Yes| F - D -->|No| G - E -->|No| G - - -With new Edge version of Checkbox in the store we can start validating it. - -Validating the Edge version ---------------------------- - -On the Certification Jenkins instance, the ``{insert_final_job_name_here}`` job checks the store API for new Edge versions of Checkbox. -The job is defined in the |hwcert-jenkins-jobs|_ repository. - -This job is also responsible for checking if all of the necessary snaps were published (for other series and architectures). -Once confirmed, the "Canary Test Plan" is defined in `Canary test plan `_. - -.. mermaid:: - - graph TB - - A[Detect New Edge Version in Store] - B{Check if All Necessary Snaps are Published} - C{Run Canary Test Plan on Devices} - D[Tag as `edge-validation-failed``] - E[Tag as `edge-validation-completed``] - - A --> B - B -->|No| D - B -->|Yes| C - C -->|Yes| E - C -->|No| D - - - - -There are multiple entities participating in the chain of validating a Checkbox snap. - -.. mermaid:: - - sequenceDiagram - - participant Pipeline as Checkbox Validation Pipeline - participant Jenkins as Jenkins Job - participant TServer as Testflinger Server - participant TAgent as Testflinger Agent - participant Docker as Docker container (Checkbox Controller) - participant Device as Device Under Test - participant CAgent as Checkbox Agent - note over Device,CAgent: Same device - - Pipeline->>Jenkins: Trigger Jenkins Job - activate Pipeline - Jenkins->>TServer: Submit Testing Job - activate TServer - loop Poll for Job - TAgent-->>TServer: Check for Available Jobs - end - activate TAgent - TAgent->>Device: Provision Device & Start Checkbox Agent - activate Device - activate CAgent - TAgent->>Docker: Run Checkbox Controller - activate Docker - Docker->>CAgent: Start Canary Test Plan - CAgent-->>Docker: Return Test Results - deactivate CAgent - Docker-->>TAgent: Report Results - deactivate Docker - TAgent-->>TServer: Job Completion Status - deactivate TAgent - TServer-->>Jenkins: Inform Jenkins of Outcome - deactivate TServer - Jenkins-->>Pipeline: Update Pipeline with Job Outcome - deactivate Pipeline - -.. add code format to link text -.. |hwcert-jenkins-jobs| replace:: ``hwcert-jenkins-jobs`` -.. _hwcert-jenkins-jobs: https://github.com -