Fix preset yaml editing, use pyaml_env instead of yaml safe_load. Upd…

…ated Test documentation

Signed-off-by: Luigi Pellecchia <[email protected]>

api/api.py

@@ -11,7 +11,6 @@
 from uuid import uuid4
 
 import gitlab
-import yaml
 from flask import Flask, request, send_file, send_from_directory
 from flask_cors import CORS
 from flask_restful import Api, Resource, reqparse
@@ -6283,7 +6282,7 @@ def get(self):
 
         if os.path.exists(TESTRUN_PRESET_FILEPATH):
             try:
-                presets = parse_config(TESTRUN_PRESET_FILEPATH)
+                presets = parse_config(path=TESTRUN_PRESET_FILEPATH)
                 if plugin in presets.keys():
                     if isinstance(presets[plugin], list):
                         return [x["name"] for x in presets[plugin] if "name" in x.keys()]
@@ -6337,7 +6336,7 @@ def get(self):
             return UNAUTHORIZED_MESSAGE, UNAUTHORIZED_STATUS
 
         if preset:
-            presets = parse_config(TESTRUN_PRESET_FILEPATH)
+            presets = parse_config(path=TESTRUN_PRESET_FILEPATH)
 
             if plugin in presets.keys():
                 tmp = [x for x in presets[plugin] if x["name"] == preset]
@@ -6668,8 +6667,8 @@ def put(self):
 
         # Validate the content
         try:
-            new_content = yaml.safe_load(request_data["content"])  # noqa: F841
-        except yaml.YAMLError as exc:
+            new_content = parse_config(data=request_data["content"])  # noqa: F841
+        except Exception as exc:
             return f"{BAD_REQUEST_MESSAGE} {exc}", BAD_REQUEST_STATUS
 
         f = open(TESTRUN_PRESET_FILEPATH, 'w')

docs/source/test_run.rst

@@ -1,13 +1,50 @@
 .. image:: ../../app/src/app/bgimages/basil_black.svg
 
-Test Run and Test Results
-=========================
+Test Runs and Test Results
+==========================
+
+BASIL test infrastructure is based on the `tmt <https://tmt.readthedocs.io/en/stable/>`_ project.
+
+tmt is the acronym of Test Management Tool.
+
+It's a python module that provide an abstraction layer to be able to interact and execute any kind of test suites, regardless of the programming language they are implemented in.
+Moreover it provides provisioning features to be able to execute the desider tests against vaiours test environments such as container, virtual machines or physical hardwares.
+
+To summarize, if you want to run a test from BASIL you need to create an fmf file for your test that describes its entrypoint and dependencies in terms of required packages and libraries.
+
+Starting from version 1.5, BASIL is extending its testing capabilities to external infrastructures.
+Main reason behind that is that, sometimes, a company or an open source project introduces a tool like BASIL in the middle of the development.
+That means they already have their own test infrastructure configured and with high probability its a complex one.
+So, instead of asking for a migration effort, we can leverage existing test infrastrucutre and just link external test results to BASIL Test Cases to estabilish the required traceability matrix.
+
+What is actually supported?
+
+
++----------------+---------------------------+------------------+---------------------+
+|     Plugin     | Trigger Monitor and Trace | Filter and Trace | Test Infrastructure |
++================+===========================+==================+=====================+
+|      tmt       |           Yes             |       Yes        |      Embedded       |
++----------------+---------------------------+------------------+---------------------+
+|  Testing Farm  |           Yes             |       No         |      External       |
++----------------+---------------------------+------------------+---------------------+
+|   Gitlab CI    |           Yes             |       Yes        |      External       |
++----------------+---------------------------+------------------+---------------------+
+| Github Actions |           Yes             |       Yes        |      External       |
++----------------+---------------------------+------------------+---------------------+
+|    KernelCI    |            No             |       Yes        |      External       |
++----------------+---------------------------+------------------+---------------------+
+
+
+Trigger, Monitor and Trace: Trigger test execution on the external test infrastructure, follow the test execution and trace the test result to a BASIL Test Case.
+
+Filter and Trace: Navigate and filter test results executed on the external test infrastructure and trace them to a BASIL Test Case.
+
 
 ---
 tmt
 ---
 
-BASIL is using `tmt <https://tmt.readthedocs.io/en/stable/>`_ as Test Management Tool.
+BASIL internal test infrastructure is based on `tmt <https://tmt.readthedocs.io/en/stable/>`_ a Test Management Tool.
 
 **tmt** introduce a level of abstraction that allow BASIL to run any kind of test.
 
@@ -89,9 +126,27 @@ That will help you on scaling and modifying the structure of your test repositor
   :align: center
   :width: 100%
 
---------
+
+# BASIL tmt plan
+^^^^^^^^^^^^^^^^
+
+To run a test with **tmt** we need to specify a **tmt plan**.
+
+A **tmtm** plan is an **fmf** file that describes which tests we want to run, which preparation and finish steps we want to apply to the System Under Test, which report we want to generate and more.
+
+In a **tmt** plan we can filter tests from git repositories with different criteria under the **discover** statement.
+
+To let **tmt** able to discover any test in BASIL, we created a general purpose **tmt plan**.
+
+It is available at **BASIL/api/tmt-plan.fmf**.
+
+It is using environment variables to describe the git repository url, the branch and then relative path and those variable are passed by the test plugin BASIL/api/testrun_tmt.py
+
+If all your tests need to apply some preparation/finish steps you can extend this **BASIL/api/tmt-plan.fmf** accordingly.
+
+
 Test Run
---------
+^^^^^^^^
 
 You can run a test clicking on the "Run" button in the Test Case menu.
 A modal window will show up to let you able to configure the run.
@@ -105,7 +160,7 @@ You can specify a title and add your notes, mostly to be able to identify it lat
 Once you defined the test run title is time to specify the test run configuration.
 You can create a new one or re-use an existing one.
 
-BASIL support 2 different provisioning types:
+BASIL *tmt* plugin supports 2 different provisioning types:
 
  + Fedora Container
  + Remote machine via SSH
@@ -136,13 +191,13 @@ Pay attention that BASIL is automatically exporting some environment variables t
  + basil_test_run_config_title: Title of the Test Run Configuration
  + basil_user_email: Email of the User that created the Test Run
 
-Any other environment variable can be added by default in your BASIL instance customizing the api/tmttestrun.py file at TmtTestRunner() class initialization.
+Any other environment variable can be added by default in your BASIL instance customizing the api/testrun_tmt.py file at TestRunnerTmtPlugin() class initialization.
 
 
 # Test in Fedora Container
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Selecting **Fedora Container** as provisioning type you will trigger the execution against a that environment that runs as part of the BASIL deployment.
+Selecting **Fedora Container** as provisioning type you will trigger the execution against that environment that runs as part of the BASIL deployment.
 That is possible thanks to the package tmt[provision-container].
 
 .. image:: _static/_images/test_run_config_container.png
@@ -174,6 +229,182 @@ You can reach the SSH Keys management page clicking on the avatar icon on top ri
 
 Once ready, press **Run**.
 
+---------
+Gitlab CI
+---------
+
+You can use BASIL to run a test in your gitlab pipeline.
+On doing so you can specify a gitlab ci plugin preset as done in the following example:
+
+
+.. code-block:: yaml
+
+  gitlab_ci:
+    - name: lpellecc-ci-training
+      url: https://www.gitlab.com/
+      git_repo_ref: main
+      project_id: 64922856
+      trigger_token: !ENV ${GITLAB_CI_64922856_TRIGGER_TOKEN}
+      private_token: !ENV ${GITLAB_CI_64922856_PRIVATE_TOKEN}
+      stage: test
+      job: unit-test-job
+      env:
+        var1: value1
+        var2: value2
+
+Note: name, url, git_repo_ref, trigger_token, private_token are mandatory fields.
+      If you specify the `stage` statement, the plugin will check that all the jobs of that stage succeded.
+      If you specify the `stage` statement and `job` statement, the plugin will check that, the job you selected
+      is part of the selected stage and the job succeded.
+      Filtering external test results, updated_after and updated_before fields format is YYYY-MM-DD.
+
+You can use the same plugin preset configuration to navigate pipelines execution and trace a test run to your BASIL test case.
+On doing that you can apply additional filters.
+
+Click the Test Case menu and then Test Results. Oce the Test Results modal appears, select the tab `External Test Runs`.
+Then select the `gitlab ci` plugin and then the desired preset. Apply additional filter if needed and click `Add Filter`.
+Once you have all the filter in place, click `Search`.
+
+You can open the gitlab pipeline details page by clicking on the link provided on each line.
+Once you identified the right test run, you can click on `import` to trace it to your test case.
+
+
+--------------
+Githib Actions
+--------------
+
+You can use BASIL to run a test in your github pipeline.
+On doing so you can specify a github actions plugin preset as done in the following example:
+
+
+.. code-block:: yaml
+
+  github_actions:
+    - name: basil-test
+      url: https://www.github.com/elisa-tech/BASIL
+      git_repo_ref: main
+      private_token: !ENV ${GITHUB_ACTIONS_BASIL_PRIVATE_TOKEN}
+      workflow_id: build.yaml
+      job: test
+      inputs:
+        uuid: XXX-YYY-ZZZ
+
+Note: name, url, git_repo_ref, private_token, workflow_id are mandatory fields.
+      If you specify the `job` statement, the plugin will check that, the job you selected succeded.
+      Otherwise the plugin will check that all the jobs of the pipleine succeded.
+      If your pipeline need some inputs, you can add them under the `inputs` statement.
+      Pay attention that github only accept inputs configured in the repository. If you add input not supported the trigger can fail.
+
+To change the run name using the uuid variable from the inputs statement, you can configure your workflow as following:
+
+
+.. code-block:: yaml
+
+  name: ci
+  run-name: ci [${{ inputs.uuid && inputs.uuid || 'N/A' }}]
+
+  on:
+    push:
+    workflow_dispatch:
+      inputs:
+        uuid:
+          description: 'Unique ID'
+          required: false
+
+You can use the same plugin preset configuration to navigate pipelines execution and trace a test run to your BASIL test case.
+On doing that you can apply additional filters.
+
+Click the Test Case menu and then Test Results. Oce the Test Results modal appears, select the tab `External Test Runs`.
+Then select the `gitlab ci` plugin and then the desired preset. Apply additional filter if needed and click `Add Filter`.
+Once you have all the filter in place, click `Search`.
+
+You can open the github actions pipeline details page by clicking on the link provided on each line.
+Once you identified the right test run, you can click on `import` to trace it to your test case.
+
+
+--------
+KernelCI
+--------
+
+`KernelCI <https://kernelci.org/>`_ is a community-based open source distributed test automation system.
+It is focused on the upstream kernel development and the test execution is CI oriented.
+That means the tests are triggered on commit.
+
+From BASIL a user can navigate the KernelCI Database to be able to select the desired test run to link to a Test Case.
+
+On doing so you can specify a plugin preset in your BASIL instance as the following:
+
+
+.. code-block:: yaml
+
+  KernelCI:
+    - name: production
+      url: https://kernelci-api.westus3.cloudapp.azure.com/latest
+      private_token: !ENV ${KERNEL_CI_PROD_TOKEN}
+
+Note: name, url and private_token are mandatory fields.
+      created_after and created_before fields format is YYYY-MM-DD
+
+
+.. image:: _static/_images/testrun_KernelCI_filtering.png
+  :alt: BASIL Test Run Fail and Bug example
+  :align: center
+  :width: 100%
+
+
+------------
+Testing Farm
+------------
+Testing Farm is an open-source testing system offered as a service. It has been designed and is maintained by Red Hat.
+Testing Farm uses tmt as Test Management tool. That means that each tests should be described by an fmf metadata file and tests should be collected in tmt plans.
+
+To learn more about Testing Farm, refer to the official `Testing Farm documentation <https://docs.testing-farm.io/Testing%20Farm/0.1/index.html>`_.
+
+Useful link: `Testing Farm repository <https://gitlab.com/testing-farm>`_
+
+You can use BASIL to run a test against Testing Farm.
+On doing so you can specify a plugin preset in your BASIL instance as the following:
+
+
+.. code-block:: yaml
+
+  testing_farm:
+    - name: public_ranch
+      arch: x86_64
+      compose: Fedora-40
+      private_token: !ENV ${TESTING_FARM_API_TOKEN}
+      url: https://api.dev.testing-farm.io/v0.1/requests
+      context:
+        context_variable1: value1
+        context_variable2: value2
+      env:
+        environment_variable1: value1
+        environment_variable2: value2
+
+
+Note: name, arch, compose, private_token and url are mandatory fields.
+
+By default, BASIL will use the general purpose tmt plan avaiable at `api/tmt-plan.fmf`.
+This plan is using environment variables to specify the git repository, the branch and the tmt test name as following:
+
+.. code-block:: yaml
+
+  - discover+:
+    - name: General purpose discover
+      how: fmf
+      url: $basil_test_repo_url
+      ref: $basil_test_repo_ref
+      test:
+        - $basil_test_relative_path
+
+Those variables are automatically populated by BASIL plugin from the Test Case information as:
+
+`
+basil_test_repo_url = test_case.repository
+basil_test_relative_path = test_case.relative_path
+
+basil_test_repo_ref = From Preset or From Test Run Connfiguration as per user needs
+`
 
 -----------------
 Test Results
@@ -192,20 +423,19 @@ Here you can filter test results with a free text field that will search across
 
 Moreover from that view you can understand if a Bug was filed after a Test Run and you can Re-Run any previous job.
 
-Pay attention here that if a a Test Run was executed via SSH and you delete the SSH Key used at that time, BASIL will not be able to trigger the execution.
+Pay attention here that if a Test Run was executed via SSH and you delete the SSH Key used at that time, BASIL will not be able to trigger the execution via Re-Run.
 In that case from the Test Run view, you can Re-Use that particular test run configuration, changing the SSH Key with a fresh one and press 'Run'.
 
-To access Test Result information, you can click on the row of the desired one.
+To access the Test Result information, you can click the Details button on the right side of the desired row.
 
-That will load the data that you can navigate in all other tabs:
+You will be able to see:
 
  + Info (Details of the Test Run and of the Test Run Configuration)
- + execution log (Here you can see a log of the code that start tmt request, it is useful in case of exceptions)
- + log.txt (Verbose tmt log that completely describe the test execution)
- + stdout/stderr (stdout and stderr of the tmt request)
- + Report a Bug (to add/edit Bug reference and Notes, if bugs filed is populated that will show an icon in the Test Results table)
+ + Log
+ + Report a Bug /Fixes and edit Notes
  + Artifacts (where you see artifacts generated by test execution)
 
+
 # Bugs
 ^^^^^^
 
@@ -214,6 +444,7 @@ That will load the data that you can navigate in all other tabs:
   :align: center
   :width: 100%
 
+
 # Artifacts
 ^^^^^^^^^^^
 
@@ -227,21 +458,9 @@ So if you need to collect a particular evidence, you can copy it, during the tes
   :width: 100%
 
 
-----------------
-# BASIL tmt plan
-----------------
-
-To let **tmt** able to discover any test, BASIL implements a general purpose **tmt plan**.
-
-It is available at **BASIL/api/tmt-plan.fmf**.
-
-It is using environment variables to describe the git repository url, branch and relative path and those variable are passed by the test plugin BASIL/api/tmttestrun.py
-
-If all your tests need to apply some preparation/finish steps you can extend this tmt-plan.fmf accordingly.
-
 
 ------------
-# User Roles
+User Roles
 ------------
 
 Not all the users can request a test execution in BASIL and that depends on the user Role.