Created using advanced configs (infra) (#642)

* Created using advanced configs

Minor: new plugin to put todos in the docs

* Fixed spelling

* Minor: fixed typo

* Implemented feedback from review

* Apply suggestions from code review

Co-authored-by: Matias Piipari <[email protected]>

* update format in reference

* add config inheritance to explanation

* add auto-retry how-to guide

* add ui verbosity how-to guide

* rework tutorial

* add syntax highlighting

* add spelling check exception

---------

Co-authored-by: tang-mm <[email protected]>
Co-authored-by: Matias Piipari <[email protected]>
Co-authored-by: MengT <[email protected]>

docs/.sphinx/spellingcheck.yaml

@@ -19,9 +19,10 @@ matrix:
       - code
       - div.relatedlinks
       - div.visually-hidden
+      - div.code-block-caption
       - img
       - link
       - div.mermaid
       - pre
       - spellexception
-      - title
+      - title

docs/.sphinx/wordslist.txt

@@ -35,6 +35,7 @@ github
 GitHub
 GPLv
 hostname
+html
 http
 https
 INI

docs/explanation/configs.rst

@@ -3,10 +3,10 @@
 Checkbox Configs
 ^^^^^^^^^^^^^^^^
 
-Configuration values resolution order
-=====================================
+Configuration file path
+=======================
 
-The directories that are searched for config files are:
+By default, Checkbox searches for configuration files in the following directories:
 
 * ``/etc/xdg/``
 * ``~/.config/``
@@ -26,14 +26,141 @@ using the ``config_filename`` variable from the ``[config]`` section (see
 :ref:`launcher_config` for more information). If it's not present,
 ``checkbox.conf`` is used.
 
-Note that if same configuration variable is defined in more than one place, the
-value resolution is as follows:
+
+Configuration values resolution order
+=====================================
+
+If the same configuration variable is defined in more than one place, the order
+of value resolution is as follows (from the highest to lowest priority):
 
 1. launcher being invoked
 2. config file from ``~/.config``
 3. config file from ``/etc/xdg``
 4. config file from ``$SNAP_DATA``
 
+If a configuration is specified in a launcher, values specified in other files
+for the same configuration are overridden.
+
+For example, if the following config file is created at a custom location
+``/tmp/my_config_name.conf``:
+
+.. code-block:: none
+   :caption: /tmp/my_config_name.conf
+
+   [test plan] 
+   unit = com.canonical.certification::smoke 
+   forced = yes
+
+   [test selection] 
+   forced = yes
+
+And another config file at the one of the default lookup locations
+``~/.config/checkbox.conf`` contains a duplicated value:
+
+.. code-block:: none
+   :caption: ~/.config/checkbox.conf
+
+   [test plan] 
+   unit = wrong_name
+
+Then invoke Checkbox with the following launcher:
+
+.. code-block:: none
+   :caption: my_launcher
+   :emphasize-lines: 2
+
+   [config] 
+   config_filename = /tmp/my_config_name.conf
+
+Checkbox will load the correct test plan specified in the launcher. The ``unit``
+value in the default location is ignored.
+
+
+Configuration inheritance
+=========================
+
+To maintain a clean setup for different use cases, it is useful to define a
+global configuration for Checkbox and a few smaller configurations that are
+specific to each situation. You can use the ``config_filename`` option to bring
+values from other configuration files into a config or a launcher.
+
+For example, the following config file contains some global configurations at
+``~/.config/checkbox_global.conf``:
+
+.. code-block:: none
+  :caption: ~/.config/checkbox_global.conf
+
+  [ui]
+  output = hide-automated
+
+  [launcher]
+  session_title = My machine name
+  stock_reports = [text]
+
+  [exporter:text]
+  unit = com.canonical.plainbox::text
+
+  [transport:out_to_file]
+  type = file
+  path = /tmp/.last_checkbox_out.txt
+
+  [report:screen]
+  exporter = text
+  transport = out_to_file
+
+  [manifest]
+  com.canonical.certification::my_manifest_key = True
+
+If you invoke Checkbox with a launcher file that refers to this global config,
+both configuration sources are taken into account:
+
+.. code-block:: none
+   :caption: my_launcher
+   :emphasize-lines: 2
+ 
+   [config]
+   config_filename = ~/.config/checkbox_global.conf
+
+   [test plan]
+   unit = com.canonical.certification::smoke
+   force = True
+
+
+If the same configuration option is defined in different sources, the value
+defined in the importing file overrides the one from the imported config.
+
+For example, the following launcher configures the test report and submission,
+where the ``stock_reports`` value overrides the imported value:
+
+.. code-block:: none
+   :caption: my_second_launcher
+   :emphasize-lines: 2, 9
+
+   [config]
+   config_filename = ~/.config/checkbox_global.conf
+
+   [test plan]
+   unit = com.canonical.certification::smoke
+   forced = True
+
+   [launcher]
+   stock_reports = [text, certification, submission_files]
+   local_submission = True
+
+The configuration value inheritance (when a config or a launcher imports
+another config/launcher) allows every value to be inherited and
+overridden. It is helpful to use the :ref:`'check-config' command <check_config_cmd>` to track 
+the origin of config values before running tests.
+
+.. warning::
+
+   Circular import is not allowed. We advise you to use this feature in
+   moderation since whilst it can simplify the maintenance of multiple
+   configurations by avoiding copy-pasting values around, it can also make
+   debugging a configuration complicated. 
+
+.. _check_config_cmd:
+
 Configuration checker
 =====================
 
@@ -78,7 +205,7 @@ placed in ``/home/user/.config/``:
 
 .. code-block:: none
 
-    [tset plan]
+    [test plan]
     filter = *wireless*
 
     [test selection]
@@ -89,7 +216,7 @@ When running the ``check-config`` command, the following will be reported:
 .. code-block:: none
 
     Problems:
-    -  Unexpected section [tset plan]. Origin: /home/user/.config/checkbox.conf
+    -  Unexpected section [test plan]. Origin: /home/user/.config/checkbox.conf
     -  Unexpected variable 'wrong_var' in section [test selection] Origin: /home/user/.config/checkbox.conf
 
 Indeed, there is a typo in the name of the ``[test plan]`` section, and

docs/how-to/index.rst

@@ -5,9 +5,11 @@ These how-to guides cover key operations and processes in Checkbox.
 
 .. toctree::
    :maxdepth: 1
+   :glob:
 
    testing-ubuntu-core
    custom-app
    nested-test-plan
    side-loading
    agent-service
+   launcher/*

docs/how-to/launcher/auto-retry.rst

@@ -0,0 +1,62 @@
+Configuring auto-retry for failing tests
+==========================================
+
+You can use launchers to configure Checkbox to automatically retry failing jobs.
+
+Enable auto-retry
+------------------
+
+To apply the auto-retry function to all failing test jobs, add a ``ui`` section
+in your launcher and set the ``ui.auto-retry`` option to ``yes``. You can also
+specify the maximum number of attempts and the delay between each retry.
+
+For example:
+
+.. code-block:: ini
+  :caption: my_launcher
+  :emphasize-lines: 9-10, 12
+
+  [test plan]
+  unit = com.canonical.certification::smoke
+  forced = yes
+
+  [test selection]
+  forced = yes
+
+  [ui]
+  auto_retry = yes
+  max_attempts = 2
+  # the delay is in seconds
+  delay_before_retry = 2
+
+After every test was executed, all failing tests were retried up to two times,
+waiting 2 seconds between one attempt and the next. This may be useful if, for
+example, a test relies on an external factor like WiFi access.
+
+For more details about the ``ui`` section in Checkbox launchers, see
+:doc:`../../reference/launcher`.
+
+Skip auto-retry
+---------------- 
+
+When ``auto_retry`` is set to ``yes``, **every** failing job will be retried.
+This can be a problem: for instance, for jobs that take a really long time
+to run. To avoid this, you can use the ``auto-retry=no`` inline override
+in the test plan to explicitly mark each job you do not wish to see
+retried.
+
+For example:
+
+.. code-block:: yaml
+  :caption: my_test_plan.pxu
+  :emphasize-lines: 5
+
+  id: foo-bar-and-froz
+  _name: Tests Foo, Bar and Froz
+  include:
+    foo
+    bar     auto-retry=no
+    froz
+
+In this case, even if the job ``bar`` fails and auto-retry is activated, it
+will not be retried.

docs/how-to/launcher/output-verbosity.rst

@@ -0,0 +1,88 @@
+Changing output verbosity
+===============================
+
+When test are executed, by default, Checkbox prints errors and warnings of all
+test jobs on the standard output. But sometimes you may want to know more on the
+tests that are executing, or you may only care about the results. 
+
+To customize the types of output information produced while running tests, you can
+either apply verbosity options when you launch Checkbox, or change the UI output 
+options in a launcher: ``ui.output`` and ``ui.verbosity``.
+
+Hide output by job types
+-------------------------
+
+You can hide output from resource jobs and automatic jobs by toggling the
+``ui.output`` option. 
+
+For example, when resource jobs are plenty, the standard output may fill up with their
+output. In this case, consider the ``hide-resource-and-attachment`` option in the 
+following launcher: 
+
+.. code-block:: ini
+  :emphasize-lines: 9
+
+  [test plan]
+  unit = com.canonical.certification::smoke
+  forced = yes
+
+  [test selection]
+  forced = yes
+
+  [ui]
+  output = hide-resource-and-attachment
+
+Similarly, to hide the standard output of automatic jobs, use the
+``hide-automated`` option as in the following launcher:
+
+.. code-block:: ini
+  :emphasize-lines: 10
+
+  [test plan]
+  unit = com.canonical.certification::smoke
+  forced = yes
+
+  [test selection]
+  forced = yes
+
+  [ui]
+  # This also hides resource and attachments, they are automated as well!
+  output = hide-automated
+
+Change verbosity level
+-----------------------
+
+By default, Checkbox only prints errors and warnings to the output. If you want to have more detailed information about Checkbox execution, run Checkbox tests with one of the verbosity levels:
+
+* verbose - report informational messages during execution, such as job starting
+* debug - report all debug messages
+
+Using command options
+~~~~~~~~~~~~~~~~~~~~~
+
+When you invoke Checkbox, add either the ``--verbose`` or ``--debug`` option respectively.
+
+For example::
+
+  $ checkbox.checkbox-cli --debug launcher mylauncher
+
+Using launcher configurations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Add the ``verbosity`` option in the ``ui`` section of your launcher file. For example:
+
+.. code-block:: ini
+  :emphasize-lines: 10
+
+  [test plan]
+  unit = com.canonical.certification::smoke
+  forced = yes
+
+  [test selection]
+  forced = yes
+
+  [ui]
+  # Also, `debug` is available
+  verbosity = verbose
+
+For more information about the ``ui`` section, see :doc:`../../reference/launcher`.