From 07260d6596d06cb2f3674d72842d6b4f8c40cac2 Mon Sep 17 00:00:00 2001 From: Massimiliano Date: Tue, 9 Jan 2024 20:44:10 +0100 Subject: [PATCH] Created using advanced configs (infra) (#642) * Created using advanced configs Minor: new plugin to put todos in the docs * Fixed spelling * Minor: fixed typo * Implemented feedback from review * Apply suggestions from code review Co-authored-by: Matias Piipari * update format in reference * add config inheritance to explanation * add auto-retry how-to guide * add ui verbosity how-to guide * rework tutorial * add syntax highlighting * add spelling check exception --------- Co-authored-by: tang-mm <3333407+tang-mm@users.noreply.github.com> Co-authored-by: Matias Piipari Co-authored-by: MengT --- docs/.sphinx/spellingcheck.yaml | 3 +- docs/.sphinx/wordslist.txt | 1 + docs/explanation/configs.rst | 141 +++++- docs/how-to/index.rst | 2 + docs/how-to/launcher/auto-retry.rst | 62 +++ docs/how-to/launcher/output-verbosity.rst | 88 ++++ docs/reference/launcher.rst | 415 ++++++++---------- docs/tutorial/index.rst | 1 + .../using-checkbox/generate-report.rst | 307 +++++++++++++ docs/tutorial/using-checkbox/launcher.rst | 24 +- 10 files changed, 789 insertions(+), 255 deletions(-) create mode 100644 docs/how-to/launcher/auto-retry.rst create mode 100644 docs/how-to/launcher/output-verbosity.rst create mode 100644 docs/tutorial/using-checkbox/generate-report.rst diff --git a/docs/.sphinx/spellingcheck.yaml b/docs/.sphinx/spellingcheck.yaml index 229625addd..a93dd1ab63 100644 --- a/docs/.sphinx/spellingcheck.yaml +++ b/docs/.sphinx/spellingcheck.yaml @@ -19,9 +19,10 @@ matrix: - code - div.relatedlinks - div.visually-hidden + - div.code-block-caption - img - link - div.mermaid - pre - spellexception - - title \ No newline at end of file + - title diff --git a/docs/.sphinx/wordslist.txt b/docs/.sphinx/wordslist.txt index a7c4083379..91561434f9 100644 --- a/docs/.sphinx/wordslist.txt +++ b/docs/.sphinx/wordslist.txt @@ -35,6 +35,7 @@ github GitHub GPLv hostname +html http https INI diff --git a/docs/explanation/configs.rst b/docs/explanation/configs.rst index 76acbc3c0c..216bd31b33 100644 --- a/docs/explanation/configs.rst +++ b/docs/explanation/configs.rst @@ -3,10 +3,10 @@ Checkbox Configs ^^^^^^^^^^^^^^^^ -Configuration values resolution order -===================================== +Configuration file path +======================= -The directories that are searched for config files are: +By default, Checkbox searches for configuration files in the following directories: * ``/etc/xdg/`` * ``~/.config/`` @@ -26,14 +26,141 @@ using the ``config_filename`` variable from the ``[config]`` section (see :ref:`launcher_config` for more information). If it's not present, ``checkbox.conf`` is used. -Note that if same configuration variable is defined in more than one place, the -value resolution is as follows: + +Configuration values resolution order +===================================== + +If the same configuration variable is defined in more than one place, the order +of value resolution is as follows (from the highest to lowest priority): 1. launcher being invoked 2. config file from ``~/.config`` 3. config file from ``/etc/xdg`` 4. config file from ``$SNAP_DATA`` +If a configuration is specified in a launcher, values specified in other files +for the same configuration are overridden. + +For example, if the following config file is created at a custom location +``/tmp/my_config_name.conf``: + +.. code-block:: none + :caption: /tmp/my_config_name.conf + + [test plan] + unit = com.canonical.certification::smoke + forced = yes + + [test selection] + forced = yes + +And another config file at the one of the default lookup locations +``~/.config/checkbox.conf`` contains a duplicated value: + +.. code-block:: none + :caption: ~/.config/checkbox.conf + + [test plan] + unit = wrong_name + +Then invoke Checkbox with the following launcher: + +.. code-block:: none + :caption: my_launcher + :emphasize-lines: 2 + + [config] + config_filename = /tmp/my_config_name.conf + +Checkbox will load the correct test plan specified in the launcher. The ``unit`` +value in the default location is ignored. + + +Configuration inheritance +========================= + +To maintain a clean setup for different use cases, it is useful to define a +global configuration for Checkbox and a few smaller configurations that are +specific to each situation. You can use the ``config_filename`` option to bring +values from other configuration files into a config or a launcher. + +For example, the following config file contains some global configurations at +``~/.config/checkbox_global.conf``: + +.. code-block:: none + :caption: ~/.config/checkbox_global.conf + + [ui] + output = hide-automated + + [launcher] + session_title = My machine name + stock_reports = [text] + + [exporter:text] + unit = com.canonical.plainbox::text + + [transport:out_to_file] + type = file + path = /tmp/.last_checkbox_out.txt + + [report:screen] + exporter = text + transport = out_to_file + + [manifest] + com.canonical.certification::my_manifest_key = True + +If you invoke Checkbox with a launcher file that refers to this global config, +both configuration sources are taken into account: + +.. code-block:: none + :caption: my_launcher + :emphasize-lines: 2 + + [config] + config_filename = ~/.config/checkbox_global.conf + + [test plan] + unit = com.canonical.certification::smoke + force = True + + +If the same configuration option is defined in different sources, the value +defined in the importing file overrides the one from the imported config. + +For example, the following launcher configures the test report and submission, +where the ``stock_reports`` value overrides the imported value: + +.. code-block:: none + :caption: my_second_launcher + :emphasize-lines: 2, 9 + + [config] + config_filename = ~/.config/checkbox_global.conf + + [test plan] + unit = com.canonical.certification::smoke + forced = True + + [launcher] + stock_reports = [text, certification, submission_files] + local_submission = True + +The configuration value inheritance (when a config or a launcher imports +another config/launcher) allows every value to be inherited and +overridden. It is helpful to use the :ref:`'check-config' command ` to track +the origin of config values before running tests. + +.. warning:: + + Circular import is not allowed. We advise you to use this feature in + moderation since whilst it can simplify the maintenance of multiple + configurations by avoiding copy-pasting values around, it can also make + debugging a configuration complicated. + +.. _check_config_cmd: + Configuration checker ===================== @@ -78,7 +205,7 @@ placed in ``/home/user/.config/``: .. code-block:: none - [tset plan] + [test plan] filter = *wireless* [test selection] @@ -89,7 +216,7 @@ When running the ``check-config`` command, the following will be reported: .. code-block:: none Problems: - - Unexpected section [tset plan]. Origin: /home/user/.config/checkbox.conf + - Unexpected section [test plan]. Origin: /home/user/.config/checkbox.conf - Unexpected variable 'wrong_var' in section [test selection] Origin: /home/user/.config/checkbox.conf Indeed, there is a typo in the name of the ``[test plan]`` section, and diff --git a/docs/how-to/index.rst b/docs/how-to/index.rst index ad56614eac..237bd9d6ab 100644 --- a/docs/how-to/index.rst +++ b/docs/how-to/index.rst @@ -5,9 +5,11 @@ These how-to guides cover key operations and processes in Checkbox. .. toctree:: :maxdepth: 1 + :glob: testing-ubuntu-core custom-app nested-test-plan side-loading agent-service + launcher/* \ No newline at end of file diff --git a/docs/how-to/launcher/auto-retry.rst b/docs/how-to/launcher/auto-retry.rst new file mode 100644 index 0000000000..0905763eb2 --- /dev/null +++ b/docs/how-to/launcher/auto-retry.rst @@ -0,0 +1,62 @@ +Configuring auto-retry for failing tests +========================================== + +You can use launchers to configure Checkbox to automatically retry failing jobs. + +Enable auto-retry +------------------ + +To apply the auto-retry function to all failing test jobs, add a ``ui`` section +in your launcher and set the ``ui.auto-retry`` option to ``yes``. You can also +specify the maximum number of attempts and the delay between each retry. + +For example: + +.. code-block:: ini + :caption: my_launcher + :emphasize-lines: 9-10, 12 + + [test plan] + unit = com.canonical.certification::smoke + forced = yes + + [test selection] + forced = yes + + [ui] + auto_retry = yes + max_attempts = 2 + # the delay is in seconds + delay_before_retry = 2 + +After every test was executed, all failing tests were retried up to two times, +waiting 2 seconds between one attempt and the next. This may be useful if, for +example, a test relies on an external factor like WiFi access. + +For more details about the ``ui`` section in Checkbox launchers, see +:doc:`../../reference/launcher`. + +Skip auto-retry +---------------- + +When ``auto_retry`` is set to ``yes``, **every** failing job will be retried. +This can be a problem: for instance, for jobs that take a really long time +to run. To avoid this, you can use the ``auto-retry=no`` inline override +in the test plan to explicitly mark each job you do not wish to see +retried. + +For example: + +.. code-block:: yaml + :caption: my_test_plan.pxu + :emphasize-lines: 5 + + id: foo-bar-and-froz + _name: Tests Foo, Bar and Froz + include: + foo + bar auto-retry=no + froz + +In this case, even if the job ``bar`` fails and auto-retry is activated, it +will not be retried. diff --git a/docs/how-to/launcher/output-verbosity.rst b/docs/how-to/launcher/output-verbosity.rst new file mode 100644 index 0000000000..248599eceb --- /dev/null +++ b/docs/how-to/launcher/output-verbosity.rst @@ -0,0 +1,88 @@ +Changing output verbosity +=============================== + +When test are executed, by default, Checkbox prints errors and warnings of all +test jobs on the standard output. But sometimes you may want to know more on the +tests that are executing, or you may only care about the results. + +To customize the types of output information produced while running tests, you can +either apply verbosity options when you launch Checkbox, or change the UI output +options in a launcher: ``ui.output`` and ``ui.verbosity``. + +Hide output by job types +------------------------- + +You can hide output from resource jobs and automatic jobs by toggling the +``ui.output`` option. + +For example, when resource jobs are plenty, the standard output may fill up with their +output. In this case, consider the ``hide-resource-and-attachment`` option in the +following launcher: + +.. code-block:: ini + :emphasize-lines: 9 + + [test plan] + unit = com.canonical.certification::smoke + forced = yes + + [test selection] + forced = yes + + [ui] + output = hide-resource-and-attachment + +Similarly, to hide the standard output of automatic jobs, use the +``hide-automated`` option as in the following launcher: + +.. code-block:: ini + :emphasize-lines: 10 + + [test plan] + unit = com.canonical.certification::smoke + forced = yes + + [test selection] + forced = yes + + [ui] + # This also hides resource and attachments, they are automated as well! + output = hide-automated + +Change verbosity level +----------------------- + +By default, Checkbox only prints errors and warnings to the output. If you want to have more detailed information about Checkbox execution, run Checkbox tests with one of the verbosity levels: + +* verbose - report informational messages during execution, such as job starting +* debug - report all debug messages + +Using command options +~~~~~~~~~~~~~~~~~~~~~ + +When you invoke Checkbox, add either the ``--verbose`` or ``--debug`` option respectively. + +For example:: + + $ checkbox.checkbox-cli --debug launcher mylauncher + +Using launcher configurations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Add the ``verbosity`` option in the ``ui`` section of your launcher file. For example: + +.. code-block:: ini + :emphasize-lines: 10 + + [test plan] + unit = com.canonical.certification::smoke + forced = yes + + [test selection] + forced = yes + + [ui] + # Also, `debug` is available + verbosity = verbose + +For more information about the ``ui`` section, see :doc:`../../reference/launcher`. diff --git a/docs/reference/launcher.rst b/docs/reference/launcher.rst index c60db69a08..cf0971043c 100644 --- a/docs/reference/launcher.rst +++ b/docs/reference/launcher.rst @@ -24,36 +24,30 @@ External configuration files Launcher can specify external file(s) to load values from. ``[config]`` - -Beginning of the configuration section. + Beginning of the configuration section. ``config_filename`` + Name of the configuration file to look for. The value may be a file name or an absolute path, and may use environment variables. If no directory is specified, the default directories where the file will be searched for are ``/etc/xdg/`` and ``~/.config/``. -Name of the configuration file to look for. Default value: ``checkbox.conf`` + Default value: ``checkbox.conf`` -The directories that will be searched for the file are ``/etc/xdg/`` and -``~/.config/``. +Examples: -Example:: +.. code-block:: ini + # Looks for ``/etc/xdg/testing.conf`` and ``~/.config/testing.conf`` [config] config_filename = testing.conf -This will make checkbox look for ``/etc/xdg/testing.conf`` and -``~/config/testing.conf`` files. - -The ``config_filename`` may be an absolute path, and may use environment -variables - -Example:: - + # Use an environment variable [config] config_filename = $MYCONFIGS/testing.conf + # Use an absolute path [config] config_filename = /home/ubuntu/next-testing.conf -For more details about value resolution order see :doc:`configs <../explanation/configs>` +For more details about value resolution order, see :doc:`../explanation/configs`. Launcher meta-information ========================= @@ -62,76 +56,67 @@ Launcher meta-information helps to provide consistent checkbox behavior in the future. ``[launcher]`` - -Beginning of the launcher meta-information section. + Beginning of the launcher meta-information section. ``app_id`` - -This fields helps to differentiate between checkbox front-ends. This way -sessions started with launcher with one ``app_id`` won't interfere with -sessions started with a different launcher (provided it has ``app_id`` set to -other value). The app_id should be in a IQN form. Default value: -``com.canonical:checkbox-cli`` + This fields helps to differentiate between checkbox front-ends. This way + sessions started with launcher with one ``app_id`` won't interfere with + sessions started with a different launcher (provided it has ``app_id`` set to + other value). The app_id should be in a IQN form. Default value: + ``com.canonical:checkbox-cli`` ``app_version`` - -This field is purely informational. + This field is purely informational. ``launcher_version`` - -Version of the launcher language syntax and semantics to use. + Version of the launcher language syntax and semantics to use. ``api_flags`` - -API flags variable determines optional feature set. -List of API flags that this launcher requires. Items should be separated by -spaces or commas. The default value is an empty list. + API flags variable determines optional feature set. + List of API flags that this launcher requires. Items should be separated by + spaces or commas. The default value is an empty list. ``api_version`` - -API version determines the behavior of the launcher. Each checkbox feature is -added at a specific API version. Default behaviors don't change silently; -explicit launcher change is required. Default value: ``0.99`` + API version determines the behavior of the launcher. Each checkbox feature is + added at a specific API version. Default behaviors don't change silently; + explicit launcher change is required. Default value: ``0.99`` ``session_title`` - -A title to be applied to the sessions created using this launcher. This can be -be used to identify a stored sessions and can be used in report generation. + A title to be applied to the sessions created using this launcher. This can be + be used to identify a stored sessions and can be used in report generation. ``session_desc`` - -A string that can be applied to sessions created using this launcher. Useful -for storing some contextual information about the session. + A string that can be applied to sessions created using this launcher. Useful + for storing some contextual information about the session. ``stock_reports`` - -Stock reports are shortcuts in creating common reports. Instead of having to -specify exporter, transport and a report section in a launcher, you can use any -number of the stock ones. In launchers version 1 there are 4 stock reports you -may use: - -* ``text``: Print results as text on standard output -* ``submission_files``: Write ``html``, ``json`` and ``tar.xz`` - files to ``$XDG_DATA_HOME`` directory (or to ``~/.local/share/`` if - ``$XDG_DATA_HOME`` is not defined) -* ``certification``: Send results to certification site -* ``certification-staging``: Send results to staging version of - certification site - -If you don't want to have any stock report automatically generated use -``none`` as the value. - -This field is a list; use commas or spaces to separate stock reports. The -default value: ``text, certification, submission_files``. - -When using ``certification`` stock report, the ``secure_id`` variable may be -overridden by the launcher. -To do this define ``secure_id`` in a ``transport:c3`` section (this is the -transport that's used by the ``certification`` stock reports). + Stock reports are shortcuts in creating common reports. Instead of having to + specify exporter, transport and a report section in a launcher, you can use any + number of the stock ones. In launchers version 1 there are 4 stock reports you + may use: + + * ``text``: Print results as text on standard output + * ``submission_files``: Write ``html``, ``json`` and ``tar.xz`` + files to ``$XDG_DATA_HOME`` directory (or to ``~/.local/share/`` if + ``$XDG_DATA_HOME`` is not defined) + * ``certification``: Send results to certification site + * ``certification-staging``: Send results to staging version of + certification site + + If you don't want to have any stock report automatically generated use + ``none`` as the value. + + This field is a list; use commas or spaces to separate stock reports. The + default value: ``text, certification, submission_files``. + + When using ``certification`` stock report, the ``secure_id`` variable may be + overridden by the launcher. + To do this define ``secure_id`` in a ``transport:c3`` section (this is the + transport that's used by the ``certification`` stock reports). Launcher section example: -:: +.. code-block:: ini [launcher] app_id = com.foobar:system-testing @@ -142,14 +127,14 @@ Launcher section example: Launcher using all defaults with overridden secure_id: -:: +.. code-block:: ini [transport:c3] secure_id = 001122334455667788 Launcher that disables all stock reports: -:: +.. code-block:: ini [launcher] app_id = com.foobar:system-testing @@ -163,23 +148,19 @@ This section provides control over which test plans are visible in the menus and optionally forces the app to use particular one. ``[test plan]`` - -Beginning of the test plan section. + Beginning of the test plan section. ``unit`` - -An ID of a test plan that should be selected by default. By default nothing is -selected. + An ID of a test plan that should be selected by default. By default nothing is + selected. ``filter`` - -Glob that test plan IDs have to match in order to be visible. Default value: -``*`` + Glob that test plan IDs have to match in order to be visible. Default value: + ``*`` ``forced`` - -If set to ``yes``, test plan selection screen will be skipped. Requires -``unit`` field to be set. Default value: ``no``. + If set to ``yes``, test plan selection screen will be skipped. Requires + ``unit`` field to be set. Default value: ``no``. Test selection section @@ -187,25 +168,22 @@ Test selection section This section provides control over test selection. ``[test selection]`` - -Beginning of the test selection section + Beginning of the test selection section ``forced`` - -If set to ``yes``, test selection screen will be skipped and all test specified -in the test plan will be selected. Default value: ``no`` + If set to ``yes``, test selection screen will be skipped and all test specified + in the test plan will be selected. Default value: ``no`` ``exclude`` - -List of regex patterns that job ids will be matched against. The matched jobs -will be excluded from running in both stages of the session: bootstrapping and -normal stage. Note that if you specify a pattern that matches a resource job -that is used to instantiate template units those units won't get generated. The -patterns should be separated with whitespace. Examples: + List of regex patterns that job ids will be matched against. The matched jobs + will be excluded from running in both stages of the session: bootstrapping and + normal stage. Note that if you specify a pattern that matches a resource job + that is used to instantiate template units those units won't get generated. The + patterns should be separated with whitespace. Examples: Exclude all jobs containing 'bluetooth' in their id: -:: +.. code-block:: ini [test selection] exclude = .*bluetooth.* @@ -214,7 +192,7 @@ Exclude all jobs containing 'bluetooth' in their id: Exclude all jobs containing ``bluetooth`` in their id, or having ids starting with ``com.canonical.certification::dock/wireless``: -:: +.. code-block:: ini [test selection] exclude = .*bluetooth.* com.canonical.certification::dock/wireless.* @@ -239,140 +217,115 @@ User Interface section This section controls which type of UI to use. ``[ui]`` - -Beginning of the user interface section + Beginning of the user interface section ``type`` - -Type of UI to use. This has to be set to ``interactive``, ``silent``, -``converged``, or ``converged-silent``. - -``interactive`` runs the standard Checkbox command line version that prompts -user in non-automated tests. - -``silent`` skips the tests that would require human interaction. It's not -'silent' in the traditional command-line tool sense. - -Default value: ``interactive``. - -Note: the ``converged`` and ``converged-silent`` UI types will launch the QML -interface and requires checkbox-converged to be installed on your system. -Note: using ``silent`` or ``converged-silent`` UI types requires forcing -test selection and test plan selection. + Type of UI to use: + + * ``interactive`` runs the standard Checkbox command line version that prompts + user in non-automated tests. + * ``silent`` skips the tests that would require human interaction. This UI + type requires forcing test selection and test plan selection. It's not + 'silent' in the traditional command-line tool sense. + * ``converged`` launches the QML interface. It requires ``checkbox-converged`` + to be installed on your system. + * ``converged-silent`` launches the QML interface and skips the tests that + would require human interaction. It requires ``checkbox-converged`` to be + installed on your system. This UI type requires forcing test selection and + test plan selection. + + Default value: ``interactive``. ``dont_suppress_output`` + .. warning:: -.. warning:: + This field is deprecated, use 'output' to specify which jobs should have + their output printed to the screen. - This field is deprecated, use 'output' to specify which jobs should have - their output printed to the screen. - -Setting this field to ``yes`` disables hiding of command output for jobs of -type ``resource`` and ``attachment``. Default value: ``no``. + Setting this field to ``yes`` disables hiding of command output for jobs of + type ``resource`` and ``attachment``. Default value: ``no``. ``output`` + This setting lets you hide output of commands run by checkbox. It can be set to + one of the following values: -This setting lets you hide output of commands run by checkbox. It can be set to -one of the following values: - -- ``show`` - output of all jobs will be printed -- ``hide-resource-and-attachment`` - output of resource and attachment jobs - will be hidden, output of other job types will be printed -- ``hide-automated`` - output of shell jobs as well as attachment and resource - jobs will be hidden. Only interactive job command's output will be shown -- ``hide`` - same as ``hide-automated``. This value is deprecated, use - ``hide-automated`` + - ``show`` - output of all jobs will be printed + - ``hide-resource-and-attachment`` - output of resource and attachment jobs + will be hidden, output of other job types will be printed + - ``hide-automated`` - output of shell jobs as well as attachment and resource + jobs will be hidden. Only interactive job command's output will be shown + - ``hide`` - same as ``hide-automated``. This value is deprecated, use + ``hide-automated`` -Default value: ``show`` + Default value: ``show`` -.. note:: + .. note:: - Individual jobs can have their output hidden by specifying - 'suppress-output' in their definition. + Individual jobs can have their output hidden by specifying + 'suppress-output' in their definition. ``verbosity`` + This setting makes checkbox report more information from checkbox internals. + Possible values are: -This setting makes checkbox report more information from checkbox internals. -Possible values are: - -- ``normal`` - report only warnings and errors. -- ``verbose`` - report important events that take place during execution (E.g. - adding units, starting jobs, changing the state of the session) + - ``normal`` - report only warnings and errors. + - ``verbose`` - report important events that take place during execution (E.g. + adding units, starting jobs, changing the state of the session) + - ``debug`` - print out everything -- ``debug`` - print out everything + Default value: ``normal`` -Default value: ``normal`` + .. note:: -.. note:: - - You can also change this behavior when invoking Checkbox by using - ``--verbose`` and ``--debug`` options respectively. + You can also change this behavior when invoking Checkbox by using + ``--verbose`` and ``--debug`` options respectively. ``auto_retry`` + If set to ``yes``, all failed jobs will automatically be retried at the end of + the testing session. In addition, the re-run screen (where user can select + failed and skipped jobs to re-run) will not be shown. Default value: ``no``. + + .. note:: -If set to ``yes``, failed jobs will automatically be retried at the end of -the testing session. In addition, the re-run screen (where user can select -failed and skipped jobs to re-run) will not be shown. Default value: ``no``. + You can use ``auto-retry=no`` inline in the test plan to exclude a job + from auto-retrying. For more details, see :doc:`../how-to/launcher/auto-retry`. ``max_attempts`` -Defines the maximum number of times a job should be run in auto-retry mode. -If the job passes, it won't be retried even if the maximum number of attempts -have not been reached. Default value: ``3``. + Defines the maximum number of times a job should be run in auto-retry mode. + If the job passes, it won't be retried even if the maximum number of attempts + have not been reached. Default value: ``3``. ``delay_before_retry`` -The number of seconds to wait before retrying the failed jobs at the end of -the testing session. This can be useful when the jobs rely on external -factors (e.g. a WiFi access point) and you want to wait before retrying the -same job. Default value: ``1``. - -.. warning:: - - When ``auto_retry`` is set to ``yes``, **every** failing job will be retried. - This can be a problem, for instance, for jobs that take a really long time - to run. To avoid this, you can use the ``auto-retry=no`` inline override - in the test plan to explicitly mark each job you do not wish to see - retried. - - For example:: - - id: foo-bar-and-froz - _name: Tests Foo, Bar and Froz - include: - foo - bar auto-retry=no - froz - - In that case, even if job ``bar`` fails and auto-retry is activated, it - will not be retried. - + The number of seconds to wait before retrying the failed jobs at the end of + the testing session. This can be useful when the jobs rely on external + factors (e.g. a WiFi access point) and you want to wait before retrying the + same job. Default value: ``1``. + Restart section =============== This section enables fine control over how checkbox is restarted. ``[restart]`` - -Beginning of the restart section + Beginning of the restart section ``strategy`` - -Override the restart strategy that should be used. Currently supported -strategies are ``XDG`` and ``Snappy``. By default the best strategy is -determined at runtime. + Override the restart strategy that should be used. Currently supported + strategies are ``XDG`` and ``Snappy``. By default the best strategy is + determined at runtime. Environment section =================== ``[environment]`` + Beginning of the environment section -Beginning of the environment section - -Each variable present in the ``environment`` section will be present as -environment variable for all jobs run. + Each variable present in the ``environment`` section will be present as + environment variable for all jobs run. Example: -:: +.. code-block:: ini [environment] TESTING_HOST = 192.168.0.100 @@ -383,37 +336,34 @@ Daemon-specific configuration ============================= ``[agent]`` + .. warning:: + This section was previously called ``[daemon]``. This term has been + deprecated as of Checkbox 2.9 and is planned for removal. -.. warning:: - This section was previously called ``[daemon]``. This term has been - deprecated as of Checkbox 2.9 and is planned for removal. + Beginning of the agent-specific section. -Beginning of the agent-specific section. - -Settings in this section only apply to sessions that are run by :term:`Checkbox -Agent` spawned as Systemd service. + Settings in this section only apply to sessions that are run by :term:`Checkbox + Agent` spawned as Systemd service. ``normal_user`` + Username to use when job doesn't specify which user to run as. -Username to use when job doesn't specify which user to run as. - -The systemd service run on the :term:`agent` is run by root so in order to -run some jobs as an unprivileged user this variable can be used. + The systemd service run on the :term:`agent` is run by root so in order to + run some jobs as an unprivileged user this variable can be used. Manifest section ================ ``[manifest]`` + Beginning of the manifest section. -Beginning of the manifest section. - -Each variable present in the ``manifest`` section will be used as a preset value -for the system manifest, taking precedence over the disk cache. + Each variable present in the ``manifest`` section will be used as a preset value + for the system manifest, taking precedence over the disk cache. Example: -:: +.. code-block:: ini [manifest] com.canonical.certification::has_touchscreen = yes @@ -432,21 +382,21 @@ Exporter -------- ``[exporter:exporter_name]`` - -Beginning of an exporter declaration. Note that ``exporter_name`` should be -replaced with something meaningful, like ``html``. + Beginning of an exporter declaration. Note that ``exporter_name`` should be + replaced with something meaningful, like ``html``. ``unit`` - -ID of an exporter to use. To get the list of available exporters on your system -run ``$ checkbox-cli list exporter``. + ID of an exporter to use. To get the list of available exporters on your system + run ``$ checkbox-cli list exporter``. ``options`` + A list of options that will be supplied to the exporter. Items should be separated by + spaces or commas. -A list of options that will be supplied to the exporter. Items should be separated by -spaces or commas. -Example:: +Example: + +.. code-block:: ini [exporter:html] unit = com.canonical.plainbox::html @@ -455,13 +405,12 @@ Transport --------- ``[transport:transport_name]`` -Beginning of a transport declaration. Note that ``transport_name`` should be -replaced with something meaningful, like ``standard_out``. + Beginning of a transport declaration. Note that ``transport_name`` should be + replaced with something meaningful, like ``standard_out``. ``type`` - -Type of a transport to use. Allowed values are: ``stream``, ``file``, and -``certification``. + Type of a transport to use. Allowed values are: ``stream``, ``file``, and + ``submission-service``. Depending on the type of transport there might be additional fields. @@ -483,13 +432,13 @@ Depending on the type of transport there might be additional fields. +------------------------+---------------+----------------+----------------------+ | ``submission-service`` | ``secure-id`` | secure-id to | ``[transport:c3]`` | | | | use when | | -| | | uploading to | ``secure_id = 01``\ | -| | | certification | ``23456789ABCD`` | +| | | uploading to | ``type = sub`` \ | +| | | certification | ``mission-service`` | | | | sites | | -| | | | ``staging = yes`` | -| | | | | +| | | | ``secure_id = 01``\ | +| | | | ``23456789ABCD`` | | +---------------+----------------+ | -| | ``staging`` | determines if | | +| | ``staging`` | determines if | ``staging = yes`` | | | | staging site | | | | | should be used | | | | | Default: | | @@ -504,26 +453,22 @@ Report ------ ``[report:report_name]`` - -Beginning of a report declaration. Note that ``report_name`` should be -replaced with something meaningful, like ``to_screen``. + Beginning of a report declaration. Note that ``report_name`` should be + replaced with something meaningful, like ``to_screen``. ``exporter`` - -Name of the exporter to use + Name of the exporter to use ``transport`` - -Name of the transport to use + Name of the transport to use ``forced`` - -If set to ``yes`` will make checkbox always produce the report (skipping the -prompt). Default value: ``no``. + If set to ``yes`` will make checkbox always produce the report (skipping the + prompt). Default value: ``no``. Example of all three sections working to produce a report: -:: +.. code-block:: ini [exporter:text] unit = com.canonical.plainbox::text @@ -545,7 +490,7 @@ Launcher examples 'com.canonical.certification::smoke' test plan concluded by producing text report to standard output. -:: +.. code-block:: ini #!/usr/bin/env checkbox-cli @@ -578,7 +523,7 @@ report to standard output. 2) Interactive testing of FooBar project. Report should be uploaded to the staging version of certification site and saved to /tmp/submission.tar.xz -:: +.. code-block:: ini #!/usr/bin/env checkbox-cli @@ -620,7 +565,7 @@ customized environment configuration. The launcher -:: +.. code-block:: ini #!/usr/bin/env checkbox-cli [launcher] @@ -643,9 +588,9 @@ The launcher delay_before_retry = 15 -The launcher configuration ``laucher.conf`` +The launcher configuration ``launcher.conf`` -:: +.. code-block:: ini #!/usr/bin/env checkbox-cli [launcher] diff --git a/docs/tutorial/index.rst b/docs/tutorial/index.rst index 68c703a4b3..95a36ec6c8 100644 --- a/docs/tutorial/index.rst +++ b/docs/tutorial/index.rst @@ -35,6 +35,7 @@ journey through the product. using-checkbox/manifest using-checkbox/remote using-checkbox/advanced-commands + using-checkbox/generate-report Extended tutorial ----------------- diff --git a/docs/tutorial/using-checkbox/generate-report.rst b/docs/tutorial/using-checkbox/generate-report.rst new file mode 100644 index 0000000000..d62cd5a6c3 --- /dev/null +++ b/docs/tutorial/using-checkbox/generate-report.rst @@ -0,0 +1,307 @@ +.. _generate-report: + +============================ +Customize test output format +============================ + +Checkbox generates test reports after executing test plans. These reports +usually include a summary in HTML and other submission files. You can also +customize how each test report should be generated for different use cases. + +In Checkbox, the representation of a test report is controlled by two major +configurations in launcher: + +- ``exporter``: the format in which test report should be presented, such as + text, HTML, JSON and others +- ``transporter``: the output destination of each representation, such as the + standard output (``stdout``) or a file + +In this tutorial, you will learn to use launcher configurations to save test +reports in local file, produce output in different formats and generate multiple +reports for one single test run. + +For a more comprehensive overview about test reports, please refer to the +:ref:`test-report` section. + + +Save test reports to file +========================== + +When Checkbox completes executing all test jobs, a test summary is displayed in +the running console in the textual form. For example, the following text is a +summary displayed after running the ``tutorial-base`` test plan: + +.. code-block:: none + + ☑ : A job that always passes + ☒ : A job that always fails + ⚠ : A job that always crashes + ☑ : A job that depends on other job that passes + ☐ : A job that is skipped because it depends on a job that fails + ☑ : A job that generates different resources for tutorial purposes + ☑ : A job that requires a resource and it is available + ☐ : A job that requires a resource but it's not available + ☑ : A job that displays an environment variable, if set + ☑ : A manual job + ☑ : A semi-automated job where the outcome is set automatically + ☑ : A semi-automated job where the user manually sets the outcome + + +It may be tempting to redirect this output to file manually, but it is possible +to save it in a file using the Checkbox ``transport`` configuration. + +The following example shows how to define a ``transport`` section in a launcher +file. In the section header, the colon symbol (``:``) leads the name of the +transport (``out_to_file``). The value ``type = file`` specifies that the output +is directed to a file that is located at the given path. + +.. code-block:: ini + + [transport:out_to_file] + type = file + path = /tmp/output.txt + +Similarly, you can also direct the output to the standard output as a stream: + +.. code-block:: ini + + [transport:out_to_stdout] + type = stream + # standard out, you can also try "stderr" for standard error + stream = stdout + +However, the transport section is effective only if it is used in a ``report`` +section. If you invoke Checkbox with either of the above sections in a launcher +file without further edit, it will not generate any extra output. + +Now let's add a few more sections in the launcher to save the test summary to a +file. Try the following launcher: + +.. code-block:: ini + :caption: Save text report to file + :emphasize-lines: 9-10, 17-19 + + [test plan] + unit = com.canonical.certification::tutorial-base + forced = yes + + [test selection] + forced = yes + + # ":" delimits the name of the exporter + [exporter:text] + unit = com.canonical.plainbox::text + + [transport:out_to_file] + type = file + path = /tmp/output.txt + + # define a custom report + [report:file_report] + exporter = text + transport = out_to_file + +In this example, you defined two new sections: + +- ``exporter``: named ``text``, specifies that the output unit being used is + ``com.canonical.plainbox::text``. +- ``report``: named ``file_report``, specifies that a customized report + configuration is used. The customized report uses an exporter called ``text`` + and a transport called ``out_to_file``, which you defined in the same file. + +Launch Checkbox, and after the jobs are completed, you should see a new line in +the console output:: + + file:///tmp/output.txt + +Now you have a text report to check at ``/tmp/output.txt``. + +.. note:: + + Checkbox will ask you if you want to submit the ``file_report`` report. This is + a confirmation for producing the report. Respond yes. + See the example below to know how to avoid having to give confirmation + (using `forced`). + +Export report in different formats +=================================== + +If you want to process the test results in another application or visualize the +test report, you need to create representations other than the plain text form. + +In Checkbox, it is the ``exporter`` configuration that defines the form of +report output, including HTML, JSON, and other common formats. + +To view the supported types of exporters on your machine, run:: + + $ checkbox.checkbox-cli list exporter + +You might see a list similar to the following result: + +.. code-block:: none + + exporter 'com.canonical.plainbox::html' + exporter 'com.canonical.plainbox::html-multi-page' + exporter 'com.canonical.plainbox::json' + exporter 'com.canonical.plainbox::text' + exporter 'com.canonical.plainbox::tar' + exporter 'com.canonical.plainbox::xlsx' + exporter 'com.canonical.plainbox::global' + exporter 'com.canonical.plainbox::junit' + exporter 'com.canonical.plainbox::tp-export' + +Now let's configure Checkbox to generate a report in JSON for the same test +jobs. Create a new launcher: + +.. code-block:: ini + :caption: Save JSON report to file + :emphasize-lines: 8-9, 13, 16 + + [test plan] + unit = com.canonical.certification::tutorial-base + forced = yes + + [test selection] + forced = yes + + [exporter:json] + unit = com.canonical.plainbox::json + + [transport:out_to_file] + type = file + path = /tmp/output.json + + [report:file_report] + exporter = json + transport = out_to_file + # This tells Checkbox to always produce this report + # without asking any confirmation + forced = yes + +Run Checkbox again with the new launcher, a new file is generated at +``/tmp/output.json``. This JSON report contains much more detailed information +about the test job execution: + +.. code-block:: json + + { + "title": "session title", + "testplan_id": "com.canonical.certification::tutorial-base", + "custom_joblist": false, + "results": [ + { + "id": "tutorial/crashing", + "full_id": "com.canonical.certification::tutorial/crashing", + "name": "A job that always crashes", + "certification_status": "unspecified", + "category": "Tutorial", + "category_id": "com.canonical.certification::tutorial", + "status": "fail", + "outcome": "crash", + "comments": null, + "io_log": "This job crashes because we run a command to kill it before it's finished.\n", + "type": "test", + "project": "certification", + "duration": 0.23536920547485352, + "plugin": "shell" + }, + { + "id": "tutorial/failing", + "full_id": "com.canonical.certification::tutorial/failing", + "name": "A job that always fails", + "certification_status": "unspecified", + "category": "Tutorial", + "category_id": "com.canonical.certification::tutorial", + "status": "fail", + "outcome": "fail", + "comments": null, + "io_log": "This job fails!\n", + "type": "test", + "project": "certification", + "duration": 0.1324455738067627, + "plugin": "shell" + } + // ... + ] + } + + +Generate multiple reports +============================ + +You can configure multiple exporters in the same launcher for different use +cases. When a test session is completed, you will obtain multiple reports for +the same test results. + +Try the following launcher to produce HTML, JSON and textual reports for the +same test results: + +.. code-block:: ini + + [test plan] + unit = com.canonical.certification::tutorial-base + forced = yes + + [test selection] + forced = yes + + # exporter + [exporter:text] + unit = com.canonical.plainbox::text + + [exporter:json] + unit = com.canonical.plainbox::json + + [exporter:html] + unit = com.canonical.plainbox::html + + # transport + [transport:out_to_text] + type = file + path = ~/.last_result.txt + + [transport:out_to_json] + type = file + path = /tmp/upload.json + + [transport:out_to_html] + type = file + path = /tmp/upload.html + + # report + [report:test_report] + exporter = text + transport = out_to_text + forced = yes + + [report:json_report] + exporter = json + transport = out_to_json + forced = yes + + [report:html_report] + exporter = html + transport = out_to_html + forced = yes + + +Three files are generated when the test job are completed. Take a look at the +beautiful HTML report at the specified path:: + + file:///tmp/upload.html + file:///tmp/upload.json + file:///home/user/.last_result.txt + + +.. note:: + + If you start Checkbox with this launcher, remember that it will + create a file in ``~/.last_result.txt``. You may want + to remove it after this experiment. + +Wrapping up +=========== + +Congratulations! You now know how to customize your test reports for various use +cases. If you want to learn more about the configurations in a launcher, see the +:doc:`../../reference/launcher` reference document. diff --git a/docs/tutorial/using-checkbox/launcher.rst b/docs/tutorial/using-checkbox/launcher.rst index 5649ea5a56..d90e87e75f 100644 --- a/docs/tutorial/using-checkbox/launcher.rst +++ b/docs/tutorial/using-checkbox/launcher.rst @@ -22,7 +22,7 @@ filter them to only select the ones that matter to you. Create a file named ``mylauncher`` and add the following information in it: -.. code-block:: none +.. code-block:: ini :caption: mylauncher :name: initial-launcher @@ -34,7 +34,7 @@ Create a file named ``mylauncher`` and add the following information in it: Save it, then launch Checkbox using this launcher: -.. code-block:: none +.. code-block:: bash $ checkbox.checkbox-cli launcher mylauncher @@ -61,7 +61,7 @@ Select a test plan by default Set the filter back to ``*tutorial*`` in the ``[test plan]`` section and add the following line: -.. code-block:: none +.. code-block:: ini :caption: mylauncher :name: launcher-filter :emphasize-lines: 7 @@ -86,7 +86,7 @@ same test plan, and you always want to run all the tests in it. You don't want to spend time in the test plan selection screen nor the test selection screen. Modify your launcher so it looks like that: -.. code-block:: none +.. code-block:: ini :caption: mylauncher :name: launcher-forced-selection :emphasize-lines: 7-10 @@ -104,7 +104,7 @@ screen. Modify your launcher so it looks like that: Run Checkbox with this modified version of the launcher: -.. code-block:: none +.. code-block:: bash $ checkbox.checkbox-cli launcher mylauncher @@ -123,7 +123,7 @@ Customize test cases with environment variables One of the test cases in the Tutorial test plan uses the value set in an environment variable. Add the following lines in the launcher: -.. code-block:: none +.. code-block:: ini :caption: mylauncher :name: launcher-environment :emphasize-lines: 12-13 @@ -163,7 +163,7 @@ interested in the text summary and the test reports. Edit the launcher file: -.. code-block:: none +.. code-block:: ini :caption: mylauncher :name: launcher-stock-reports :emphasize-lines: 4 @@ -207,7 +207,7 @@ the launcher when running Checkbox. Create the file ``~/.config/checkbox.conf`` and add the following content in it: -.. code-block:: none +.. code-block:: ini [launcher] launcher_version = 1 @@ -226,7 +226,7 @@ in it: Now, run Checkbox without any argument: -.. code-block:: none +.. code-block:: bash $ checkbox.checkbox-cli @@ -332,7 +332,7 @@ file that can be interpreted, similarly to an executable bash script. At the top of the launcher file, add this line: -.. code-block:: none +.. code-block:: ini :caption: mylauncher :name: launcher-shebang :emphasize-lines: 1 @@ -356,13 +356,13 @@ At the top of the launcher file, add this line: Make the launcher executable: -.. code-block:: none +.. code-block:: bash $ chmod +x mylauncher Run it: -.. code-block:: none +.. code-block:: bash $ ./mylauncher