Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Experimental command to fetch results from the Dashboard API #60

Merged
merged 6 commits into from
Dec 13, 2024
Merged

Experimental command to fetch results from the Dashboard API #60

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
5908819
cmds: rename 'results' cmd to 'maestro-results'
padovan Dec 6, 2024
64a1cfa
common: add click.echo wrappers
padovan Dec 10, 2024
721542c
cmds: add experimental results fetch from Dashboard API
padovan Dec 9, 2024
2cc540a
docs: refactor to place Maestro commands in their corner
padovan Dec 10, 2024
ee1df43
docs: Clarify install and usage for a wider audience
padovan Dec 10, 2024
84668d9
docs: add results cmd documentation.
padovan Dec 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

> *cmdline tool for interact with KernelCI*

kci-dev is a cmdline tool for interact with a enabled KernelCI server
Stand alone tool for Linux Kernel developers and maintainers to interact with KernelCI services.

## Quickstart

Expand Down
54 changes: 41 additions & 13 deletions docs/_index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,29 +4,51 @@ date = 2024-01-14T07:07:07+01:00
description = 'Tool for interact programmatically with KernelCI instances.'
+++

kci-dev is a cmdline tool for interact with a enabled KernelCI server.
Purpose of this tool to provide a easy way to use features of KernelCI Pipeline instance.
Stand alone tool for Linux Kernel developers and maintainers to interact with KernelCI.

Purpose of this tool to provide an easy-to-use command line tool for developers and maintainers request test from KernelCI, view results, download logs, integrate with scripts, etc.

## Installation

### Using PyPI and virtualenv
You may want to use python virtual environment.
If you are not familiar with it, check [this](https://docs.python.org/3/library/venv.html).

To quickly setup it:

```sh
virtualenv .venv
source .venv/bin/activate
```

### Using package from PyPI

Simply install it using `pip`:

```sh
pip install kci-dev
```

### Using poetry and virtualenv
### Development snapshot through poetry

Clone the `kci-dev` repo you want, select the desired branch and run:

```sh
virtualenv .venv
source .venv/bin/activate
pip install poetry
poetry install
poetry run kci-dev
```

Then, to execute kci-dev:

```sh
poetry run kci-dev <options>
```

## Configuration

> Configuration is only necessary if you are using any of the Maestro Commands listed in the Maestro section.

kci-dev searches for and loads a configuration file in the following order of priority:
1) The global configuration file located at /etc/kci-dev.toml.
2) The user-specific configuration file at ~/.config/kci-dev/kci-dev.toml
Expand Down Expand Up @@ -61,17 +83,17 @@ pipeline is the URL of the KernelCI Pipeline API endpoint, api is the URL of the
If you are using KernelCI Pipeline instance, you can get the token from the project maintainers.
If it is a local instance, you can generate your token using [kernelci-pipeline/tools/jwt_generator.py](https://github.com/kernelci/kernelci-pipeline/blob/main/tools/jwt_generator.py) script.

## Options
### Configuration options

### instance
#### --instance
You can provide the instance name to use for the command.

Example:
```sh
kci-dev --instance staging
```

### settings
#### --settings

You can provide the configuration file path to use for the command.

Expand All @@ -80,17 +102,23 @@ Example:
kci-dev --settings /path/to/.kci-dev.toml
```

## Commands
### General Commands

#### results

Pull results from the Dashboard. See detailed [documentation](results).

### Maestro Commands

### checkout
#### checkout

- [checkout](checkout)

### testretry
#### testretry

- [testretry](testretry)

### results
#### maestro-results

- [results](results)
- [maestro-results](maestro-results)

115 changes: 115 additions & 0 deletions docs/maestro-results.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
+++
title = 'maestro-results'
date = 2024-01-14T07:07:07+01:00
description = 'Command for show Maestro test results.'
+++

This command is Maestro-specific and will show the test result by node id.

Do not use it, unless you are requesting test on Maestro through `kci-dev`.

Example:
```sh
kci-dev maestro-results --nodeid <str: testnodeid>
```

This command will show the results of tests by page nodes limit and page offset.

Example:
```sh
kci-dev maestro-results --nodes --limit <int: page nodes limit> --offset <int: page nodes offset>
```

Result sample:
```yaml
{'artifacts': None,
'created': '2024-10-04T00:49:15.691000',
'data': {'arch': 'x86_64',
'compiler': 'gcc-12',
'config_full': 'x86_64_defconfig+lab-setup+x86-board',
'defconfig': 'x86_64_defconfig',
'device': None,
'error_code': None,
'error_msg': None,
'job_context': None,
'job_id': None,
'kernel_revision': {'branch': 'staging-mainline',
'commit': '232edaea0fd9b4b7feb7b43508834bba7e820584',
'commit_message': 'staging-mainline-20241004.0',
'commit_tags': ['staging-mainline-20241004.0'],
'describe': 'staging-mainline-20241004.0',
'patchset': None,
'tip_of_branch': False,
'tree': 'kernelci',
'url': 'https://github.com/kernelci/linux.git',
'version': {'extra': '-rc1-115-g232edaea0fd9b',
'name': None,
'patchlevel': 12,
'sublevel': None,
'version': 6}},
'kernel_type': 'bzimage',
'platform': 'hp-14b-na0052xx-zork',
'regression': None,
'runtime': 'lava-collabora',
'test_revision': None,
'test_source': None},
'debug': None,
'group': 'kselftest-exec',
'holdoff': None,
'id': '66ff3b8c0abcc4c8343d1c71',
'jobfilter': None,
'kind': 'test',
'name': 'exec_execveat_Check_success_of_execveat_20_4096',
'owner': 'staging.kernelci.org',
'parent': '66ff3b8b0abcc4c8343d1b8a',
'path': ['checkout',
'kbuild-gcc-12-x86',
'kselftest-exec',
'kselftest-exec',
'exec_execveat_Check_success_of_execveat_20_4096'],
'result': 'pass',
'state': 'done',
'submitter': 'service:pipeline',
'timeout': '2024-10-04T06:49:15.691000',
'treeid': 'a44035dadc31327a5c30db4013b0e7e90acbb6a8fc45f94a6d91671e76cdfd8a',
'updated': '2024-10-04T00:49:15.691000',
'user_groups': []}
```

testnodeid is the node id of the test job, which you can get from the KernelCI dashboard.
Usually it is hexadecimal string.

Additionally, for --nodes you can provide optional parameters --filter to filter the results by the given key and value.
For example:
```sh
./kci-dev.py results --nodes --filter treeid=e25266f77837de335edba3c1b8d2a04edc2bfb195b77c44711d81ebea4494140 --filter kind=test
```
This command will show the nodes of tests in particular tree checkout.
But as you might see, there is a lot of fields you might be not interested in.

For this we have additional option --field, that will restrict output only to specified fields.
For example:
```sh
./kci-dev.py results --nodes --filter treeid=e25266f77837de335edba3c1b8d2a04edc2bfb195b77c44711d81ebea4494140 --filter kind=test --field name --field result
```
Example:

```json
{'name': 'kver', 'result': 'pass'}
{'name': 'config', 'result': 'pass'}
{'name': 'build', 'result': 'pass'}
{'name': 'example_init_test', 'result': 'pass'}
{'name': 'time64_to_tm_test_date_range', 'result': 'pass'}
{'name': 'test_one_cpu', 'result': 'skip'}
{'name': 'test_many_cpus', 'result': 'skip'}
{'name': 'test_one_task_on_all_cpus', 'result': 'skip'}
{'name': 'test_two_tasks_on_all_cpus', 'result': 'skip'}
{'name': 'test_one_task_on_one_cpu', 'result': 'skip'}
{'name': 'test_one_task_mixed', 'result': 'skip'}
{'name': 'test_two_tasks_on_one_cpu', 'result': 'skip'}
{'name': 'test_two_tasks_on_one_all_cpus', 'result': 'skip'}
{'name': 'test_task_on_all_and_one_cpu', 'result': 'skip'}
{'name': 'resource_test_union', 'result': 'pass'}
{'name': 'resource_test_intersection', 'result': 'pass'}
....
```
139 changes: 46 additions & 93 deletions docs/results.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,113 +1,66 @@
+++
title = 'results'
date = 2024-01-14T07:07:07+01:00
description = 'Command for show test results.'
date = 2024-12-10T07:07:07+01:00
description = 'Fetch results from the KernelCI ecosystem.'
+++

This command will show the test result by node id.
`kci-dev` pulls from our Dashboard API. As of now, it is an EXPERIMENTAL tooling under development with close collaboration from Linux kernel maintainers.

> KNOWN ISSUE: The Dashboard endpoint we are using returns a file of a few megabytes in size, so download may take
a few seconds, slowing down your usage of `kci-dev results`. We are working on [it](https://github.com/kernelci/dashboard/issues/661).

## Base parameters

### --origin

Set the KCIDB origin desired. 'maestro' is the default.

### --giturl

The url of the tree to fetch results

### --branch

The branch to get results for

### --commit

The tip of tree commit being tested. It needs to be the full commit hash.

Unfortunately the Dashboard API doesn't support git tags as parameters yet.

## Results actions

### --action=summary

Shows a numeric summary of the build, boot and test results.
If `--action` is omitted, it will show the summary by default.

Example:

```sh
kci-dev results --nodeid <str: testnodeid>
kci-dev results --giturl 'https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git' --branch master --commit d1486dca38afd08ca279ae94eb3a397f10737824 --action=summary
```

This command will show the results of tests by page nodes limit and page offset.
### --action=failed-builds

List failed builds.

Example:

```sh
kci-dev results --nodes --limit <int: page nodes limit> --offset <int: page nodes offset>
kci-dev results --giturl 'https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git' --branch master --commit d1486dca38afd08ca279ae94eb3a397f10737824 --action failed-builds
```

Result sample:
```yaml
{'artifacts': None,
'created': '2024-10-04T00:49:15.691000',
'data': {'arch': 'x86_64',
'compiler': 'gcc-12',
'config_full': 'x86_64_defconfig+lab-setup+x86-board',
'defconfig': 'x86_64_defconfig',
'device': None,
'error_code': None,
'error_msg': None,
'job_context': None,
'job_id': None,
'kernel_revision': {'branch': 'staging-mainline',
'commit': '232edaea0fd9b4b7feb7b43508834bba7e820584',
'commit_message': 'staging-mainline-20241004.0',
'commit_tags': ['staging-mainline-20241004.0'],
'describe': 'staging-mainline-20241004.0',
'patchset': None,
'tip_of_branch': False,
'tree': 'kernelci',
'url': 'https://github.com/kernelci/linux.git',
'version': {'extra': '-rc1-115-g232edaea0fd9b',
'name': None,
'patchlevel': 12,
'sublevel': None,
'version': 6}},
'kernel_type': 'bzimage',
'platform': 'hp-14b-na0052xx-zork',
'regression': None,
'runtime': 'lava-collabora',
'test_revision': None,
'test_source': None},
'debug': None,
'group': 'kselftest-exec',
'holdoff': None,
'id': '66ff3b8c0abcc4c8343d1c71',
'jobfilter': None,
'kind': 'test',
'name': 'exec_execveat_Check_success_of_execveat_20_4096',
'owner': 'staging.kernelci.org',
'parent': '66ff3b8b0abcc4c8343d1b8a',
'path': ['checkout',
'kbuild-gcc-12-x86',
'kselftest-exec',
'kselftest-exec',
'exec_execveat_Check_success_of_execveat_20_4096'],
'result': 'pass',
'state': 'done',
'submitter': 'service:pipeline',
'timeout': '2024-10-04T06:49:15.691000',
'treeid': 'a44035dadc31327a5c30db4013b0e7e90acbb6a8fc45f94a6d91671e76cdfd8a',
'updated': '2024-10-04T00:49:15.691000',
'user_groups': []}
```
## Downloading logs

testnodeid is the node id of the test job, which you can get from the KernelCI dashboard.
Usually it is hexadecimal string.
`--download-logs` Download failed logs when used with `--action=failed-*` commands.

Additionally, for --nodes you can provide optional parameters --filter to filter the results by the given key and value.
For example:
Example:
```sh
./kci-dev.py results --nodes --filter treeid=e25266f77837de335edba3c1b8d2a04edc2bfb195b77c44711d81ebea4494140 --filter kind=test
kci-dev results --giturl 'https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git' --branch master --commit d1486dca38afd08ca279ae94eb3a397f10737824 --action failed-builds --download-logs
```
This command will show the nodes of tests in particular tree checkout.
But as you might see, there is a lot of fields you might be not interested in.

For this we have additional option --field, that will restrict output only to specified fields.
For example:
```sh
./kci-dev.py results --nodes --filter treeid=e25266f77837de335edba3c1b8d2a04edc2bfb195b77c44711d81ebea4494140 --filter kind=test --field name --field result
```
Example:

```json
{'name': 'kver', 'result': 'pass'}
{'name': 'config', 'result': 'pass'}
{'name': 'build', 'result': 'pass'}
{'name': 'example_init_test', 'result': 'pass'}
{'name': 'time64_to_tm_test_date_range', 'result': 'pass'}
{'name': 'test_one_cpu', 'result': 'skip'}
{'name': 'test_many_cpus', 'result': 'skip'}
{'name': 'test_one_task_on_all_cpus', 'result': 'skip'}
{'name': 'test_two_tasks_on_all_cpus', 'result': 'skip'}
{'name': 'test_one_task_on_one_cpu', 'result': 'skip'}
{'name': 'test_one_task_mixed', 'result': 'skip'}
{'name': 'test_two_tasks_on_one_cpu', 'result': 'skip'}
{'name': 'test_two_tasks_on_one_all_cpus', 'result': 'skip'}
{'name': 'test_task_on_all_and_one_cpu', 'result': 'skip'}
{'name': 'resource_test_union', 'result': 'pass'}
{'name': 'resource_test_intersection', 'result': 'pass'}
....
```


Loading
Loading