diff --git a/README.md b/README.md index 9563118..a3efd42 100644 --- a/README.md +++ b/README.md @@ -3,100 +3,83 @@ Kantra is a CLI that unifies analysis and transformation capabilities of Konveyor. ## Installation -The easiest way to install Kantra is to get it via the container image. To download latest container image, run: -### Linux - -```sh -podman cp $(podman create --name kantra-download quay.io/konveyor/kantra:latest):/usr/local/bin/kantra . && podman rm kantra-download -``` - -### MacOS - -**Note:** There is a known [issue](https://github.com/containers/podman/issues/16106) -with limited number of open files in mounted volumes on MacOS, which may affect kantra performance. - -Prior to starting your podman machine, run: - -```sh -ulimit -n unlimited -``` - - - This must be run after each podman machine reboot. - -In order to correctly mount volumes, your podman machine must contain options: - -```sh -podman machine init -v $HOME:$HOME -v /private/tmp:/private/tmp -v /var/folders/:/var/folders/ -``` - -Increase podman resources: - -```sh -podman machine set --cpus 4 --memory 4096 -``` - - -Ensure that we use the connection to the VM `` we created earlier by default: - -```sh -podman system connection default -``` - -```sh -podman pull quay.io/konveyor/kantra:latest && podman run --name kantra-download quay.io/konveyor/kantra:latest 1> /dev/null 2> /dev/null && podman cp kantra-download:/usr/local/bin/darwin-kantra kantra && podman rm kantra-download -``` - -### Windows - -```sh -podman pull quay.io/konveyor/kantra:latest && podman run --name kantra-download quay.io/konveyor/kantra:latest 1> /dev/null 2> /dev/null && podman cp kantra-download:/usr/local/bin/windows-kantra kantra && podman rm kantra-download -``` - ---- - -The above will copy the binary into your current directory. Move it to PATH for system-wide use: - -```sh -sudo mv ./kantra /usr/local/bin/ -``` - -To confirm Kantra is installed, run: - -```sh -kantra --help -``` - -This should display the help message. +The easiest way to install Kantra is to get it via the container image. + +1. To download latest container image using _podman_, follow instructions for your operating system: + + * For Linux, run: + + ```sh + podman cp $(podman create --name kantra-download quay.io/konveyor/kantra:latest):/usr/local/bin/kantra . && podman rm kantra-download + ``` + + * For MacOS + + Prior to starting your podman machine, run: + + ```sh + ulimit -n unlimited + ``` + > This must be run after each podman machine reboot. + + Init your _podman_ machine : + + ```sh + podman machine init -v $HOME:$HOME -v /private/tmp:/private/tmp -v /var/folders/:/var/folders/ + ``` + + Increase podman resources: + + ```sh + podman machine set --cpus 4 --memory 4096 + ``` + + Ensure that we use the connection to the VM `` we created earlier by default: + + ```sh + podman system connection default + ``` + + Finally, run: + + ```sh + podman pull quay.io/konveyor/kantra:latest && podman run --name kantra-download quay.io/konveyor/kantra:latest 1> /dev/null 2> /dev/null && podman cp kantra-download:/usr/local/bin/darwin-kantra kantra && podman rm kantra-download + ``` + + * For Windows, run: + + ```sh + podman pull quay.io/konveyor/kantra:latest && podman run --name kantra-download quay.io/konveyor/kantra:latest 1> /dev/null 2> /dev/null && podman cp kantra-download:/usr/local/bin/windows-kantra kantra && podman rm kantra-download + ``` + +2. The above will copy the binary into your current directory. Move it to PATH for system-wide use: + + ```sh + sudo mv ./kantra /usr/local/bin/ + ``` + +3. To confirm Kantra is installed, run: + + ```sh + kantra --help + ``` + + This should display the help message. ## Usage -Kantra has two subcommands - `analyze` and `transform`: +Kantra has three subcommands: +1. _analyze_: This subcommand allows running source code analysis on input source code or a binary. -```sh -A cli tool for analysis and transformation of applications - -Usage: - kantra [command] - -Available Commands: - analyze Analyze application source code - completion Generate the autocompletion script for the specified shell - help Help about any command - transform Transform application source code or windup XML rules - -Flags: - -h, --help help for kantra - --log-level int log level (default 5) - -Use "kantra [command] --help" for more information about a command. -``` +2. _transform_: This subcommand allows - converting XML rules to YAML, and running OpenRewrite recipes on source code. +3. _test_: This subcommand allows testing YAML rules. ### Analyze -Analyze allows running source code and binary analysis using [analyzer-lsp](https://github.com/konveyor/analyzer-lsp) +_analyze_ subcommand allows running source code and binary analysis using [analyzer-lsp](https://github.com/konveyor/analyzer-lsp) To run analysis on application source code, run: @@ -104,14 +87,11 @@ To run analysis on application source code, run: kantra analyze --input= --output= ``` +_--input_ must point to a source code directory or a binary file, _--output_ must point to a directory to contain analysis results. + All flags: ```sh -Analyze application source code - -Usage: - kantra analyze [flags] - Flags: --analyze-known-libraries analyze known open-source libraries -h, --help help for analyze @@ -125,46 +105,35 @@ Flags: --skip-static-report do not generate static report -s, --source stringArray source technology to consider for analysis. Use multiple times for additional sources: --source --source ... -t, --target stringArray target technology to consider for analysis. Use multiple times for additional targets: --target --target ... - -Global Flags: - --log-level int log level (default 5) ``` ### Transform -Transform has two subcommands - `openrewrite` and `rules`. +Transform has two subcommands: -```sh -Transform application source code or windup XML rules +1. _openrewrite_: This subcommand allows running one or more available OpenRewrite recipes on input source code. -Usage: - kantra transform [flags] - kantra transform [command] +2. _rules_: This sucommand allows converting Windup XML rules into the analyzer-lsp YAML format. -Available Commands: - openrewrite Transform application source code using OpenRewrite recipes - rules Convert XML rules to YAML +#### OpenRewrite -Flags: - -h, --help help for transform +_openrewrite_ subcommand allows running [OpenRewrite](https://docs.openrewrite.org/) recipes on source code. -Global Flags: - --log-level int log level (default 5) +To transform applications using OpenRewrite, run: -Use "kantra transform [command] --help" for more information about a command. +```sh +kantra transform openrewrite --input= --target= ``` -#### OpenRewrite - -`openrewrite` subcommand allows running [OpenRewrite](https://docs.openrewrite.org/) recipes on source code. - +The value of _--target_ option must be one of the available OpenRewrite recipes. To list all available recipes, run: ```sh -Transform application source code using OpenRewrite recipes +kantra transform --list-targets +``` -Usage: - kantra transform openrewrite [flags] +All flags: +```sh Flags: -g, --goal string target goal (default "dryRun") -h, --help help for openrewrite @@ -172,43 +141,48 @@ Flags: -l, --list-targets list all available OpenRewrite recipes -s, --maven-settings string path to a custom maven settings file to use -t, --target string target openrewrite recipe to use. Run --list-targets to get a list of packaged recipes. - -Global Flags: - --log-level int log level (default 5) ``` -To run `transform openrewrite` on application source code, run: +#### Rules + +_rules_ subcommand allows converting Windup XML rules to analyzer-lsp YAML rules using [windup-shim](https://github.com/konveyor/windup-shim) + +To convert Windup XML rules to the analyzer-lsp YAML format, run: ```sh -kantra transform openrewrite --input= --target= +kantra transform rules --input= --output= ``` -#### Rules +_--input_ flag should point to a file or a directory containing XML rules, _--output_ should point to an output directory for YAML rules. -`rules` subcommand allows converting Windup XML rules to analyzer-lsp YAML rules using [windup-shim](https://github.com/konveyor/windup-shim) +All flags: ```sh -Convert XML rules to YAML - -Usage: - kantra transform rules [flags] - Flags: -h, --help help for rules -i, --input stringArray path to XML rule file(s) or directory -o, --output string path to output directory - -Global Flags: - --log-level int log level (default 5) ``` -To run `transform rules` on application source code, run: +### Test + +_test_ subcommand allows running tests on YAML rules written for analyzer-lsp. + +The input to test runner will be one or more test files and / or directories containing tests written in YAML. ```sh -kantra transform rules --input= --output= +kantra test /path/to/a/single/tests/file.test.yaml ``` -### analyze and transform [examples](./docs/example.md) +The output of tests is printed on the console. + +See different ways to run the test command in the [test runner doc](./docs/testrunner.md#running-tests) + +## References + +- [Example usage scenarios](./docs/examples.md) +- [Test runner for YAML rules](./docs/testrunner.md) ## Code of Conduct + Refer to Konveyor's Code of Conduct [here](https://github.com/konveyor/community/blob/main/CODE_OF_CONDUCT.md). diff --git a/docs/example.md b/docs/examples.md similarity index 100% rename from docs/example.md rename to docs/examples.md diff --git a/docs/testrunner.md b/docs/testrunner.md new file mode 100644 index 0000000..4776395 --- /dev/null +++ b/docs/testrunner.md @@ -0,0 +1,232 @@ +# Test Runner for YAML rules + +Via the _test_ subcommand, _kantra_ exposes a test runner. + +It allows testing YAML rules written for [analyzer-lsp](https://github.com/konveyor/analyzer-lsp). + +The input to the test runner are tests written in YAML, the output of the test runner is a report. + +## Usage + +This section covers: + +1. [Writing tests](#writing-tests) +2. [] + +### Writing tests + +Tests for a rules file are written in a YAML file with names ending in `.test.yaml` suffix. + +A tests file contains three fields _rulesPath_, _providers_ and _tests_ at the top level: + +```yaml +rulesPath: "/optional/path/to/rules/file" +providers: + - name: "go" + dataPath: "/path/to/data/for/this/provider" +tests: + - ruleID: "rule-id-for-this-test" + testCases: + - name: "test-case-name" + [...] +``` + +* _rulesPath_: Relative path to a file containing rules these tests are applicable to +* _providers_: List of configs, each containing configuration for a specific provider to be used when running tests +* _tests_: List of tests to run, each containing test definition for a specific rule in the associated rules file + +> Note that _rulesPath_ is optional. If it is not specified, the runner will look for a file in the same directory with the same name as the tests file except the _.tests.yaml_ suffix in the name. + +#### Defining providers + +The field _providers_ defines a list of configs, each specific to a provider: + +```yaml +providers: + - name: + dataPath: +tests: + [...] +``` + +_name_ is the name of the provider to which the config applies to, and _dataPath_ is the relative path to the test data to be used when testing rules for that provider. + +> Note that _dataPath_ must be relative to the directory in which tests file exists. + +If all tests under a _ruleset_ share values of _providers_ field (e.g. they use common data directory in all tests for a given provider), this config can also be defined at ruleset level under a special file `testing-config.yaml`. In that case, config present in this file will apply to all tests in that directory. A more specific config for a certain file can still be defined in the tests file. In that case, values in the tests file will take precedance over values at the _ruleset_ level. + +See an example of ruleset level config in [../pkg/testing/examples/ruleset/testing-config.yaml](../pkg/testing/examples/ruleset/testing-config.yaml). + +> Note that a config for every providers present in the rules file _must_ be defined. + +#### Defining tests + +The field _tests_ defines a list of tests, each specific to a rule in the rules file: + +```yaml +providers: + [...] +tests: + - ruleID: test-00 + testCases: + - name: test-tc-00 + analysisParams: + depLabelSelector: "!konveyor.io/source=open-source" + mode: "full" + hasIncidents: + exactly: 10 + messageMatches: "test" + codeSnipMatches: "test" + - name: test-tc-01 + analysisParams: + mode: "source-only" + hasTags: + - "test" + hasIncidents: + locations: + - lineNumber: 10 + fileURI: file://test + messageMatches: "message" +``` +###### Test + +| Field | Type | Required | Description | +| --------- | --------- | -------- | --------------------------------------------- | +| ruleID | string | Yes | ID of the rule this test applies to | +| testCases | []TestCase | Yes | List of test cases (See [TestCase](#testcase)) | + +###### TestCase + +| Field | Type | Required | Description | +| --------- | --------- | -------- | ------------------------------------------------------------------------------------------------------ | +| name | string | Yes | Unique name for the test case, can be used to filter test case. | +| analysisParams | AnalysisParams | Yes | Analysis parameters to use when running this test case (See [AnalysisParams](#analysisparams)) | +| hasIncidents | HasIncidents | No | Passing criteria that compares produced incidents (See [HasIncidents](#hasincidents)) | +| hasTags | []string | No | Passing criteria that compares produced tags, passes test case when all tags are present in the output | +| isUnmatched | bool | No | Passes the test case when rule is NOT matched | + +###### AnalysisParams + +| Field | Type | Required | Description | +| ---------------- | ------- | -------- | ------------------------------------------------------------------------------------ | +| depLabelSelector | string | No | Dependency label selector expression to pass as --dep-label-selector to the analyzer | +| mode | string | No | Analysis mode, one of - _source-only_ or _full_ | + +###### HasIncidents + +_HasIncidents_ defines a criteria for passing the test case. It provides two ways to define a criteria, either one of the two can be defined in a test case: + +1. _Count based_: This criteria is based on count of incidents. It can be defined using following fields under _hasIncidents_: + + | Field | Type | Required | Description | + | --------------- | ---- | -------- | --------------------------------------------------------------------------------------- | + | exactly | int | Yes | Produced incidents should be exactly equal to this number for test case to pass | + | atLeast | int | Yes | Produced incidents should be greater than or equal to this number for test case to pass | + | atMost | int | Yes | Produced incidents should be less than or equal to this number for test case to pass | + | messageMatches | int | No | In all incidents, message should match this pattern for test case to pass | + | codeSnipMatches | int | No | In all incidents, code snippet should match this pattern for test case to pass | + + > Only one of _exactly_, _atLeast_, or _atMost_ can be defined at a time + +2. _Location based_: This criteria is based on location of each incident. It can be defined using following fields under _hasIncidents_: + + | Field | Type | Required | Description | + | --------------- | ---- | -------- | ----------------------------------------------------------------------------------- | + | locations | []Location | No | Passing criteria that is based on location of each incident rather than just count | + + Each _Location_ has following fields: + + | Field | Type | Required | Description | + | --------------- | ---- | -------- | ------------------------------------------------------------------- | + | fileURI | string | Yes | An incident must be found in this file for test case to pass | + | lineNumber | string | Yes | An incident must be found on this line number for test case to pass | + | messageMatches | int | No | Message should match this pattern for test case to pass | + | codeSnipMatches | int | No | Code snippet should match this pattern for test case to pass | + +### Running tests + +To run tests in a single file: + +```yaml +kantra test /path/to/a/single/tests/file.test.yaml +``` + +To run tests in a ruleset: + +```yaml +kantra test /path/to/a/ruleset/directory/ +``` + +To run tests in multiple different paths: + +```yaml +kantra test /path/to/a/ruleset/directory/ /path/to/a/test/file.test.yaml +``` + +To run specific tests by rule IDs: + +```yaml +kantra test /path/to/a/ruleset/directory/ -t "RULE_ID_1, RULE_ID_2" +``` + +_-t_ option allows specifying a list of rule IDs (separated by commas) to select specific tests. + +A specific test case in a test can also be selected using the _-t_ option. + +To run specific test cases in a test, each value in the comma separated list of _-t_ becomes _#_: + +```yaml +kantra test /path/to/a/ruleset/directory/ -t RULE_ID_1#TEST_CASE_1 +``` + +> Note that # is a reserved character used to seperate test case name in the filter. The name of the test case itself _must not_ contain #. + +### Test Output + +When a test passes, the runner creates output that looks like: + +```sh +- 156-java-rmi.windup.test.yaml 2/2 PASSED + - java-rmi-00000 1/1 PASSED + - java-rmi-00001 1/1 PASSED +------------------------------------------------------------ + Rules Summary: 2/2 (100.00%) PASSED + Test Cases Summary: 2/2 (100.00%) PASSED +------------------------------------------------------------ +``` + +The runner will clean up all temporary directories when all tests in a file pass. + +If a test fails, the runner will create output that looks like: + +```sh +- 160-local-storage.windup.test.yaml 0/1 PASSED + - local-storage-00001 0/1 PASSED + - tc-1 FAILED + - expected at least 48 incidents, got 18 + - find debug data in /tmp/rules-test-242432604 +------------------------------------------------------------ + Rules Summary: 0/1 (0.00%) PASSED + Test Cases Summary: 0/1 (0.00%) PASSED +------------------------------------------------------------ +``` + +In this case, the runner leaves the temporary directories behind for debugging. In the above example, the temporary directory is `/tmp/rules-test-242432604`. + +Among other files, the important files needed for debugging in this directory are: + +* _analysis.log_: This file contains the full log of analysis +* _output.yaml_: This file contains the output generated post analysis +* _provider\_settings.json_: This file contains the provider settings used for analysis +* _rules.yaml_: This file contains the rules used for analysis +* _reproducer.sh_: This file contains a command you can run directly on your system to reproduce the analysis as-is. + +> In the temporary directory, there could be files generated by the providers including their own logs. Those files can be useful for debugging too. + +### References + +- OpenAPI schema for tests: [Tests schema](../test-schema.json) + +- Example tests for a ruleset: [Ruleset tests](../pkg/testing/examples/ruleset/) + +- Example tests for a rules file: [Rules file tests](../pkg/testing/examples/rules-file.test.yaml)