update provider configs

Signed-off-by: Pranav Gaikwad <[email protected]>

enhancements/analyzer-rules-testing/README.md

@@ -52,8 +52,13 @@ Lastly, the project has been mainly relying on rules from the Windup project whi
 Rule authors should should be able to "declare" a test for a rule in a YAML file:
 
 ```yaml
+providers:
+- name: java
+  location: ./java-sample-data/
+- name: go
+  location: ./go-sample-data/
+tests:
 - ruleID: same-rule-id-as-the-rule-to-be-tested
-  data: /path/to/sample/data/
   testCases:
   - description: In source-only mode, should match 10 times
     analysisParams:
@@ -66,9 +71,11 @@ Rule authors should should be able to "declare" a test for a rule in a YAML file
     hasTags:
     - Category=tag1
 ```
-
+* providers: List of configs needed to setup providers. The runner will take a settings file at runtime and mutate it with values defined here for every run. This can also be defined at ruleset level. Values here will take precedance:
+  * name: Name of the provider to match in _provider_settings.json_
+  * location: Path to sample data for this provider
+* tests: List of tests, each testing a distinct rule
   * ruleID: ID of the rule this test applies to. 
-  * data: Optional path to a directory containing sample data for the test. By default, path declared at the ruleset level will be used (see next section for details).
   * testCases: A list of different test cases for this test with each case containing following fields:
     * description: This is what will be printed as text next to PASS / FAIL result
     * analysisParams: Additional _optional_ parameters to configure analysis when running this test. By default, these won't be set.
@@ -82,11 +89,7 @@ Rule authors should should be able to "declare" a test for a rule in a YAML file
 
 Notice that there could be more than one _testCases_ in a test with different _analysisParams_. This is useful when you want to test the same rule but under different analysis parameters. 
 
-Note that above example contains a test for one rule. The YAML file will contain a list of such tests, each item being a test for a distinct rule.
-
-Having a schema for the tests would be nice. That will allow users to auto-generate the boilerplate.
-
-Test passes when all of the following conditions are met:
+A testcase passes when all of the following conditions are met:
 
 - either one of _hasTags_ or _hasIncidents_ is defined
 - _hasTags_ is not defined OR all given tags are present in the output.
@@ -95,19 +98,22 @@ Test passes when all of the following conditions are met:
   - _messageContains_ is not defined OR (it is defined AND message is found in all incidents)
   - _codeSnipContains_ is not defined OR (it is defined AND code snip is found in all incidents)
 
+> Having a schema for the tests would be nice. It will allow users to auto-generate the boilerplate.
+
 #### Organizing test files in a ruleset
 
 The YAML file we described in the previous section will be saved in the same directory as its rules file. The name of this test file will be same as of the rules file appended with `_test` suffix. For instance, tests for a rules file `rules.yaml` will be stored in `rules_test.yaml` in the same directory. The analyzer-lsp engine will ignore any files that have `_test.yaml` during rules parsing. 
 
-Each rule test above has an optional field `data` that points to the sample data directory to use for the test. In most cases, rule authors will end up re-using the same sample data for a ruleset. Therefore, a new field `testData` will be introduced in `ruleset.yaml` file:
+The providers can also be defined at a ruleset level in case users just want to re-use for all tests in a rulest. A new field `testConfig` can be introduced in `ruleset.yaml` file:
 
 ```yaml
 name: my-ruleset
-testData: ./data/
+testConfig:
+  providers:
+  - name: java
+    location: /path/to/java/
 ```
 
-This sample data will be used as input in case a test case doesn't explicitely specify the `data` field.
-
 #### Running the tests
 
 We are proposing a new CLI that takes the tests as input. 
@@ -186,6 +192,16 @@ This output will be later used by the CI. When a new PR is opened in _konveyor/r
 The Go structs are defined below:
 
 ```go
+type TestConfig struct {
+  Providers []ProviderConfig `yaml:"providers"`
+  Tests     []RuleTest       `yaml:"tests"`
+}
+
+type ProviderConfig struct {
+  Name     string `yaml:"name,omitempty"`
+  Location string `yaml:"location,omitempty"`
+}
+
 type RuleTest struct {
 	RuleID    string     `yaml:"ruleID"`
 	Data      string     `yaml:"data,omitempty"`
@@ -226,9 +242,11 @@ Here are options for the  test runner CLI:
 
 #### Running Tests
 
-Depending on the `analysisParams` used, how many rulesets are given and what distinct data directories are present, there will be different combinations of tests with overalapping _analysisParams_ and _data_ directories. It only makes sense for the test runner to first, group tests by data directories, and second, group further by analysis params. This is faster than running every file at once. One drawback of this is that output of a certain rule _may_ change in presence of another rule in the system. Analyzer-lsp engine will eventually run all rules at once though, so _maybe_ it's better to surface these problems early on.
+The runner will run each test file at once setting up the provider by merging two things - provider settings passed at runtime via `--provider-settings` and locations specified in the test file itself for each provider. Each test will run at once. This can be run concurrently in future.
+
+Based on different analysis params used, there will be different combinations of analyses. The overlapping analyses runs can be grouped together to make things faster. It might potentially affect the results if there are rules that somehow affect each other. But maybe there's profit in catching those things early on.
 
-The groups of tests will look something like with 4 instances of different tests running at a time:
+The groups of tests will look something like:
 
 ```sh
 .
@@ -248,8 +266,6 @@ The groups of tests will look something like with 4 instances of different tests
         └── networking_rules_test.yaml
 ```
 
-For this system to work correctly, it's necessary that every test specifies the correct _ruleID_ so that the runner can pick the correct rules.
-
 The test output and logs will be generated in a temp directory per analysis run:
 
 ```sh