Skip to content

Commit

Permalink
docs: add documentation
Browse files Browse the repository at this point in the history
* docs: add documentation for recipe resource

* docs: update readme and example for better clarity

* docs: add framework schema and definition

* docs: add procedure and output target to recipe documentation

* docs: add command documentation
  • Loading branch information
irainia authored Jan 13, 2022
1 parent 72efb93 commit c043021
Show file tree
Hide file tree
Showing 11 changed files with 562 additions and 71 deletions.
117 changes: 66 additions & 51 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,85 +8,100 @@ Valor is a tool that can be used to validate and evaluate resource.

### 1. Clone Repo

Clone this repository into your local by running the following command:
Clone this repository into the local by running the following command:

```zsh
git clone github.com/gojek/optimus-extension-valor
```

### 2. Create Recipe

Create a recipe named `valor.yaml` or rename the recipe file in
this project directory from `valor.example.yaml` into `valor.yaml`.
The content of the recipe should look like the following.
Assets referred by the specified recipe are available under `./example` directory.
Copy the recipe example `valor.example.yaml` to `valor.yaml`. The resource that will be validated and evaluated from the recipe is like the following:

```yaml
resources:
resources:
- name: user_account
type: file
path: ./example/resource/user_account.json
type: dir
path: ./example/resource
format: json
framework_names:
- user_account_validation

frameworks:
- name: user_account_validation
schemas:
- name: user_account_rule
type: file
format: json
path: ./example/schema/user_account_rule.json
procedures:
- name: enrich_user_account
type: file
format: jsonnet
path: ./example/procedure/enrich_user_account.jsonnet
output_targets:
- name: std_output
type: std
format: json
- user_account_evaluation
...
```

In this example, Resource named `user_account` will be evaluated agaist
framework named `user_account_validation`. The targetted framework
specifies three things:
As stated in the recipe, the resource is available under directory `./example/resource`. One example of the resource is like the following:

* Schema, which follows JSON schema format to validate the Resource. It is
required to validate whether the structure for the `user_account`
Resource contains error or not.
* Procedure, which follows JSONNET format to evaluate the `user_account` Resource.
It can be for validation or any execution that can't be handled by
using Schema.
* Output Target, which specifies on how to write the output of either Schema or
Procedure or both.
```json
{
"email": "[email protected]",
"membership_id": 1,
"is_active": true
}
```

Note that the `membership_id` is in numeric.

### 3. Prepare Valor

### 3. Execute Pipeline
To execute the pipeline, one can either:

After the preparation is done, try running valor by executing the following
command:
* download the latest binary from [here](https://github.com/gojek/optimus-extension-valor/releases) and put it in this project directory, or
* try building it by following [this guide](#how-to-build)

### 4. Execute Pipeline

After the preparation is done, try executing Valor from the CLI like the following:

```zsh
./out/valor execute
```

Or, if you have not build it, try building it by following [#HowToBuild](#how-to-build).
Make sure to have the required dependencies specified under [#Dependency](#dependency).
The output of the above solution should look like the following:
And done. Based on the recipe, the output will be printed to the std out, or in other words in the CLI. The output should contain one of the modified input, like the following:

```zsh
user_account: ./example/resource/user_account.json
...
{
"email": "[email protected]",
"is_active": true,
"is_valid": true,
"membership": "premium"
}
...
```

## Dependency
### 5. Explanation

What Valor does is actually stated in the recipe file `valor.yaml`. Behind the scene, the process is like the following:

1. read the first resource
2. execute framework pointed by field `framework_names`
3. in the framework, run validation based on `schemas`
4. if no error is found, load the required definition under `definitions`
5. if no error is found, execute proceduresstated under `procedures`
6. if no error is found, write the result based on `output_targets`

The explanation here is quite brief. For further explanation, try checkout the documentation [here](#documentation).

## Documentation

The complete documentation is available like the following:

* [recipe](./docs/recipe.md), which explains more about recipe and how to set it up for execution
* [command](./docs/command.md), which explains more about the available commands to be run

## Development

Community contribution is very welcome. Though, because of diverse background, one might not be able to easily understand the reasoning behind every decision or approach. So, the following is the general guideline to contribute:

1. create a dedicated branch
2. after all changes are done, create a PR
3. in that PR, please describes the issue or reasining or even approach for such changes
4. if no issue is found, merge will be done shortly

In order to follow the above steps, one might need to setup the local environment. The following is the general information to set it up.

### Dependency

### GO Language
#### GO Language

The GO programming language version `go1.17.1` need to be installed
in the system. Go to [this link](https://golang.org/doc/install) and follow
Expand All @@ -102,7 +117,7 @@ example output:
go version go1.17.1 darwin/amd64
```

## How to Test
#### How to Test

In this project root directory, run the following command to test:

Expand All @@ -117,9 +132,9 @@ make coverage
```

You can check into `coverage.html` file in root project directory.
This command also will open interactive coverage tool in your browser if you have one.
This command also will open interactive coverage tool in the browser.

## How to Build
#### How to Build

To build the binary executable, in this project root directory, run the following command:

Expand All @@ -137,7 +152,7 @@ make dist

There will be a new directory named `dist` with few executable files.

## How to Run
#### How to Run

In order to run this project, after building the binary executable, run
the following command in this project root directory:
Expand Down
101 changes: 101 additions & 0 deletions docs/command.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
# Command

## Description

This documentation explains more about the available commands in Valor.
Assuming Valor is already built (see [here](../README.md#how-to-run)), then when the user run the command:

```zsh
./out/valor
```

The following output will be shown in CLI:

```zsh
Usage:
valor [command]

Available Commands:
completion generate the autocompletion script for the specified shell
execute Execute pipeline based on the specified recipe
help Help about any command
profile Profile the recipe specified by path

Flags:
-h, --help help for valor

Use "valor [command] --help" for more information about a command.
```

## Profile

Profile is a command that will profile the specified Valor recipe. By default, the recipe being profiled is read from `valor.yaml` in the active directory. Like the following example:

```zsh
./out/valor profile
```

If the recipe from the provided example `valor.example.yaml` is copied into `valor.yaml`, then the output will be like the following:

```zsh
RESOURCE:
+--------------+--------+------+--------------------+-------------------------+
| NAME | FORMAT | TYPE | PATH | FRAMEWORK |
+--------------+--------+------+--------------------+-------------------------+
| user_account | json | dir | ./example/resource | user_account_evaluation |
+--------------+--------+------+--------------------+-------------------------+

FRAMEWORK:
+-------------------------+------------+---------------------+
| FRAMEWORK | TYPE | NAME |
+-------------------------+------------+---------------------+
| user_account_evaluation | definition | memberships |
+ +------------+---------------------+
| | schema | user_account_rule |
+ +------------+---------------------+
| | procedure | enrich_user_account |
+ +------------+---------------------+
| | output | std_output |
+-------------------------+------------+---------------------+
```

The user can also specify other recipe by using flag `--recipe-path` like the following:

```zsh
./out/valor profile --recipe-path=valor.example.yaml
```

## Execute

Execute is a command that execute pipeline based on the provided recipe. By default, the recipe being executed is read from `valor.yaml` in the active directory. An example of running this command:

```zsh
./out/valor execute
```

Running the above command will execute all frameworks under `valor.yaml` recipe. This command also has several flags.

Flag | Description | Format
--- | --- | ---
--batch-size | specify the number of data to be executed paralelly in one batch | it should be an integer more than 0 (zero). it is optional with default value 4 (four).
--progress-type | specify the progress type during execution | currently available: `verbose` (default) and `simple`
--recipe-path | customize the recipe that will be executed | it is optional. the value should be a valid recipe path

This command also has sub-command. The currently available sub-commands are explained below.

### Resource

Recipe is required in every execution. However, sometime the user just want a certain resource to executed. The user can define a new recipe to accomplish such requirement. Or, another alternative is by using resource sub-command. Resource sub-command is a command that allows the user to specify which resource to be executed. So, with he same recipe, the user can select the only resource they need to be executed. To use this sub-command, run it like the following:

```zsh
./out/valor execute resource
```

This sub-command inherit the flags from the execute command. In addition to that, it also has its own flags like the following.

Flag | Description | Format
--- | --- | ---
--name | the name of the resource to be executed | it is mandatory and should refer to the resource in the recipe
--format | the format of the input resource | if it's not specified, Valor will use the format in recipe. but if it is, then Valor will use it instead.
--path | the path of the input resource | if it's not specified, Valor will use the format in recipe. but if it is, then Valor will use it instead.
--type | the type of path for the specified resource | if it's not specified, Valor will use the format in recipe. but if it is, then Valor will use it instead.
Loading

0 comments on commit c043021

Please sign in to comment.