Skip to content

Latest commit

 

History

History
108 lines (65 loc) · 2.86 KB

000001-compatibility-tests.md

File metadata and controls

108 lines (65 loc) · 2.86 KB
Raw

Compatibility Tests

Submitters

  • Fran Tufro (Hidden People Club)

Change Log

Use Case(s)

To ensure a healthy ecosystem around the cuentitos programming language, we need to make sure that all the runtimes are compatible with each other.

For that we'll create a set of tests that can be run against a command line implementation of the runtime with a very specific input support.

Context

We need to make sure that all the runtimes are compatible with each other, so that a script written in cuentitos can be run in any runtime.

We need a specific ADR for this to document the work I'll do to create the compatibility tests and the runner.

Proposed Design

This ADR proposes:

  1. A Test Case Format
  2. A Test Runner
  3. CLI Requirements
  4. Documentation

I think that with these three elements, we'll be able to create a set of tests that can be run against any runtime.

Test Case Format

We'll need to create a format for the test cases that we'll use to test the runtimes.

The approach I'm thinking of is to create a Markdown file with a very specific structure:

# Test Name

A description of the test.

## Script
```cuentitos 
// The script to run
```

## Input
```input
// A linebreak-separated list of inputs
```

## Result
```result
// The expected output in the command line
```

Test Case Runner

We'll need a tool that can run the test cases and generate a report.

This tool will be created in the compat directory, and it will be called cuentitos-compat.

To run all the tests you should be able to do:

./cuentitos-compat [runtime_cli] [test_cases_directory]

In the case you wanted to run just one test:

./cuentitos-compat [runtime_cli] [test_case_file_path]

The test runner will match the output generated by the Runtime CLI to what's in the test file.

This includes scripts that contain errors: if a test case includes a malformed cuentitos script, it will test the resulting output with the result section in the test case, and fail if they differ.

If you want to avoid this behavior, you can call:

./cuentitos-compat [runtime_cli] [test_case_file_path] --ignore-errors

This will ignore test cases that had errors in the report.

CLI requirements

The Runtime CLI should have the following parameters:

./runtime-cli run [script_file_path] --input [input_file_path]

It should return 0 if the script was executed successfully, and 1 if it failed.

It should also print all the output to the standard output to be captured by the test runner.

Documentation

I will create a document that explains how to create a new test, with details on how to define the input and expected output.