Create a data-driven test case framework

[dogweather]

The Idea: enable non-programmers to write test cases.

Context: The English version has its tests in rome_statute_test.py. Each language would need its own test file. Eventually these could be refactored together. But regardless, a programmer would need to do this work.

But we can allow people with good language skills to contribute if we completely remove the expected output from the test file and store it separately. Maybe in a spreadsheet or YAML file which the Python test file would read. This separate "database" would be 2-dimensional. One dimension are the tests such as "document title". The other dimension is the human languages such as English, Russian, etc.

See also:

• https://medium.com/ranorex-webtestit/data-driven-testing-using-python-and-the-parameterized-module-37e2cb863915

[dogweather]

Complications

Not all tests simply look for a string.

• One test case expects a list of lines, List[str].
• Some do a .startswith() match.

The second one above should make no difference. But the one that looks for a List[str] ... one idea is to re-write it to combine the two lines into one string.

[dogweather]

Thoughts about the database:

• Something that's committed into the Repo?
• Some kind of schema-checking in CI? That'd be YAML or JSON. But a table form would be more intuitive, I think. Going with it, YAML could look like:

Tests:
  Title Works Correctly:
    French:  Statut de Rome de la Cour pénale internationale
    English: Rome Statute of the International Criminal Court

  Language Works Correctly:
    French:  fr-FR
    English: en-US

A bonus with YAML is that it supports values of list of strings.

The "schema checking" could be just a few lines of Python that attempts to read in the data. Or, it could be a JSON-schema.

• It could be Python code using a list of NamedTuple or DataClass:

add(TestCase(
    test_name = "Title works correctly",
    french    = "Statut de Rome de la Cour pénale internationale",
    english   = "Rome Statute of the International Criminal Court"
))

add(TestCase(
    test_name = "Language Works Correctly",
    french    = "fr-FR",
    english   = "en-US"
))

• Maybe an actual database? Because that'd do schema-checking as well, and with some front-end would show the contents in a table form.

[dogweather]

Idea for testing the tests:

Create a NullParser which simply returns blank strings. if a test case is written, every test should fail when run with this parser.

This wouldn't provide a lot of assurance, but it'd be some. (?)