Add tests that simply run all (SIL) configs in everest-core/config

[hikinggrass]

This should help in finding regressions that need configuration changes or detect broken configs

[Dominik-K]

The multiplied, same-functionality code should be generalized (see comment).

[Dominik-K]

How long will these tests take? Tried to answer myself but the tests are currently failing on initialisation.

If it takes too long, one option would be to run them in parallel. But unfortunately, GitHub Actions doesn't support this out of the box, AFAIK. CircleCI has a neat helper for it. But perhaps there's a good tool for GitHub Actions, too.

[Dominik-K]

Sorry, clicked wrong 🙈

[hikinggrass]

How long will these tests take? Tried to answer myself but the tests are currently failing on initialisation.

If it takes too long, one option would be to run them in parallel. But unfortunately, GitHub Actions doesn't support this out of the box, AFAIK. CircleCI has a neat helper for it. But perhaps there's a good tool for GitHub Actions, too.

Let's see how long they take, them failing right now is already being worked on (basically we use a version of everest-testing that's too old in the ci tests)

[Dominik-K]

If it takes too long, one option would be to run them in parallel. But unfortunately, GitHub Actions doesn't support this out of the box, AFAIK. CircleCI has a neat helper for it. But perhaps there's a good tool for GitHub Actions, too.

Yeah, there are tools for GitHub Actions. Was quite simple to find searching for "split tests":

• https://imhoff.blog/posts/parallelizing-jest-with-github-actions (mentioning an undocumented field jobs.<job-id>.strategy.matrix.test-files
• Split done by test-timing: https://github.com/leonid-shevtsov/split_tests

[hikinggrass]

All 15 tests that are implemented right now run sequentially and take ~67 seconds, so <5s/test. Codacy for example usually takes much longer than that, so I think this is negligible for now.

[Dominik-K]

LGTM with comments (in the meaning: I'm sure you'll address my comments.)

[hikinggrass]

For this PR to work we need to have these merged:
#475
EVerest/everest-ci#12

[andistorm]

🥇