Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: add parsing of task_id, test_code and user_output to bring test-runner to v3 #72

Merged
merged 16 commits into from
Aug 14, 2023

Conversation

vaeng
Copy link
Contributor

@vaeng vaeng commented Aug 12, 2023

Introducing the task_id, test_code, and user output into the test runner is a rather significant change, but necessary to get the track tooling to v3.

This PR enables passing the task_id as a tag in test cases.

task_id usage

It is currently limited to 99 tasks.

TEST_CASE("divisible_by_400", "[task_3]") { ... }

New Catch2 version usage

To get the tag from the Catch2 testing framework, the framework had to be updated to version 3.0+.
This requires a new include in the test files.

#ifdef EXERCISM_TEST_SUITE
#include <catch2/catch.hpp> // Catch2 v2.0+
#else
#include "test/catch.hpp"
#endif

to:

#ifdef EXERCISM_TEST_SUITE
#include <catch2/catch_all.hpp>  // Catch2 v3.0+
#else
#include "test/catch.hpp"
#endif

The test-runner will change the old include to the new header automatically, so no changes for existing exercises are necessary.

Output usage

Students can now output their debugging via std::cout. The result is visible in the output field in the final json file.

test_code implementation

The test code comes from scanning the test file from one test to the next. It deletes common code elements that might appear between to test case bodies, but not everything. So keep your tests clean 😄

Reasons for the changes

The current framework is version is limited and will not be easy to publish results as specified in the v3 test runner.

The tag is not shown in the current junit testing reporter and had thus to be switched to pure XML.
Instead of rewriting the Python parser to accommodate the changes, a new cpp parser was written.

To use modern C++17, the alpine version of the container was updated as well. This asked for some of the expected_results.json to change to mirror the new error messages from the system correctly.

Maintainers thus do not need to know about Python to take care of the cpp track and the parsing is faster.
A future advantage might be using smaller docker images, that do not need a Python installation.

@vaeng vaeng self-assigned this Aug 12, 2023
@vaeng vaeng requested a review from a team as a code owner August 12, 2023 22:28
@vaeng vaeng added enhancement New feature or request x:action/improve Improve existing functionality/content x:knowledge/advanced Comprehensive Exercism knowledge required x:module/test-runner Work on Test Runners x:type/docker Work on Dockerfiles v3-migration 🤖 Preparing for Exercism v3 x:rep/massive Massive amount of reputation labels Aug 12, 2023
@vaeng vaeng changed the title feature: add task_id parsing feature: add parsing of task_id, test_code and user_output to bring test-runner to v3 Aug 13, 2023
@vaeng vaeng mentioned this pull request Aug 13, 2023
Copy link
Member

@ErikSchierboom ErikSchierboom left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@vaeng vaeng merged commit 01ef38a into main Aug 14, 2023
1 check passed
@vaeng vaeng deleted the add-task_id-parsing branch August 14, 2023 11:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request v3-migration 🤖 Preparing for Exercism v3 x:action/improve Improve existing functionality/content x:knowledge/advanced Comprehensive Exercism knowledge required x:module/test-runner Work on Test Runners x:rep/massive Massive amount of reputation x:type/docker Work on Dockerfiles
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants