-
-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feature: add parsing of task_id, test_code and user_output to bring test-runner to v3 #72
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vaeng
added
enhancement
New feature or request
x:action/improve
Improve existing functionality/content
x:knowledge/advanced
Comprehensive Exercism knowledge required
x:module/test-runner
Work on Test Runners
x:type/docker
Work on Dockerfiles
v3-migration 🤖
Preparing for Exercism v3
x:rep/massive
Massive amount of reputation
labels
Aug 12, 2023
vaeng
changed the title
feature: add task_id parsing
feature: add parsing of task_id, test_code and user_output to bring test-runner to v3
Aug 13, 2023
Closed
ErikSchierboom
approved these changes
Aug 14, 2023
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
enhancement
New feature or request
v3-migration 🤖
Preparing for Exercism v3
x:action/improve
Improve existing functionality/content
x:knowledge/advanced
Comprehensive Exercism knowledge required
x:module/test-runner
Work on Test Runners
x:rep/massive
Massive amount of reputation
x:type/docker
Work on Dockerfiles
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introducing the task_id, test_code, and user output into the test runner is a rather significant change, but necessary to get the track tooling to v3.
This PR enables passing the task_id as a tag in test cases.
task_id usage
It is currently limited to 99 tasks.
New Catch2 version usage
To get the tag from the Catch2 testing framework, the framework had to be updated to version 3.0+.
This requires a new include in the test files.
to:
The test-runner will change the old include to the new header automatically, so no changes for existing exercises are necessary.
Output usage
Students can now output their debugging via
std::cout
. The result is visible in the output field in the final json file.test_code implementation
The test code comes from scanning the test file from one test to the next. It deletes common code elements that might appear between to test case bodies, but not everything. So keep your tests clean 😄
Reasons for the changes
The current framework is version is limited and will not be easy to publish results as specified in the v3 test runner.
The tag is not shown in the current junit testing reporter and had thus to be switched to pure XML.
Instead of rewriting the Python parser to accommodate the changes, a new cpp parser was written.
To use modern C++17, the alpine version of the container was updated as well. This asked for some of the expected_results.json to change to mirror the new error messages from the system correctly.
Maintainers thus do not need to know about Python to take care of the cpp track and the parsing is faster.
A future advantage might be using smaller docker images, that do not need a Python installation.