autograder_utils

autograder_utils

This package provides some helper classes which assist in developing autograders for the Gradescope autograder platform. These utilities work together to make it easy to use Python's built in unittest module for creating autograders that work on Gradescope.

decorators

Decorators are provided for annotating test cases.

@weight

The @weight decorator allows setting the point value of a test case. This information is used by the JSONTestRunner in producing its output. This is required for any tests which award points.

Example:

@weight(5.0)
def test_functionality():
    ...

@tags

The @tags decorator allows setting tags on a test case. This is optional, but may be used in the future to enable analytics on the concepts being tested in programming assignments.

Example:

@tags("conditionals", "recursion")
def test_search():
    ...

@visibility

The @visibility decorator allows setting the visibility of a test case.

Example:

@visibility("after_due_date")
def test_search():
    ...

unittest utilities

JSONTestRunner

This utility class is a custom unittest test runner which produces JSON output in the format that Gradescope's infrastructure expects. It works with test cases annotated with the @weight annotation to determine the point value of each test case.

Example:

suite = unittest.defaultTestLoader.discover('tests')
with open('results.json', 'w') as f:
    JSONTestRunner(stream=f, verbosity=2, buffer=True).run(suite)

JSONTestResult

This class is used by JSONTestRunner to format output in JSON.

Example output:

{
    "output": "Printing inside a test case\n",
    "score": 3,
    "max_score": 3,
    "name": "Test parsing a simple expression",
    "tags": [
        "parsing",
        "lexing"
    ]
}