Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a fixture facility to nnbench #173

Merged
merged 2 commits into from
Nov 26, 2024
Merged

Add a fixture facility to nnbench #173

merged 2 commits into from
Nov 26, 2024

Conversation

nicholasjng
Copy link
Collaborator

@nicholasjng nicholasjng commented Nov 25, 2024

Fixtures give a CLI user the option of defining benchmark input parameters
as "fixtures", i.e. results of functions locate in a special named module,
enabling better customization and configuration.

Unlike pytest, we operate only on raw Python callables, and do not expose
features like scopes, fixture names, and IDs. This is because our
motivation is slightly different from that of pytest, in that we need to
track parameter values and try to keep them constant ("session-scoped")
across benchmarks, except in niche cases.

Our magic filename is conf.py, similarly to conftest.py for pytest.

This change incurs changes in how we construct and typecheck parameters -
since we now allow heterogeneous parameters in benchmarks by virtue of
fixtures, we need to typecheck each individual parameter set. This will
be done in a followup.

Fixtures give a CLI user the option of defining benchmark input parameters
as "fixtures", i.e. results of functions locate in a special named module,
enabling better customization and configuration.

Unlike pytest, we operate only on raw Python callables, and do not expose
features like scopes, fixture names, and IDs. This is because our
motivation is slightly different from that of pytest, in that we need to
track parameter values and try to keep them constant ("session-scoped")
across benchmarks, except in niche cases.

Our magic filename is `conf.py`, similarly to conftest.py for pytest.

This change incurs changes in how we construct and typecheck parameters -
since we now allow heterogeneous parameters in benchmarks by virtue of
fixtures, we need to typecheck each individual parameter set. This will
be done in a followup.
@nicholasjng nicholasjng marked this pull request as ready for review November 26, 2024 15:34
@nicholasjng nicholasjng merged commit 1ed24f6 into main Nov 26, 2024
14 checks passed
@nicholasjng nicholasjng deleted the fixtures branch November 26, 2024 15:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant