Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add nnbench compare subcommand, comparison function #164

Merged
merged 2 commits into from
Nov 19, 2024
Merged

Conversation

nicholasjng
Copy link
Collaborator

Works using metric (benchmark) names as columns, and records as lines. Alternatively, we could support metrics as rows and record IDs as columns in the future.

Contains logic to add parameter values and context values to the table as columns, also directly on the command line. In the future, we could make a comparison class that can be subclassed to avoid option clutter on the command line for customizing the rich table display.

Works using metric (benchmark) names as columns, and records as lines.
Alternatively, we could support metrics as rows and record IDs as columns
in the future.

Contains logic to add parameter values and context values to the table
as columns, also directly on the command line. In the future, we could
make a comparison class that can be subclassed to avoid option clutter on
the command line for customizing the rich table display.
@nicholasjng
Copy link
Collaborator Author

Works on regular records:

nnbench compare rec1.json rec2.json -P a -P b -C foo
┏━━━━━┳━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━┓
┃ add ┃ mul ┃ Params->a ┃ Params->b ┃ foo ┃
┡━━━━━╇━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━┩
│ 3   │ 2   │ 1         │ 2         │ baz │
│ 4   │ 3   │ 1         │ 3         │ baz │
└─────┴─────┴───────────┴───────────┴─────┘

as well as on records with benchmarks containing errors (example: a sub function throwing an "unsupported" error):

┏━━━━━┳━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━┓
┃ add ┃ mul ┃ sub                       ┃ Params->a ┃ Params->b ┃ foo ┃
┡━━━━━╇━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━┩
│ 3   │ 2   │ ERROR: sub is unsupported │ 1         │ 2         │ bar │
│ 4   │ 3   │ -----                     │ 1         │ 3         │ baz │
└─────┴─────┴───────────────────────────┴───────────┴───────────┴─────┘

@nicholasjng
Copy link
Collaborator Author

Merging this now, the (numerous) follow-ups can be tackled in other issues, but don't have to be implemented for the next release yet.

@nicholasjng nicholasjng merged commit d4e55f0 into main Nov 19, 2024
14 checks passed
@nicholasjng nicholasjng deleted the add-comparison branch November 19, 2024 13:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant