Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend --magnitude to support specific metrics #2

Open
smccarthy788 opened this issue Dec 2, 2021 · 0 comments
Open

Extend --magnitude to support specific metrics #2

smccarthy788 opened this issue Dec 2, 2021 · 0 comments
Labels
enhancement New feature or request

Comments

@smccarthy788
Copy link
Collaborator

Users report wanting to filter different metrics at different magnitudes (e.g. p99 vs p50). However, the --magnitude argument filters across all metrics.

One option could be to allow setting multiple metric=min_magnitude pairs with the --magnitude flag, or exclusively a single float. This would be pretty verbose... Ex:

hunter ... --magnitude p50=0.05 --magnitude p99=0.2

Another option would be setting the min_magnitude in the config for each test's metric, with support for inheritance so each test doesn't need to define it. This option should also support overriding the config.

templates:
  common_metrics:
    metrics:
      p50:
        scale: 1.0e-6
        direction: -1
        min_magnitude: 0.05
      p99:
        scale: 1.0e-6
        direction: -1
        min_magnitude: 0.2
tests:
  local2:
    type: csv
    file: tests/resources/sample.csv
    time_column: time
    metrics:
      m1:
        column: metric1
        direction: 1
      m2:
        column: metric2
        direction: -1
        min_magnitude: 0.99 # Be able to set min_magnitude for specific tests
    attributes: [ commit ]

  remote1:
    inherit: [common_metrics] # min_magnitude set via inheritance
    prefix: "performance_regressions.my_product.%{BRANCH}.test1"
@pkolaczk pkolaczk added the enhancement New feature or request label Mar 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants