Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reexamine the conditions for marking the ratio insignificant #7

Open
andrewrk opened this issue Jun 14, 2023 · 1 comment
Open

reexamine the conditions for marking the ratio insignificant #7

andrewrk opened this issue Jun 14, 2023 · 1 comment
Labels
enhancement New feature or request

Comments

@andrewrk
Copy link
Owner

andrewrk commented Jun 14, 2023

Currently it is done like this:

const is_sig = @fabs(percent) >= 0.01;

but maybe there is better way to do it, that is more well accepted in the field of statistics.

(perhaps std deviation should be involved?)

@andrewrk andrewrk added the enhancement New feature or request label Jun 14, 2023
@matu3ba
Copy link

matu3ba commented Nov 27, 2023

perhaps std deviation should be involved?

  1. To be statistically correct, you would have first to do sufficient retries to estimate, if the standard deviation is the correct guess.
    https://stats.stackexchange.com/questions/108578/what-does-standard-deviation-tell-us-in-non-normal-distribution.
    This may or may not be practical depending on the system behavior (influences of other processes may make the exact distribution unobservable in practice), so it is usually skipped and Gaussean distribution assumed.
  2. The next thing is the to estimate confidence intervals, ie theory here https://medium.com/swlh/a-simple-refresher-on-confidence-intervals-1e29a8580697 and more practical motivation https://www.brainvitamins.net/blog/confidence-intervals-for-benchmarks/, however the Welch test with abbreviated context and optional explanation would be much better, see Welch T-Test #2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants