New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature request/question: Clustered questions vs Independent Questions #2

Open

bfdykstra opened this issue Dec 4, 2024 · 0 comments

bfdykstra commented Dec 4, 2024

Hi there!

Thanks so much for getting this framework started, will definitely be useful for evals :)

In this blog from Anthropic https://www.anthropic.com/research/statistical-approach-to-model-evals and accompanying paper they recommend using a different formula to calculate the standard errors of clustered questions and thus the confidence intervals.

Would you find this useful as a feature for this library? If so, I'd be happy to discuss how you would want to represent the clusters and add a PR

Thanks again!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment