New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FR] Add more test cases #5

Open

svilupp opened this issue Feb 8, 2024 · 1 comment

Owner

svilupp commented Feb 8, 2024

It would be great to add more test cases across different areas.

It should be:

widely applicable, ie, something that a common Julia user can come across
measurable (to be able to score it), eg, unit tests etc
well-defined (eg, have a clear answer and ideally very little ambiguity)
be "safe" (ie, they shouldn't require any unsafe functionality like distributed computing, threading, or in any way extending functions in the Base/std libs) - because it can change the environment and break things

See more on contributing: https://github.com/svilupp/Julia-LLM-Leaderboard?tab=readme-ov-file#contributing-your-test-case

Owner Author

svilupp commented Feb 8, 2024

At the moment, we have 14 test cases across 2 categories. The ideal number would be at least 5 categories with at least 5 test cases each.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment