Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FEAT added support for scorers powered by thinking models #719

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

joaodunas
Copy link

Description

Changed the function _score_value_with_llm (pyrit/score/scorer.py) to support thinking models and any cases where the scorer model does not output only the requested json.

Tests and Documentation

Haven't made any tests as of yet, mainly due to lack of knowledge regarding how to do it and lack of time

@romanlutz
Copy link
Contributor

Thanks for this submission.

Can you elaborate on the problem? The code doesn't really tell me what you are running into.

What do you mean by "thinking"? Perhaps the reasoning steps models like o1 take?

In general, I think it's best to start with an issue and decide there whether or not a PR is needed, and what the change should be. It usually saves you time because maintainers can advise on what the change should look like. For example, making this change would break a lot of functionality in PyRIT as existing model endpoints depend on that logic.

Let me know what you think so we can figure out a solution for the problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants