Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend 'create_judge' command to handle multimodal judges #11

Open
rchan26 opened this issue Apr 17, 2024 · 0 comments
Open

Extend 'create_judge' command to handle multimodal judges #11

rchan26 opened this issue Apr 17, 2024 · 0 comments

Comments

@rchan26
Copy link
Collaborator

rchan26 commented Apr 17, 2024

Currently, the create_judge script only works with evaluating standard LLM outputs taking in the textual prompt and textual response. For evaluating multimodal LLMs, we may need to input media inputted with the prompt into the judge too. Potential easy fix is to add the multimodal field that is in the prompt to the judge evaluation if it exists.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant