FEAT: Gradio HiTL Scorer #722
Open
+609
−1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR introduces a new UI for scoring using Gradio, which improves the user experience by running in a separate process.
UI Implementation
The new UI is implemented using Gradio and runs in a different process. This allows the UI to wait on scoring tasks and respond accordingly, preventing the UI from flashing and ensuring it remains on screen throughout the scoring process.
Communication Mechanism
The communication between the UI and the scoring process is done using rpyc, which implements an RPC mechanism on localhost.
Gradio Features
Gradio allows us to display more than just messages. We can display entire conversations or even multimedia content. By default, the Gradio HiTL will be launched in a web view, but the web browser can be opened by setting the argument
open_browserto true.Platform Support
Currently, this implementation supports Windows and is limited to string scoring. Future updates will include support for other platforms and content types.
Dependency Changes
The dependency
aiofileswas downgraded from 24.1.0 to 23.2.1 to resolve a compatibility issue with Gradio. No issues were identified with this downgrade in PyRIT.Tests and Documentation
I am looking for feedback on the types of tests (unit, integration, etc.) and documentation (user guide, API reference, etc.) that should be added for this PR. This scorer is meant as a drop-in replacement for the current HiTL scorer.