Generating Outputs
As described on the Data Management page, we generally assume that you have externally generated model outputs.
However, factgenie also enables you to generate the outputs for your datasets without leaving the web interface!
Generating the outputs is based on prompting LLMs through an API. It is therefore very similar to the process of 🤖 gathering LLM annotations to which we point for more details.
Setting up the generation campaign is analogous to the evaluation campaign:
- Go to the
/llm_genpage and click on New LLM campaign. - Insert a unique campaign ID .
- Configure the LLM generator.
- Select the datasets and splits you want to annotate.
Compared to the evaluation campaign, there is only the following extra field in the configuration dialog:
-
Start with: A string that the model needs to use at the beginning of its generated text.
- This can be used, e.g., for easier parsing of the model outputs by prefixing them with
Here is the answer:\n"and stripping the final quote.
- This can be used, e.g., for easier parsing of the model outputs by prefixing them with
On the contrary, you will not find here the Annotation categories as they are not needed for generation.
For including the raw input data in the prompt, use the {data} placeholder.
The model outputs are received as a plain text, so you do not need to provide any specific instructions in the prompt.
You can find (and customize) the campaign files in factgenie/campaigns/<llm-eval-id>.
You can see the generated outputs immediately on the campaign detail page:
The generated outputs do not automatically appear in the Browse page or among model outputs.
To export the outputs from the campaign, click on Save as model outputs:
After filling the setup id, factgenie will make the outputs visible under this id in the browse interface:
Alternatively, you can create and run a generation campaign from the command line. See the page on command line interface.