Skip to content

Commit

Permalink
add docs for annotation queue rubrics
Browse files Browse the repository at this point in the history
  • Loading branch information
samnoyes committed Dec 4, 2024
1 parent a693c2a commit dd9e08a
Show file tree
Hide file tree
Showing 4 changed files with 18 additions and 1 deletion.
19 changes: 18 additions & 1 deletion docs/evaluation/how_to_guides/annotation_queues.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,28 @@ While you can always [annotate runs inline](./annotate_traces_inline), annotatio
To create an annotation queue, navigate to the **Annotation queues** section through the homepage or left-hand navigation bar.
Then click **+ New annotation queue** in the top right corner.

![](./static/annotation_queue_form.png)
![](./static/create_annotation_queue_new.png)

### Basic Details

Fill in the form with the **name** and **description** of the queue.
You can also assign a **default dataset** to queue, which will streamline the process of sending the inputs and outputs of certain runs to datasets in your LangSmith workspace.

### Annotation Rubric

First, type some high-level instructions for your annotators, which will be shown in the sidebar on every run.

Next, click "+ Desired Feedback" to add feedback keys to your annotation queue. Annotators will be presented with these feedback keys on each run.
Add a description for each, as well as a short description of each category if the feedback is categorical.

![annotation queue rubric](./static/create_annotation_rubric.png)

Reviewers will see this:

![rubric for annotators](./static/rubric_for_annotators.png)

### Collaborator Settings

There are a few settings related to multiple annotators:

- **Number of reviewers per run**: This determines the number of reviewers that must mark a run as "Done" for it to be removed from the queue. If you check "All workspace members review each run," then a run will remain in the queue until all workspace members have marked it "Done".
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit dd9e08a

Please sign in to comment.