diff --git a/docs/evaluation/how_to_guides/annotation_queues.mdx b/docs/evaluation/how_to_guides/annotation_queues.mdx index 305bd598..400b82ee 100644 --- a/docs/evaluation/how_to_guides/annotation_queues.mdx +++ b/docs/evaluation/how_to_guides/annotation_queues.mdx @@ -14,11 +14,28 @@ While you can always [annotate runs inline](./annotate_traces_inline), annotatio To create an annotation queue, navigate to the **Annotation queues** section through the homepage or left-hand navigation bar. Then click **+ New annotation queue** in the top right corner. -![](./static/annotation_queue_form.png) +![](./static/create_annotation_queue_new.png) + +### Basic Details Fill in the form with the **name** and **description** of the queue. You can also assign a **default dataset** to queue, which will streamline the process of sending the inputs and outputs of certain runs to datasets in your LangSmith workspace. +### Annotation Rubric + +First, type some high-level instructions for your annotators, which will be shown in the sidebar on every run. + +Next, click "+ Desired Feedback" to add feedback keys to your annotation queue. Annotators will be presented with these feedback keys on each run. +Add a description for each, as well as a short description of each category if the feedback is categorical. + +![annotation queue rubric](./static/create_annotation_rubric.png) + +Reviewers will see this: + +![rubric for annotators](./static/rubric_for_annotators.png) + +### Collaborator Settings + There are a few settings related to multiple annotators: - **Number of reviewers per run**: This determines the number of reviewers that must mark a run as "Done" for it to be removed from the queue. If you check "All workspace members review each run," then a run will remain in the queue until all workspace members have marked it "Done". diff --git a/docs/evaluation/how_to_guides/static/create_annotation_queue_new.png b/docs/evaluation/how_to_guides/static/create_annotation_queue_new.png new file mode 100644 index 00000000..c4e506c5 Binary files /dev/null and b/docs/evaluation/how_to_guides/static/create_annotation_queue_new.png differ diff --git a/docs/evaluation/how_to_guides/static/create_annotation_rubric.png b/docs/evaluation/how_to_guides/static/create_annotation_rubric.png new file mode 100644 index 00000000..b84dff51 Binary files /dev/null and b/docs/evaluation/how_to_guides/static/create_annotation_rubric.png differ diff --git a/docs/evaluation/how_to_guides/static/rubric_for_annotators.png b/docs/evaluation/how_to_guides/static/rubric_for_annotators.png new file mode 100644 index 00000000..ae5193cf Binary files /dev/null and b/docs/evaluation/how_to_guides/static/rubric_for_annotators.png differ