diff --git a/docs/evaluation/how_to_guides/evaluate_with_attachments.mdx b/docs/evaluation/how_to_guides/evaluate_with_attachments.mdx index a073fbb5..d4a7d0bc 100644 --- a/docs/evaluation/how_to_guides/evaluate_with_attachments.mdx +++ b/docs/evaluation/how_to_guides/evaluate_with_attachments.mdx @@ -37,118 +37,93 @@ The following features are available in the following SDK versions: { const response = await fetch(url); if (!response.ok) { -throw new Error(\`Failed to fetch \${url}\: $\{response.statusText\}\`); -} -return response.arrayBuffer(); + throw new Error(\`Failed to fetch \${url}\: $\{response.statusText\}\`); } - + return response.arrayBuffer(); +}\n // Fetch files as ArrayBuffer const pdfArrayBuffer = await fetchArrayBuffer(pdfUrl); const wavArrayBuffer = await fetchArrayBuffer(wavUrl); -const pngArrayBuffer = await fetchArrayBuffer(pngUrl); - +const pngArrayBuffer = await fetchArrayBuffer(pngUrl);\n // Create the LangSmith client (Ensure LANGCHAIN_API_KEY is set in env) -const langsmithClient = new Client(); - +const langsmithClient = new Client();\n // Create a unique dataset name -const datasetName = "attachment-test-dataset:" + uuid4().substring(0, 8); - +const datasetName = "attachment-test-dataset:" + uuid4().substring(0, 8);\n // Create the dataset const dataset = await langsmithClient.createDataset(datasetName, { -description: "Test dataset for evals with publicly available attachments", -}); - + description: "Test dataset for evals with publicly available attachments", +});\n // Define the example with attachments const exampleId = uuid4(); const example = { -id: exampleId, -inputs: { -audio_question: "What is in this audio clip?", -image_question: "What is in this image?", -}, -outputs: { -audio_answer: -"The sun rises in the east and sets in the west. This simple fact has been observed by humans for thousands of years.", -image_answer: "A mug with a blanket over it.", -}, + id: exampleId, + inputs: { + audio_question: "What is in this audio clip?", + image_question: "What is in this image?", + }, + outputs: { + audio_answer: "The sun rises in the east and sets in the west. This simple fact has been observed by humans for thousands of years.", + image_answer: "A mug with a blanket over it.", + }, attachments: { my_pdf: { mimeType: "application/pdf", @@ -166,10 +141,16 @@ data: pngArrayBuffer }; // Upload the example with attachments to the dataset -await langsmithClient.uploadExamplesMultipart(dataset.id, [example]);`), -]} -groupId="client-language" -/> +await langsmithClient.uploadExamplesMultipart(dataset.id, [example]);`, + `In the TypeScript SDK, you can use the \`uploadExamplesMultipart\` method to upload examples with attachments.\n +Note that this is a different method from the standard \`createExamples\` method, which currently does not support attachments. +Each attachment requires either a \`Uint8Array\` or an \`ArrayBuffer\` as the data type.\n + +- \`Uint8Array\`: Useful for handling binary data directly. +- \`ArrayBuffer\`: Represents fixed-length binary data, which can be converted to \`Uint8Array\` as needed.\n`), + ]} + groupId="client-language" + /> Once you upload examples with attachments, you can view them in the LangSmith UI. Each attachment will be rendered as a file with a preview, making it easy to inspect the contents. ![](./static/attachments_with_examples.png) @@ -177,11 +158,13 @@ Once you upload examples with attachments, you can view them in the LangSmith UI ### From existing runs When adding runs to a LangSmith dataset, attachments can be selectively propagated from the source run to the destination example. -To do learn more, please see [this guide](./manage_datasets_in_application#add-runs-from-the-tracing-project-ui). +To learn more, please see [this guide](./manage_datasets_in_application#add-runs-from-the-tracing-project-ui). + +![](./static/add_trace_with_attachments_to_dataset.png) ### From the LangSmith UI -You can also add examples with attachments from the LangSmith UI. You can do so by clicking the `+ Example` button in the `Examples` tab of the dataset UI. +You can also upload examples with attachments directly from the LangSmith UI. You can do so by clicking the `+ Example` button in the `Examples` tab of the dataset UI. You can then upload the attachments that you want by using the "Upload Files" button: ![](./static/create_example_with_attachments.png) @@ -194,60 +177,36 @@ Once you have a dataset that contains examples with file attachments, you can ru Now that we have a dataset that includes examples with attachments, we can define a target function to run our LLM application on these examples. -:::tip Python Target Function with Attachments -The target function must have two positional arguments in order to consume the attachments associated with the example, the first must be called `inputs` and the second must be called `attachments`. - -- The `inputs` argument is a dictionary that contains the input data for the example, excluding the attachments. -- The `attachments` argument is a dictionary that maps the attachment name to a dictionary containing a presigned url and a reader of the bytes content of the file. Either can be used to read the bytes of the file: - ``` - { - "attachment_name": { - "presigned_url": presigned_url, - "reader": reader - } - } - ``` - ::: - -:::tip Javascript Target Function with Attachments -::: - ### Define custom evaluators with attachments -In addition to using attachments inside of your target function, you can also use them inside of your evaluators as follows: +In addition to using attachments inside of your target function, you can also use them inside of your evaluators as follows. +The exact same rules apply as above to determine whether the evaluator should receive attachments. -## Managing datasets with attachments +## Manage datasets with attachments -### Managing programmatically +### Manage programmatically -In the code above we saw how we could upload examples with attachments using the SDK, and it +In the code [above](#add-examples-with-attachments-to-a-langsmith-dataset), we showed how to add examples with attachments to a dataset. is also possible to update these same examples using the SDK. +As with existing examples, datasets are versioned when you update them with attachments. Therefore, you can navigate to the dataset version history to see the changes made to each example. +To learn more, please see [this guide](./manage_datasets_in_application). + When updating an example with attachments, you can update attachments in a few different ways: - Pass in new attachments @@ -515,6 +495,8 @@ groupId="client-language" Currently, errors are not thrown if you pass the wrong attaachment name to `rename` or `retain`. New attachments **ALWAYS** take precedence over existing attachments. So if you upload a new attachment name "foo" and try to retain or rename an existing attachment to "foo", the new attachment will be used instead. + +Anything not in `rename` or `retain` will be deleted. ::: ### From the LangSmith UI diff --git a/docs/evaluation/how_to_guides/static/attachments_with_examples.png b/docs/evaluation/how_to_guides/static/attachments_with_examples.png index 806057e7..bf11aff1 100644 Binary files a/docs/evaluation/how_to_guides/static/attachments_with_examples.png and b/docs/evaluation/how_to_guides/static/attachments_with_examples.png differ