wip: eval how to revamp #525
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
docs/evaluation/how_to_guides/datasets/export_filtered_traces_to_dataset.mdx
Show resolved
Hide resolved
| - [Run an evaluation using the SDK](./how_to_guides/evaluation/evaluate_llm_application) | ||
| - [Run an evaluation asynchronously](./how_to_guides/evaluation/async) | ||
| - [Run an evaluation comparing two experiments](./how_to_guides/evaluation/evaluate_pairwise) | ||
| - [Run an evaluation of a LangChain / LangGraph object](./how_to_guides/evaluation/langchain_runnable) |
There was a problem hiding this comment.
Probably worth it to cover LangChain and LangGraph in separate guides?
There's some more interesting things in LG like evaluating a single node, traectory, etc. It might be worth it to call out the documentation for running a larger eval in the LG eval guide.
We have this tutorial I hacked together with Lance a while back but it's definitely overly complicated and I wonder if bits of it can be adapted to a more concise how to guide https://docs.smith.langchain.com/evaluation/tutorials/agents
There was a problem hiding this comment.
i kinda think this should live in langgraph docs and be linked from here
| ### Run an evaluation | ||
| - [Run an evaluation using the SDK](./how_to_guides/evaluation/evaluate_llm_application) | ||
| - [Run an evaluation asynchronously](./how_to_guides/evaluation/async) | ||
| - [Run an evaluation comparing two experiments](./how_to_guides/evaluation/evaluate_pairwise) |
There was a problem hiding this comment.
nit: Run a comparative evaluation?
There was a problem hiding this comment.
@agola11 can review this one
| const score = rootRun.outputs?.outputs === example.outputs?.output; | ||
| return { key: "correct_label", score }; | ||
| return { key: "correct", score }; |
There was a problem hiding this comment.
Is this code correct? The score doesn't have a corresponding value....
|
|
||
| `, | ||
| typescript` | ||
| // ToDo |
There was a problem hiding this comment.
Do we need to fill out before merging in?
|
|
||
| # How to define an LLM-as-a-judge evaluator | ||
|
|
||
| :::info Key concepts |
There was a problem hiding this comment.
Shouldn't be part of this PR since it'll take too much time, but users want a lot more guidance on good LLM as judge definition than we give rn. Even our concepts doc isn't very helpful.
| typescript` | ||
| import type { EvaluationResult } from "langsmith/evaluation"; | ||
| import type { Run, Example } from "langsmith/schemas"; | ||
|
|
|
|
||
| LangSmith supports both categorical and numerical metrics, and you can return either when writing a [custom evaluator](../../how_to_guides/evaluation/custom_evaluator). | ||
|
|
||
| For an evaluator result to be logged as a numerical metric, it must returned as: |
There was a problem hiding this comment.
this is only true for python, since js can't return just an int, right?
We should be clear about this
| For an evaluator result to be logged as a numerical metric, it must returned as: | ||
|
|
||
| - an `int`, `float`, or `bool` | ||
| - a dict of the form `{"key": "metric_name", "score": int | float | bool}` |
There was a problem hiding this comment.
nit: a feedback[link to reference] dict of the form
docs/evaluation/how_to_guides/evaluation/run_evals_api_only.mdx
Outdated
Show resolved
Hide resolved
| typescript, | ||
| } from "@site/src/components/InstructionsWithCode"; | ||
|
|
||
| # How to run an aggregate evaluation |
There was a problem hiding this comment.
Any way to make it clear to users who want to calculate things like f1 score that they should go here from our indexes? I feel like no one ever finds this page
No description provided.