Skip to content

Commit

Permalink
Fix doctest (#888)
Browse files Browse the repository at this point in the history
  • Loading branch information
hinthornw authored Jul 19, 2024
1 parent c594628 commit 6b094bd
Show file tree
Hide file tree
Showing 2 changed files with 5 additions and 0 deletions.
2 changes: 2 additions & 0 deletions python/langsmith/evaluation/_runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -199,6 +199,7 @@ def evaluate(
Using the `evaluate` API with an off-the-shelf LangChain evaluator:
>>> from langsmith.evaluation import LangChainStringEvaluator
>>> from langchain_openai import ChatOpenAI
>>> def prepare_criteria_data(run: Run, example: Example):
... return {
... "prediction": run.outputs["output"],
Expand All @@ -218,6 +219,7 @@ def evaluate(
... "usefulness": "The prediction is useful if it is correct"
... " and/or asks a useful followup question."
... },
... "llm": ChatOpenAI(model="gpt-4o"),
... },
... prepare_data=prepare_criteria_data,
... ),
Expand Down
3 changes: 3 additions & 0 deletions python/langsmith/evaluation/integrations/_langchain.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,13 +44,15 @@ class LangChainStringEvaluator:
Converting a LangChainStringEvaluator to a RunEvaluator:
>>> from langsmith.evaluation import LangChainStringEvaluator
>>> from langchain_openai import ChatOpenAI
>>> evaluator = LangChainStringEvaluator(
... "criteria",
... config={
... "criteria": {
... "usefulness": "The prediction is useful if"
... " it is correct and/or asks a useful followup question."
... },
... "llm": ChatOpenAI(model="gpt-4o"),
... },
... )
>>> run_evaluator = evaluator.as_run_evaluator()
Expand Down Expand Up @@ -111,6 +113,7 @@ class LangChainStringEvaluator:
... "accuracy": "Score 1: Completely inaccurate\nScore 5: Somewhat accurate\nScore 10: Completely accurate"
... },
... "normalize_by": 10,
... "llm": ChatAnthropic(model="claude-3-opus-20240229"),
... },
... prepare_data=prepare_data,
... )
Expand Down

0 comments on commit 6b094bd

Please sign in to comment.