Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

evaluate small fix #1327

Merged
merged 1 commit into from
Dec 12, 2024
Merged

evaluate small fix #1327

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions python/langsmith/evaluation/_arunner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"""V2 Evaluation Interface."""

Check notice on line 1 in python/langsmith/evaluation/_arunner.py

View workflow job for this annotation

GitHub Actions / benchmark

Benchmark results 

...........
WARNING: the benchmark result may be unstable
* the standard deviation (77.4 ms) is 12% of the mean (669 ms)

Try to rerun the benchmark with more runs, values and/or loops.
Run 'python -m pyperf system tune' command to reduce the system jitter.
Use pyperf stats, pyperf dump and pyperf hist to analyze results.
Use --quiet option to hide these warnings.

create_5_000_run_trees: Mean +- std dev: 669 ms +- 77 ms
...........
WARNING: the benchmark result may be unstable
* the standard deviation (185 ms) is 13% of the mean (1.39 sec)

Try to rerun the benchmark with more runs, values and/or loops.
Run 'python -m pyperf system tune' command to reduce the system jitter.
Use pyperf stats, pyperf dump and pyperf hist to analyze results.
Use --quiet option to hide these warnings.

create_10_000_run_trees: Mean +- std dev: 1.39 sec +- 0.19 sec
...........
WARNING: the benchmark result may be unstable
* the standard deviation (188 ms) is 13% of the mean (1.44 sec)

Try to rerun the benchmark with more runs, values and/or loops.
Run 'python -m pyperf system tune' command to reduce the system jitter.
Use pyperf stats, pyperf dump and pyperf hist to analyze results.
Use --quiet option to hide these warnings.

create_20_000_run_trees: Mean +- std dev: 1.44 sec +- 0.19 sec
...........
dumps_class_nested_py_branch_and_leaf_200x400: Mean +- std dev: 706 us +- 12 us
...........
dumps_class_nested_py_leaf_50x100: Mean +- std dev: 25.0 ms +- 0.2 ms
...........
dumps_class_nested_py_leaf_100x200: Mean +- std dev: 105 ms +- 2 ms
...........
dumps_dataclass_nested_50x100: Mean +- std dev: 25.4 ms +- 0.3 ms
...........
WARNING: the benchmark result may be unstable
* the standard deviation (17.4 ms) is 24% of the mean (72.5 ms)

Try to rerun the benchmark with more runs, values and/or loops.
Run 'python -m pyperf system tune' command to reduce the system jitter.
Use pyperf stats, pyperf dump and pyperf hist to analyze results.
Use --quiet option to hide these warnings.

dumps_pydantic_nested_50x100: Mean +- std dev: 72.5 ms +- 17.4 ms
...........
dumps_pydanticv1_nested_50x100: Mean +- std dev: 196 ms +- 3 ms

Check notice on line 1 in python/langsmith/evaluation/_arunner.py

View workflow job for this annotation

GitHub Actions / benchmark

Comparison against main 

+-----------------------------------------------+----------+------------------------+
| Benchmark                                     | main     | changes                |
+===============================================+==========+========================+
| dumps_pydanticv1_nested_50x100                | 225 ms   | 196 ms: 1.15x faster   |
+-----------------------------------------------+----------+------------------------+
| create_5_000_run_trees                        | 732 ms   | 669 ms: 1.09x faster   |
+-----------------------------------------------+----------+------------------------+
| dumps_dataclass_nested_50x100                 | 26.0 ms  | 25.4 ms: 1.03x faster  |
+-----------------------------------------------+----------+------------------------+
| dumps_class_nested_py_leaf_50x100             | 25.5 ms  | 25.0 ms: 1.02x faster  |
+-----------------------------------------------+----------+------------------------+
| create_10_000_run_trees                       | 1.41 sec | 1.39 sec: 1.02x faster |
+-----------------------------------------------+----------+------------------------+
| dumps_class_nested_py_leaf_100x200            | 105 ms   | 105 ms: 1.00x faster   |
+-----------------------------------------------+----------+------------------------+
| dumps_class_nested_py_branch_and_leaf_200x400 | 702 us   | 706 us: 1.01x slower   |
+-----------------------------------------------+----------+------------------------+
| create_20_000_run_trees                       | 1.41 sec | 1.44 sec: 1.02x slower |
+-----------------------------------------------+----------+------------------------+
| dumps_pydantic_nested_50x100                  | 71.0 ms  | 72.5 ms: 1.02x slower  |
+-----------------------------------------------+----------+------------------------+
| Geometric mean                                | (ref)    | 1.03x faster           |
+-----------------------------------------------+----------+------------------------+

from __future__ import annotations

Expand Down Expand Up @@ -844,6 +844,10 @@
f" run {run.id}: {repr(e)}",
exc_info=True,
)
if example.attachments is not None:
for attachment in example.attachments:
reader = example.attachments[attachment]["reader"]
reader.seek(0)
return ExperimentResultRow(
run=run,
example=example,
Expand Down
22 changes: 20 additions & 2 deletions python/tests/integration_tests/test_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1508,7 +1508,20 @@ async def target(
assert image_data.read() == b"fake image data for testing"
return {"answer": "test image"}

async def evaluator(
async def evaluator_1(
outputs: dict, reference_outputs: dict, attachments: dict
) -> Dict[str, Any]:
assert "image" in attachments
assert "presigned_url" in attachments["image"]
image_data = attachments["image"]["reader"]
assert image_data.read() == b"fake image data for testing"
return {
"score": float(
reference_outputs.get("answer") == outputs.get("answer") # type: ignore
)
}

async def evaluator_2(
outputs: dict, reference_outputs: dict, attachments: dict
) -> Dict[str, Any]:
assert "image" in attachments
Expand All @@ -1522,12 +1535,17 @@ async def evaluator(
}

results = await langchain_client.aevaluate(
target, data=dataset_name, evaluators=[evaluator], num_repetitions=2
target,
data=dataset_name,
evaluators=[evaluator_1, evaluator_2],
num_repetitions=2,
max_concurrency=3,
)

assert len(results) == 2
async for result in results:
assert result["evaluation_results"]["results"][0].score == 1.0
assert result["evaluation_results"]["results"][1].score == 1.0

langchain_client.delete_dataset(dataset_name=dataset_name)

Expand Down
Loading