Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add repro script #1311

Draft
wants to merge 9 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions python/bench/tracing_script.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
import os

Check notice on line 1 in python/bench/tracing_script.py

View workflow job for this annotation

GitHub Actions / benchmark

Benchmark results

........... WARNING: the benchmark result may be unstable * the standard deviation (82.1 ms) is 12% of the mean (686 ms) Try to rerun the benchmark with more runs, values and/or loops. Run 'python -m pyperf system tune' command to reduce the system jitter. Use pyperf stats, pyperf dump and pyperf hist to analyze results. Use --quiet option to hide these warnings. create_5_000_run_trees: Mean +- std dev: 686 ms +- 82 ms ........... WARNING: the benchmark result may be unstable * the standard deviation (181 ms) is 13% of the mean (1.42 sec) Try to rerun the benchmark with more runs, values and/or loops. Run 'python -m pyperf system tune' command to reduce the system jitter. Use pyperf stats, pyperf dump and pyperf hist to analyze results. Use --quiet option to hide these warnings. create_10_000_run_trees: Mean +- std dev: 1.42 sec +- 0.18 sec ........... WARNING: the benchmark result may be unstable * the standard deviation (156 ms) is 11% of the mean (1.43 sec) Try to rerun the benchmark with more runs, values and/or loops. Run 'python -m pyperf system tune' command to reduce the system jitter. Use pyperf stats, pyperf dump and pyperf hist to analyze results. Use --quiet option to hide these warnings. create_20_000_run_trees: Mean +- std dev: 1.43 sec +- 0.16 sec ........... dumps_class_nested_py_branch_and_leaf_200x400: Mean +- std dev: 693 us +- 10 us ........... dumps_class_nested_py_leaf_50x100: Mean +- std dev: 25.1 ms +- 0.3 ms ........... dumps_class_nested_py_leaf_100x200: Mean +- std dev: 104 ms +- 4 ms ........... dumps_dataclass_nested_50x100: Mean +- std dev: 25.3 ms +- 0.4 ms ........... WARNING: the benchmark result may be unstable * the standard deviation (16.8 ms) is 24% of the mean (71.4 ms) Try to rerun the benchmark with more runs, values and/or loops. Run 'python -m pyperf system tune' command to reduce the system jitter. Use pyperf stats, pyperf dump and pyperf hist to analyze results. Use --quiet option to hide these warnings. dumps_pydantic_nested_50x100: Mean +- std dev: 71.4 ms +- 16.8 ms ........... dumps_pydanticv1_nested_50x100: Mean +- std dev: 197 ms +- 3 ms

Check notice on line 1 in python/bench/tracing_script.py

View workflow job for this annotation

GitHub Actions / benchmark

Comparison against main

+-----------------------------------------------+----------+------------------------+ | Benchmark | main | changes | +===============================================+==========+========================+ | dumps_pydanticv1_nested_50x100 | 224 ms | 197 ms: 1.14x faster | +-----------------------------------------------+----------+------------------------+ | create_5_000_run_trees | 722 ms | 686 ms: 1.05x faster | +-----------------------------------------------+----------+------------------------+ | dumps_class_nested_py_branch_and_leaf_200x400 | 706 us | 693 us: 1.02x faster | +-----------------------------------------------+----------+------------------------+ | dumps_class_nested_py_leaf_50x100 | 25.5 ms | 25.1 ms: 1.02x faster | +-----------------------------------------------+----------+------------------------+ | dumps_dataclass_nested_50x100 | 25.7 ms | 25.3 ms: 1.02x faster | +-----------------------------------------------+----------+------------------------+ | dumps_class_nested_py_leaf_100x200 | 104 ms | 104 ms: 1.00x faster | +-----------------------------------------------+----------+------------------------+ | create_20_000_run_trees | 1.42 sec | 1.43 sec: 1.01x slower | +-----------------------------------------------+----------+------------------------+ | create_10_000_run_trees | 1.37 sec | 1.42 sec: 1.03x slower | +-----------------------------------------------+----------+------------------------+ | dumps_pydantic_nested_50x100 | 68.6 ms | 71.4 ms: 1.04x slower | +-----------------------------------------------+----------+------------------------+ | Geometric mean | (ref) | 1.02x faster | +-----------------------------------------------+----------+------------------------+
os.environ["LANGCHAIN_PROJECT"] = "llm_messages_test_py"
os.environ["LANGSMITH_USE_PYO3_CLIENT"] = "true"

from langsmith import traceable

@traceable
def format_prompt(subject):
return [
{
"role": "system",
"content": "You are a helpful assistant.",
},
{
"role": "user",
"content": f"What's a good name for a store that sells {subject}?"
}
]

@traceable(run_type="llm")
def invoke_llm(messages):
return {
"choices": [
{
"message": {
"role": "assistant",
"content": "Sure, how about 'Rainbow Socks'?"
}
}
]
}

@traceable
def parse_output(response):
return response["choices"][0]["message"]["content"]

@traceable
def run_pipeline():
messages = format_prompt("colorful socks")
response = invoke_llm(messages)
return parse_output(response)

if __name__ == "__main__":
run_pipeline()
21 changes: 21 additions & 0 deletions python/langsmith/_internal/_operations.py
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,27 @@ def serialize_run_dict(
attachments=attachments if attachments is not None else None,
)

def serialize_run_dict_for_compressed_ingest(
agola11 marked this conversation as resolved.
Show resolved Hide resolved
operation: Literal["post", "patch"], payload: dict
):
inputs = payload.pop("inputs", None)
outputs = payload.pop("outputs", None)
events = payload.pop("events", None)
attachments = payload.pop("attachments", None)
serialized = ...
extra = ...
return SerializedRunOperation(
operation=operation,
id=payload["id"],
trace_id=payload["trace_id"],
_none=_dumps_json(payload),
inputs=_dumps_json(inputs) if inputs is not None else None,
outputs=_dumps_json(outputs) if outputs is not None else None,
events=_dumps_json(events) if events is not None else None,
attachments=attachments if attachments is not None else None,
)



def combine_serialized_queue_operations(
ops: list[Union[SerializedRunOperation, SerializedFeedbackOperation]],
Expand Down
15 changes: 9 additions & 6 deletions python/langsmith/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -1287,9 +1287,10 @@ def create_run(
run_create.get("trace_id") is not None
and run_create.get("dotted_order") is not None
):
if self._pyo3_client is not None:
self._pyo3_client.create_run(run_create)
elif self.tracing_queue is not None:
# if self._pyo3_client is not None:
# print("RUN_CREATE", run_create)
# self._pyo3_client.create_run(run_create)
if self.tracing_queue is not None:
serialized_op = serialize_run_dict("post", run_create)
self.tracing_queue.put(
TracingQueueItem(run_create["dotted_order"], serialized_op)
Expand Down Expand Up @@ -1753,9 +1754,11 @@ def update_run(
if data["extra"]:
self._insert_runtime_env([data])

if self._pyo3_client is not None:
self._pyo3_client.update_run(data)
elif use_multipart and self.tracing_queue is not None:
print("UPDATE_RUN", data)

# if self._pyo3_client is not None:
# self._pyo3_client.update_run(data)
if use_multipart and self.tracing_queue is not None:
# not collecting attachments currently, use empty dict
serialized_op = serialize_run_dict(operation="patch", payload=data)
self.tracing_queue.put(
Expand Down
Loading