python[minor]: pytest integration

[baskaryan]

Log pytest tests to LangSmith. Useful for:

• Evaluations where the eval logic is different for each datapoint, making it difficult to use generic evaluators on a whole dataset
• Unit tests where you want both the local pytest experience and the ability to trace and share results

Install

pip install "langsmith[pytest]==0.2.11rc7"

Simple usage

# tests/test_simple.py
import pytest
from langsmith import testing as t

@pytest.mark.langsmith
def test_addition_single():
    x = 3
    y = 4
    # directly log example inputs if you don't want to use fixtures
    t.log_inputs({"x": x, "y": y})

    expected = 7
    # directly log example outputs if you don't want to use fixtures
    t.log_reference_outputs({"sum": expected})

    actual = x + y
    # directly log run outputs
    t.log_outputs({"sum": actual})
    
    # test pass/fail automatically logged to langsmith
    assert actual == expected

Run

pytest --outputs='ls' tests/test_foo.py

Results

Advanced usage

#tests/test_advanced.py
import openai
import pytest

from langsmith import wrappers
from langsmith import testing as t

oai_client = wrappers.wrap_openai(openai.Client())

@pytest.mark.langsmith
def test_openai_says_hello():
    # Traced code will be included in the test case
    text = "Say hello!"
    response = oai_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": text},
        ],
    )
    t.log_inputs({"text": text})
    t.log_outputs({"response": response.choices[0].message.content})
    t.log_reference_outputs({"response": "hello!"})

    # Use this context manager to trace any steps used for generating evaluation 
    # feedback separately from the main application logic
    with t.trace_feedback():
        grade = oai_client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {
                    "role": "system",
                    "content": "Return 1 if 'hello' is in the user message and 0 otherwise.",
                },
                {"role": "user", "content": response.choices[0].message.content},
            ],
        )
        t.log_feedback(
            key="llm_judge", score=float(grade.choices[0].message.content)
        )

    assert "hello" in response.choices[0].message.content.lower()


@pytest.mark.langsmith(output_keys=["expected"])
@pytest.mark.parametrize(
    "a, b, expected",
    [(1, 2, 3), (3, 4, 7)],
)
def test_addition_parametrized(a: int, b: int, expected: int):
    t.log_outputs({"sum": a + b})
    assert a + b == expected

Run using pytest-xdist to parallelize (pip install pytest-xdist first)

LANGSMITH_TEST_SUITE="Test suite name" LANGSMITH_EXPERIMENT="Experiment name" pytest  --outputs='ls' tests

Results: https://dev.smith.langchain.com/public/cea0e7fd-2d27-47d1-8ada-141069acdf0d/d

[hinthornw]

I like the functionality. I kinda like using a single module/object to access the methods, like

@unit
def test_example(captest):
    x = 0
    y = 1
    captest.log_inputs({"x": x, "y": y})
    captest.log_reference({"product": 0})
    captest.log_outputs({"product": x * y})
    assert x * y == 0

or something

[hinthornw]

I like

[jxnl]

why is it "reference" input vs

target, expect, etc

[baskaryan]

why is it "reference" input vs

target, expect, etc

"reference outputs" is what we call this when defining evaluators for evaluate() and how we display this info in experiments UI, want to be consistent with existing flows

historically this decision was made bc target/expected seemed a bit too narrow/prescriptive when this can be any information relevant for evaluating outputs, not necessarily the actual expected outputs. eg could just be agent trajectories not actual agent outputs. but see the argument that target/expected are more common terms and it wasn't worth us introducing a new term for such a small nuance

[baskaryan]

needs review, very open to less hacky ideas

[agola11]

can't think of any great solutions right now without async. Would add a detailed comment about this and address later as a TODO

[baskaryan]

worth review

[baskaryan]

worth review

[agola11]

looks reasonable to me

[agola11]

just one main comment

[agola11]

why do we need to submit this in an executor? run_tree.patch calls update_run which is async already

[agola11]

can't think of any great solutions right now without async. Would add a detailed comment about this and address later as a TODO

[agola11]

looks reasonable to me