Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
Since using things starting with test can make pytest unhappy
  • Loading branch information
hinthornw authored Aug 30, 2024
2 parents 3e3aefe + ea129d7 commit f4e165f
Showing 1 changed file with 13 additions and 13 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,12 @@ We recommend using LangSmith to track any unit tests, end-to-end integration tes
These should run on every commit in your CI pipeline to catch regressions early.

:::note
`@test` currently requires `langsmith` python version `>=0.1.74` (named `@unit` for versions `>=0.1.42`). If you are interested in unit testing functionality in TypeScript or other languages, please let us know at [[email protected]](mailto:[email protected]).
`@unit` currently requires `langsmith` python version `>=0.1.74` (named `@unit` for versions `>=0.1.42`). If you are interested in unit testing functionality in TypeScript or other languages, please let us know at [[email protected]](mailto:[email protected]).
:::

## Write a @test
## Write a @unit

To write a LangSmith functional test, decorate your test function with `@test`.
To write a LangSmith functional test, decorate your test function with `@unit`.
If you want to track the full nested trace of the system or component being tested, you can mark those functions with `@traceable`. For example:

```python
Expand All @@ -35,7 +35,7 @@ Then define your test:
from langsmith import test
from my_app.main import generate_sql

@test
@unit
def test_sql_generation_select_all():
user_query = "Get all users from the customers table"
sql = generate_sql(user_query)
Expand All @@ -61,7 +61,7 @@ The test suite syncs to a corresponding dataset named after your package or gith

## Going further

`@test` is designed to stay out of your way and works well with familiar `pytest` features. For example:
`@unit` is designed to stay out of your way and works well with familiar `pytest` features. For example:

#### Defining inputs as fixtures

Expand All @@ -80,7 +80,7 @@ def expected_sql():

# output_keys indicate which test arguments to save as 'outputs' in the dataset (Optional)
# Otherwise, all arguments are saved as 'inputs'
@test(output_keys=["expected_sql"])
@unit(output_keys=["expected_sql"])
def test_sql_generation_with_fixture(user_query, expected_sql):
sql = generate_sql(user_query)
assert sql == expected_sql
Expand All @@ -91,7 +91,7 @@ def test_sql_generation_with_fixture(user_query, expected_sql):
Parametrizing tests lets you run the same assertions across multiple sets of inputs. Use `pytest`'s `parametrize` decorator to achieve this. For example:

```python
@test
@unit
@pytest.mark.parametrize(
"user_query, expected_sql",
[
Expand All @@ -113,7 +113,7 @@ LangSmith provides an `expect` utility to help define expectations about your LL
```python
from langsmith import expect

@test
@unit
def test_sql_generation_select_all():
user_query = "Get all users from the customers table"
sql = generate_sql(user_query)
Expand All @@ -125,7 +125,7 @@ This will log the binary "expectation" score to the experiment results, addition
`expect` also provides "fuzzy match" methods. For example:

```python
@test
@unit
@pytest.mark.parametrize(
"query, expectation",
[
Expand Down Expand Up @@ -188,7 +188,7 @@ LANGCHAIN_TEST_CACHE=tests/cassettes ptw tests/my_llm_tests

## Explanations

The `@test` test decorator converts any test into a parametrized LangSmith example. By default, all tests within a given file will be grouped as a single "test suite" with a corresponding dataset. You can configure which test suite a test belongs to by passing the `test_suite_name` parameter to `@test`.
The `@unit` test decorator converts any test into a parametrized LangSmith example. By default, all tests within a given file will be grouped as a single "test suite" with a corresponding dataset. You can configure which test suite a test belongs to by passing the `test_suite_name` parameter to `@unit`.

The following metrics are available off-the-shelf:

Expand All @@ -207,7 +207,7 @@ from langsmith.run_helpers import get_current_run_tree

client = Client()

@test
@unit
def test_foo():
run_tree = get_current_run_tree()
client.create_feedback(run_id=run_tree.id, key="my_custom_feedback", score=1)
Expand Down Expand Up @@ -306,9 +306,9 @@ Assert the expectation value against a custom function.

### `test` API

The `@test` decorator is used to mark a function as a test case for LangSmith. It ensures that the necessary example data is created and associated with the test function. The decorated function will be executed as a test case, and the results will be recorded and reported by LangSmith.
The `@unit` decorator is used to mark a function as a test case for LangSmith. It ensures that the necessary example data is created and associated with the test function. The decorated function will be executed as a test case, and the results will be recorded and reported by LangSmith.

#### `@test(id=None, output_keys=None, client=None, test_suite_name=None)`
#### `@unit(id=None, output_keys=None, client=None, test_suite_name=None)`

Create a test case in LangSmith.

Expand Down

0 comments on commit f4e165f

Please sign in to comment.