-
Notifications
You must be signed in to change notification settings - Fork 42
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Since using things starting with test can make pytest unhappy
- Loading branch information
Showing
1 changed file
with
13 additions
and
13 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -9,12 +9,12 @@ We recommend using LangSmith to track any unit tests, end-to-end integration tes | |
These should run on every commit in your CI pipeline to catch regressions early. | ||
|
||
:::note | ||
`@test` currently requires `langsmith` python version `>=0.1.74` (named `@unit` for versions `>=0.1.42`). If you are interested in unit testing functionality in TypeScript or other languages, please let us know at [[email protected]](mailto:[email protected]). | ||
`@unit` currently requires `langsmith` python version `>=0.1.74` (named `@unit` for versions `>=0.1.42`). If you are interested in unit testing functionality in TypeScript or other languages, please let us know at [[email protected]](mailto:[email protected]). | ||
::: | ||
|
||
## Write a @test | ||
## Write a @unit | ||
|
||
To write a LangSmith functional test, decorate your test function with `@test`. | ||
To write a LangSmith functional test, decorate your test function with `@unit`. | ||
If you want to track the full nested trace of the system or component being tested, you can mark those functions with `@traceable`. For example: | ||
|
||
```python | ||
|
@@ -35,7 +35,7 @@ Then define your test: | |
from langsmith import test | ||
from my_app.main import generate_sql | ||
|
||
@test | ||
@unit | ||
def test_sql_generation_select_all(): | ||
user_query = "Get all users from the customers table" | ||
sql = generate_sql(user_query) | ||
|
@@ -61,7 +61,7 @@ The test suite syncs to a corresponding dataset named after your package or gith | |
|
||
## Going further | ||
|
||
`@test` is designed to stay out of your way and works well with familiar `pytest` features. For example: | ||
`@unit` is designed to stay out of your way and works well with familiar `pytest` features. For example: | ||
|
||
#### Defining inputs as fixtures | ||
|
||
|
@@ -80,7 +80,7 @@ def expected_sql(): | |
|
||
# output_keys indicate which test arguments to save as 'outputs' in the dataset (Optional) | ||
# Otherwise, all arguments are saved as 'inputs' | ||
@test(output_keys=["expected_sql"]) | ||
@unit(output_keys=["expected_sql"]) | ||
def test_sql_generation_with_fixture(user_query, expected_sql): | ||
sql = generate_sql(user_query) | ||
assert sql == expected_sql | ||
|
@@ -91,7 +91,7 @@ def test_sql_generation_with_fixture(user_query, expected_sql): | |
Parametrizing tests lets you run the same assertions across multiple sets of inputs. Use `pytest`'s `parametrize` decorator to achieve this. For example: | ||
|
||
```python | ||
@test | ||
@unit | ||
@pytest.mark.parametrize( | ||
"user_query, expected_sql", | ||
[ | ||
|
@@ -113,7 +113,7 @@ LangSmith provides an `expect` utility to help define expectations about your LL | |
```python | ||
from langsmith import expect | ||
|
||
@test | ||
@unit | ||
def test_sql_generation_select_all(): | ||
user_query = "Get all users from the customers table" | ||
sql = generate_sql(user_query) | ||
|
@@ -125,7 +125,7 @@ This will log the binary "expectation" score to the experiment results, addition | |
`expect` also provides "fuzzy match" methods. For example: | ||
|
||
```python | ||
@test | ||
@unit | ||
@pytest.mark.parametrize( | ||
"query, expectation", | ||
[ | ||
|
@@ -188,7 +188,7 @@ LANGCHAIN_TEST_CACHE=tests/cassettes ptw tests/my_llm_tests | |
|
||
## Explanations | ||
|
||
The `@test` test decorator converts any test into a parametrized LangSmith example. By default, all tests within a given file will be grouped as a single "test suite" with a corresponding dataset. You can configure which test suite a test belongs to by passing the `test_suite_name` parameter to `@test`. | ||
The `@unit` test decorator converts any test into a parametrized LangSmith example. By default, all tests within a given file will be grouped as a single "test suite" with a corresponding dataset. You can configure which test suite a test belongs to by passing the `test_suite_name` parameter to `@unit`. | ||
|
||
The following metrics are available off-the-shelf: | ||
|
||
|
@@ -207,7 +207,7 @@ from langsmith.run_helpers import get_current_run_tree | |
|
||
client = Client() | ||
|
||
@test | ||
@unit | ||
def test_foo(): | ||
run_tree = get_current_run_tree() | ||
client.create_feedback(run_id=run_tree.id, key="my_custom_feedback", score=1) | ||
|
@@ -306,9 +306,9 @@ Assert the expectation value against a custom function. | |
|
||
### `test` API | ||
|
||
The `@test` decorator is used to mark a function as a test case for LangSmith. It ensures that the necessary example data is created and associated with the test function. The decorated function will be executed as a test case, and the results will be recorded and reported by LangSmith. | ||
The `@unit` decorator is used to mark a function as a test case for LangSmith. It ensures that the necessary example data is created and associated with the test function. The decorated function will be executed as a test case, and the results will be recorded and reported by LangSmith. | ||
|
||
#### `@test(id=None, output_keys=None, client=None, test_suite_name=None)` | ||
#### `@unit(id=None, output_keys=None, client=None, test_suite_name=None)` | ||
|
||
Create a test case in LangSmith. | ||
|
||
|