Skip to content

Commit

Permalink
chore(langchain): disable flaky tests
Browse files Browse the repository at this point in the history
There appear to be stability issues with using snapshots and/or
LangChain in general.

There are failures in the mocked tests that look like:

```
builtins.AssertionError: assert 0 == 1
 +  where 0 = <MagicMock name='LLMObsSpanWriter().enqueue' id='127482146130048'>.call_count
 +    where <MagicMock name='LLMObsSpanWriter().enqueue' id='127482146130048'> = <MagicMock name='LLMObsSpanWriter()' id='127482147073440'>.enqueue
```

as well as failures with snapshot based tests:

```
builtins.Failed: At request <Request GET /test/session/snapshot >:
   At snapshot (token='tests.contrib.langchain.test_langchain_community.test_lcel_chain_simple'):
    - Directory: /go/src/github.com/DataDog/apm-reliability/dd-trace-py/tests/snapshots
    - CI mode: 1
    - Trace File: /go/src/github.com/DataDog/apm-reliability/dd-trace-py/tests/snapshots/tests.contrib.langchain.test_langchain_community.test_lcel_chain_simple.json
    - Stats File: /go/src/github.com/DataDog/apm-reliability/dd-trace-py/tests/snapshots/tests.contrib.langchain.test_langchain_community.test_lcel_chain_simple_tracestats.json
    At compare of 1 expected trace(s) to 0 received trace(s):
Did not receive expected traces: 'langchain.request'
```

While we investigate a more stable method of testing it makes sense to
disable the tests to avoid noise to our neighbours in the library :).

DOWN WITH FLAKY TESTS
  • Loading branch information
Kyle-Verhoog committed Nov 22, 2024
1 parent d792c3d commit 971dad7
Show file tree
Hide file tree
Showing 2 changed files with 4 additions and 0 deletions.
2 changes: 2 additions & 0 deletions tests/contrib/langchain/test_langchain_community.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ def request_vcr():
yield get_request_vcr(subdirectory_name="langchain_community")


'''
@pytest.mark.parametrize("ddtrace_config_langchain", [dict(logs_enabled=True, log_prompt_completion_sample_rate=1.0)])
def test_global_tags(ddtrace_config_langchain, langchain_openai, request_vcr, mock_metrics, mock_logs, mock_tracer):
"""
Expand Down Expand Up @@ -1535,3 +1536,4 @@ def circumference_tool(radius: float) -> float:
)
calculator.invoke("2", config={"unserializable": object()})
'''
2 changes: 2 additions & 0 deletions tests/contrib/langchain/test_langchain_llmobs.py
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,7 @@ def _invoke_tool(cls, tool, tool_input, config, mock_tracer):
return mock_tracer.pop_traces()[0][0]


'''
@pytest.mark.skipif(LANGCHAIN_VERSION >= (0, 1), reason="These tests are for langchain < 0.1.0")
class TestLLMObsLangchain(BaseTestLLMObsLangchain):
cassette_subdirectory_name = "langchain"
Expand Down Expand Up @@ -1185,3 +1186,4 @@ def test_llmobs_with_anthropic_disabled(self):
self._call_anthropic_chat(ChatAnthropic)
self._assert_trace_structure_from_writer_call_args(["llm"])
'''

0 comments on commit 971dad7

Please sign in to comment.