Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Uptrain integration tests failing #464

Closed
masci opened this issue Feb 22, 2024 · 1 comment
Closed

Uptrain integration tests failing #464

masci opened this issue Feb 22, 2024 · 1 comment
Labels
bug Something isn't working integration:uptrain

Comments

@masci
Copy link
Contributor

masci commented Feb 22, 2024

Describe the bug
I have OPENAI_API_KEY set.

hatch run test fails 4 tests with:

FAILED tests/test_evaluator.py::test_integration_run[critique_language-inputs7-None] - ValueError: UpTrain evaluator did not return an expected output for metric 'fluency'
FAILED tests/test_evaluator.py::test_integration_run[critique_tone-inputs8-metric_params8] - ValueError: UpTrain evaluator did not return an expected output for metric 'tone'
FAILED tests/test_evaluator.py::test_integration_run[guideline_adherence-inputs9-metric_params9] - polars.exceptions.SchemaFieldNotFoundError: score_somename_adherence
FAILED tests/test_evaluator.py::test_integration_run[response_matching-inputs10-metric_params10] - ValueError: Row 0 is missing required all required attributes for evaluation: {'ground_truth', 'question', 'response'}

Describe your environment (please complete the following information):

  • OS: [e.g. iOS]
  • Haystack version:
  • Integration version:
@masci masci added bug Something isn't working integration:uptrain labels Feb 22, 2024
@julian-risch
Copy link
Member

We released version 0.4.1 of our uptrain-haystack integration now, where we pin the uptrain version to 0.5.0. The issue does not occur with that older version of uptrain and integration tests are very slow but run through.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working integration:uptrain
Projects
None yet
Development

No branches or pull requests

2 participants