Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: end2end evaluation tests #7601

Merged
merged 17 commits into from
Apr 26, 2024
Merged

test: end2end evaluation tests #7601

merged 17 commits into from
Apr 26, 2024

Conversation

davidsbatista
Copy link
Contributor

Related Issues

Proposed Changes:

  • Adding end2end tests over an evaluation pipeline

Notes for the reviewer

I've opted for:

  • 3 questions and 4 documents
  • simple basic RAG pipelines x2 (for the detailed comparative scores)
  • the evaluation pipeline has all the current evaluators except for LLMEvaluator and AnswerExactMatchEvaluator

Checklist

@davidsbatista davidsbatista requested review from a team as code owners April 26, 2024 09:21
@davidsbatista davidsbatista requested review from dfokina and shadeMe and removed request for a team April 26, 2024 09:21
@github-actions github-actions bot added the type:documentation Improvements on the docs label Apr 26, 2024
@davidsbatista davidsbatista added the ignore-for-release-notes PRs with this flag won't be included in the release notes. label Apr 26, 2024
@coveralls
Copy link
Collaborator

coveralls commented Apr 26, 2024

Pull Request Test Coverage Report for Build 8849494576

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 90.096%

Totals Coverage Status
Change from base Build 8849262940: 0.0%
Covered Lines: 6313
Relevant Lines: 7007

💛 - Coveralls

e2e/pipelines/test_evaluation_pipeline.py Outdated Show resolved Hide resolved
e2e/pipelines/test_evaluation_pipeline.py Outdated Show resolved Hide resolved
haystack/evaluation/eval_run_result.py Outdated Show resolved Hide resolved
Copy link
Contributor

@shadeMe shadeMe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Can we also invoke the E2E tests in the CI to make sure that they work?

@davidsbatista davidsbatista enabled auto-merge (squash) April 26, 2024 14:03
@davidsbatista davidsbatista merged commit 8d04e53 into main Apr 26, 2024
17 checks passed
@davidsbatista davidsbatista deleted the end2end-eval-tests branch April 26, 2024 14:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2.x Related to Haystack v2.0 ignore-for-release-notes PRs with this flag won't be included in the release notes. proposal topic:build/distribution topic:tests type:documentation Improvements on the docs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add e2e tests for pipeline evaluation for 2.x
3 participants