feat(document-search): LiteLLM Reranker #109

PatrykWyzgowski · 2024-10-16T11:40:00Z

No description provided.

…hod with kwargs.

github-actions · 2024-10-16T15:03:03Z

Code Coverage Summary

Filename                                                                                                     Stmts    Miss  Cover    Missing
---------------------------------------------------------------------------------------------------------  -------  ------  -------  ------------------------------------------
packages/ragbits-core/src/ragbits/core/__init__.py                                                               0       0  100.00%
packages/ragbits-core/src/ragbits/core/config.py                                                                 7       0  100.00%
packages/ragbits-core/src/ragbits/core/embeddings/__init__.py                                                   11       0  100.00%
packages/ragbits-core/src/ragbits/core/embeddings/base.py                                                        4       0  100.00%
packages/ragbits-core/src/ragbits/core/embeddings/noop.py                                                        4       0  100.00%
packages/ragbits-core/src/ragbits/core/llms/__init__.py                                                         14       8  42.86%   27-39
packages/ragbits-core/src/ragbits/core/llms/base.py                                                             41       5  87.80%   45, 64, 134-136
packages/ragbits-core/src/ragbits/core/llms/factory.py                                                          23       4  82.61%   50, 53, 66, 77
packages/ragbits-core/src/ragbits/core/llms/litellm.py                                                          40      13  67.50%   9-10, 55, 86, 92-109
packages/ragbits-core/src/ragbits/core/llms/types.py                                                             9       2  77.78%   25, 29
packages/ragbits-core/src/ragbits/core/llms/clients/__init__.py                                                  4       0  100.00%
packages/ragbits-core/src/ragbits/core/llms/clients/base.py                                                     23       0  100.00%
packages/ragbits-core/src/ragbits/core/llms/clients/exceptions.py                                               14       6  57.14%   7-8, 17, 26-27, 36
packages/ragbits-core/src/ragbits/core/llms/clients/litellm.py                                                  49      10  79.59%   9-10, 69, 107, 121-126
packages/ragbits-core/src/ragbits/core/llms/clients/local.py                                                    36      12  66.67%   10-11, 61-69, 90-101
packages/ragbits-core/src/ragbits/core/metadata_stores/__init__.py                                              12       3  75.00%   27-30
packages/ragbits-core/src/ragbits/core/metadata_stores/base.py                                                   6       0  100.00%
packages/ragbits-core/src/ragbits/core/metadata_stores/exceptions.py                                             4       0  100.00%
packages/ragbits-core/src/ragbits/core/metadata_stores/in_memory.py                                             13       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/__init__.py                                                        2       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/base.py                                                           20       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/parsers.py                                                        35       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/prompt.py                                                        124       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/discovery/__init__.py                                              2       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/discovery/prompt_discovery.py                                     33       2  93.94%   54-55
packages/ragbits-core/src/ragbits/core/utils/_pyproject.py                                                      37       0  100.00%
packages/ragbits-core/src/ragbits/core/utils/config_handling.py                                                 16       4  75.00%   32-33, 37-38
packages/ragbits-core/src/ragbits/core/utils/decorators.py                                                      29       0  100.00%
packages/ragbits-core/src/ragbits/core/vector_stores/__init__.py                                                14       1  92.86%   28
packages/ragbits-core/src/ragbits/core/vector_stores/base.py                                                    22       0  100.00%
packages/ragbits-core/src/ragbits/core/vector_stores/chroma.py                                                  50       2  96.00%   69-70
packages/ragbits-core/src/ragbits/core/vector_stores/in_memory.py                                               24       0  100.00%
packages/ragbits-core/tests/unit/__init__.py                                                                     0       0  100.00%
packages/ragbits-core/tests/unit/llms/__init__.py                                                                0       0  100.00%
packages/ragbits-core/tests/unit/llms/test_litellm.py                                                           64       0  100.00%
packages/ragbits-core/tests/unit/llms/factory/__init__.py                                                        3       0  100.00%
packages/ragbits-core/tests/unit/llms/factory/test_get_default_llm.py                                           10       0  100.00%
packages/ragbits-core/tests/unit/llms/factory/test_get_llm_from_factory.py                                       8       0  100.00%
packages/ragbits-core/tests/unit/llms/factory/test_has_default_llm.py                                           10       0  100.00%
packages/ragbits-core/tests/unit/metadata_stores/__init__.py                                                     0       0  100.00%
packages/ragbits-core/tests/unit/metadata_stores/test_in_memory.py                                              22       0  100.00%
packages/ragbits-core/tests/unit/prompts/__init__.py                                                             0       0  100.00%
packages/ragbits-core/tests/unit/prompts/test_parsers.py                                                        65       0  100.00%
packages/ragbits-core/tests/unit/prompts/test_prompt.py                                                        155       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/__init__.py                                                   0       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/prompt_classes_for_tests.py                                  30       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/test_prompt_discovery.py                                     18       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/ragbits_tests_pkg_with_prompts/__init__.py                    2       1  50.00%   3
packages/ragbits-core/tests/unit/prompts/discovery/ragbits_tests_pkg_with_prompts/prompts/__init__.py            3       2  33.33%   2-4
packages/ragbits-core/tests/unit/prompts/discovery/ragbits_tests_pkg_with_prompts/prompts/temp_prompt1.py       14       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/ragbits_tests_pkg_with_prompts/prompts/temp_prompt2.py       14       0  100.00%
packages/ragbits-core/tests/unit/utils/test_decorators.py                                                       26       2  92.31%   17, 39
packages/ragbits-core/tests/unit/utils/pyproject/test_find.py                                                   13       0  100.00%
packages/ragbits-core/tests/unit/utils/pyproject/test_get_config.py                                              9       0  100.00%
packages/ragbits-core/tests/unit/utils/pyproject/test_get_instace.py                                            37       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/__init__.py                                                       0       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/test_chroma.py                                                   37       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/test_in_memory.py                                                76       1  98.68%   39
packages/ragbits-document-search/src/ragbits/document_search/__init__.py                                         2       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/_main.py                                           70       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/documents/__init__.py                               0       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/documents/document.py                              61       3  95.08%   56, 99, 146
packages/ragbits-document-search/src/ragbits/document_search/documents/element.py                               43       2  95.35%   53, 126
packages/ragbits-document-search/src/ragbits/document_search/documents/exceptions.py                            11       5  54.55%   7-8, 17, 26-27
packages/ragbits-document-search/src/ragbits/document_search/documents/sources.py                              116      16  86.21%   15-16, 116, 129, 212-217, 254-257, 261-262
packages/ragbits-document-search/src/ragbits/document_search/ingestion/__init__.py                               0       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/document_processor.py                    28       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/__init__.py                    10       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/base.py                        14       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/dummy.py                       11       1  90.91%   33
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/unstructured/__init__.py        4       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/unstructured/default.py        44       4  90.91%   97, 102-103, 135
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/unstructured/images.py         59      24  59.32%   79-86, 93-113, 125, 138
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/unstructured/pdf.py            19       6  68.42%   23, 35-43
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/unstructured/utils.py          38      11  71.05%   71, 82-83, 98-101, 110, 121-123
packages/ragbits-document-search/src/ragbits/document_search/retrieval/__init__.py                               0       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/__init__.py                   15       4  73.33%   39-44
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/base.py                        7       1  85.71%   32
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/llm.py                        20       9  55.00%   27-28, 45-48, 65-67
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/noop.py                        4       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/prompts.py                    16       4  75.00%   49-54
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/__init__.py                    10       1  90.00%   27
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/base.py                        15       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/litellm.py                     16       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/noop.py                         9       0  100.00%
packages/ragbits-document-search/tests/__init__.py                                                               0       0  100.00%
packages/ragbits-document-search/tests/helpers.py                                                                3       0  100.00%
packages/ragbits-document-search/tests/integration/__init__.py                                                   0       0  100.00%
packages/ragbits-document-search/tests/integration/test_rerankers.py                                            15       6  60.00%   18-38
packages/ragbits-document-search/tests/integration/test_sources.py                                              23      10  56.52%   22-32, 40-45
packages/ragbits-document-search/tests/integration/test_unstructured.py                                         47      10  78.72%   46-52, 65-71
packages/ragbits-document-search/tests/unit/test_document_processor.py                                          17       0  100.00%
packages/ragbits-document-search/tests/unit/test_document_search.py                                             75       0  100.00%
packages/ragbits-document-search/tests/unit/test_documents.py                                                   13       0  100.00%
packages/ragbits-document-search/tests/unit/test_elements.py                                                    17       1  94.12%   15
packages/ragbits-document-search/tests/unit/test_local_file_source.py                                           13       0  100.00%
packages/ragbits-document-search/tests/unit/test_providers.py                                                   31       0  100.00%
packages/ragbits-document-search/tests/unit/test_rerankers.py                                                   31       1  96.77%   21
packages/ragbits-document-search/tests/unit/test_source_discriminator.py                                        35       0  100.00%
packages/ragbits-document-search/tests/unit/test_sources.py                                                     25       0  100.00%
TOTAL                                                                                                         2325     197  91.53%

Diff against main

Filename                                                                                        Stmts    Miss  Cover
--------------------------------------------------------------------------------------------  -------  ------  --------
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/__init__.py       -2       0  -1.67%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/base.py           +9       0  +100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/litellm.py       +16       0  +100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/noop.py           +3       0  +100.00%
packages/ragbits-document-search/tests/integration/test_rerankers.py                              +15      +6  +60.00%
packages/ragbits-document-search/tests/unit/test_rerankers.py                                     +31      +1  +96.77%
TOTAL                                                                                             +72      +7  -0.04%

Results for commit: 4139ab2

Minimum allowed coverage is 60%

♻️ This comment has been updated with latest results

github-actions · 2024-10-17T08:25:09Z

Trivy scanning results.

.venv/lib/python3.10/site-packages/PyJWT-2.9.0.dist-info/METADATA (secrets)

Total: 1 (MEDIUM: 1, HIGH: 0, CRITICAL: 0)

MEDIUM: JWT (jwt-token)
════════════════════════════════════════
JWT token
────────────────────────────────────────
.venv/lib/python3.10/site-packages/PyJWT-2.9.0.dist-info/METADATA:80
────────────────────────────────────────
78 >>> encoded = jwt.encode({"some": "payload"}, "secret", algorithm="HS256")
79 >>> print(encoded)
80 [ *********************************************************************************************************
81 >>> jwt.decode(encoded, "secret", algorithms=["HS256"])
────────────────────────────────────────

.venv/lib/python3.10/site-packages/litellm/llms/huggingface_llms_metadata/hf_text_generation_models.txt (secrets)

Total: 1 (MEDIUM: 0, HIGH: 0, CRITICAL: 1)

CRITICAL: HuggingFace (hugging-face-access-token)
════════════════════════════════════════
Hugging Face Access Token
────────────────────────────────────────
.venv/lib/python3.10/site-packages/litellm/llms/huggingface_llms_metadata/hf_text_generation_models.txt:36162
────────────────────────────────────────
36160 mncai/Llama2-7B-Active_3rd-floor-LoRA-dim64_epoch4
36161 ajcdp/CM
36162 [ Nagharjun17/*************************************
36163 BigSalmon/InformalToFormalLincoln114Paraphrase
────────────────────────────────────────

.venv/lib/python3.10/site-packages/litellm/proxy/_types.py (secrets)

Total: 1 (MEDIUM: 1, HIGH: 0, CRITICAL: 0)

MEDIUM: Slack (slack-web-hook)
════════════════════════════════════════
Slack Webhook
────────────────────────────────────────
.venv/lib/python3.10/site-packages/litellm/proxy/_types.py:1288
────────────────────────────────────────
1286 alert_to_webhook_url: Optional[Dict] = Field(
1287 None,
1288 [ bhook_url: {'budget_alerts': '*****************************************************************************'}`",
1289 )
────────────────────────────────────────

packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/litellm.py

packages/ragbits-document-search/examples/reranker_example.py

packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/litellm.py

Initial litellm ranker implementation. Extended Base Class static met…

ce27275

…hod with kwargs.

PatrykWyzgowski requested a review from mhordynski October 16, 2024 11:40

PatrykWyzgowski linked an issue Oct 16, 2024 that may be closed by this pull request

feat(rerankers): implement litellm rerankers #96

Closed

PatrykWyzgowski added 3 commits October 16, 2024 15:05

Able to call the API. But we don't have ranks to rerank.

2a50318

Sorting the original chunks with orded given by reranking.

fa9e528

Mypy silenced.

2dcf188

Additional pre-commit fixes.

8f71200

PatrykWyzgowski added 6 commits October 17, 2024 15:37

Config now belongs to an instance. No kwargs needed.

985fa0d

Linting, formatting and so on.

1dfe701

An example with LiteLLMReranker from config.

43ef956

Removing unnecessary comments.

2ecde29

Providing unit tests for LiteLLMReranker class.

9046252

Black+isort.

9be5157

PatrykWyzgowski marked this pull request as ready for review October 18, 2024 13:14

Default language version - 3.10

5efc234

akotyla reviewed Oct 21, 2024

View reviewed changes

packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/litellm.py Outdated Show resolved Hide resolved

packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/litellm.py Outdated Show resolved Hide resolved

akotyla reviewed Oct 21, 2024

View reviewed changes

packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/litellm.py Show resolved Hide resolved

PatrykWyzgowski added 3 commits October 22, 2024 14:50

Going asynchronous with reranking.

11a203f

Fix lack of await.

90a65f0

Adding docs for LiteLLMReranker class.

4b542f9

micpst reviewed Oct 23, 2024

View reviewed changes

packages/ragbits-document-search/examples/reranker_example.py Outdated Show resolved Hide resolved

micpst reviewed Oct 23, 2024

View reviewed changes

packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/litellm.py Outdated Show resolved Hide resolved

micpst reviewed Oct 23, 2024

View reviewed changes

packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/litellm.py Outdated Show resolved Hide resolved

micpst reviewed Oct 23, 2024

View reviewed changes

packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/litellm.py Outdated Show resolved Hide resolved

mhordynski mentioned this pull request Oct 24, 2024

feat(documents): unified API for text representations of Element models #143

Closed

PatrykWyzgowski and others added 3 commits October 24, 2024 12:11

Fixed tests with mocking entire namespace.

0f330bb

merge main

011a17d

refactor rerankers

161ea29

fix linters

4139ab2

micpst approved these changes Nov 6, 2024

View reviewed changes

micpst merged commit 2ea2875 into main Nov 6, 2024
6 checks passed

micpst deleted the 96-featrerankers-implement-litellm-rerankers branch November 6, 2024 01:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(document-search): LiteLLM Reranker #109

feat(document-search): LiteLLM Reranker #109

PatrykWyzgowski commented Oct 16, 2024

github-actions bot commented Oct 16, 2024 •

edited

Loading

github-actions bot commented Oct 17, 2024 •

edited

Loading

feat(document-search): LiteLLM Reranker #109

feat(document-search): LiteLLM Reranker #109

Conversation

PatrykWyzgowski commented Oct 16, 2024

github-actions bot commented Oct 16, 2024 • edited Loading

Code Coverage Summary

Diff against main

github-actions bot commented Oct 17, 2024 • edited Loading

.venv/lib/python3.10/site-packages/PyJWT-2.9.0.dist-info/METADATA (secrets)

.venv/lib/python3.10/site-packages/litellm/llms/huggingface_llms_metadata/hf_text_generation_models.txt (secrets)

.venv/lib/python3.10/site-packages/litellm/proxy/_types.py (secrets)

github-actions bot commented Oct 16, 2024 •

edited

Loading

github-actions bot commented Oct 17, 2024 •

edited

Loading