Implement filtering for WeaviateDocumentStore
#278
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
the
type:documentation
silvanocerza
force-pushed
the
weaviate-filters
branch
from
d45b860 to
a5c157d
Compare
silvanocerza
commented
Feb 13, 2024
| @@ -28,6 +26,7 @@ dependencies = [ | |||
| "haystack-ai", | |||
| "weaviate-client==3.*", | |||
| "haystack-pydoc-tools", | |||
| "python-dateutil", | |||
There was a problem hiding this comment.
I had to add this to have proper parsing of ISO 8601 dates as datetime.fromisoformat() supports it properly only from 3.11.
anakin87
reviewed
Feb 13, 2024
Comment on lines
126
to
137
| if value is None: | ||
| # When the value is None and '>' is used we create a filter that would return a Document | ||
| # if it has a field set and not set at the same time. | ||
| # This will cause the filter to match no Document. | ||
| # This way we keep the behavior consistent with other Document Stores. | ||
| return { | ||
| "operator": "And", | ||
| "operands": [ | ||
| {"path": field, "operator": "IsNull", "valueBoolean": False}, | ||
| {"path": field, "operator": "IsNull", "valueBoolean": True}, | ||
| ], | ||
| } |
There was a problem hiding this comment.
If I understood correctly, this is a trick to create a filter that returns no Documents.
It is done to keep the global filtering logic unaffected. Right?
If so, can we centralize that logic in a utility function?
| if "field" not in condition: | ||
| # 'field' key is only found in comparison dictionaries. | ||
| # We assume this is a logic dictionary since it's not present. | ||
| return _parse_logical_condition(condition) |
There was a problem hiding this comment.
Some time ago, in another PR, Massi made me notice that this might be better handled in the caller. 🙂
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR add support for
filtersinWeaviateDocumentStore.filter_documents()method.All tests are passing except for
test_comparison_not_equal_with_dataframe.For some reason that test is not returning what we expect from Weaviate.
I decided to skip it for the time being as I wanted to get this out of the door.