[DS-9] wikidata online end point #409
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…to DS_9-Wikidata_OnlineEndPoint
kwang2049
reviewed
Feb 10, 2023
There was a problem hiding this comment.
Some notes:
- There are a lot of unrelated changes. Please remove them from this PR and create a new PR for them.
- Subgraph methods should be also supported for WikiData, which have not been added yet in this PR.
- The API callings should be placed inside each router method, i.e. move these
square-core/datastore-api/app/core/wikidata.py
Lines 185 to 205 in 1bbe9a5
into the correspondingrouters/kgs.pymethods. One can refer to the Bing search PR for this fea: add bing search datastore and tests #369 - This branch is out of date. Please pull the master branch
| @@ -427,14 +427,14 @@ async def get_node_by_name_msearch(self, kg_name, names): | |||
| index = f'{kg_name}{self.datastore_suffix}' | |||
| body = [] | |||
| for name in names: | |||
| body.append({'index': index}) | |||
| body.append({'index': index}) | |||
There was a problem hiding this comment.
The changes to this file connector.py are unrelated to this PR indent. Please remove them and create a new PR if needed
Comment on lines
-40
to
+44
| async def new_binding(self, request: Request, item_value: str, item_type: str): | ||
| user_id = await self._decode_user_id(request) | ||
| async def new_binding(self, request: Request, item_value: str, item_type: str, default_user_id=None): | ||
| if default_user_id is not None: | ||
| user_id = default_user_id | ||
| else: | ||
| user_id = await self._decode_user_id(request) |
There was a problem hiding this comment.
Why do we need this? It seems that default_user_id is never used
| raise HTTPException(status_code=500, detail=str(e)) | ||
|
|
||
| # This should not be reached | ||
| return await call_next(request) |
There was a problem hiding this comment.
No newline at the end of file
Suggested change
| return await call_next(request) | |
| return await call_next(request) | |
Comment on lines
48
to
49
| return BingSearch() No newline at end of file | ||
| return BingSearch() |
| "weight": None, | ||
| "in_id": None, | ||
| "out_id": None, | ||
| "id": self._WikiData__without_link(res['item']['value']) |
There was a problem hiding this comment.
Where is _WikiData__without_link defined?
Comment on lines
-17
to
20
| assert len(response.json()) == 1 | ||
| assert len(response.json()) >= 1 | ||
| assert sorted(response.json()[0]['fields'], key=lambda x: x['name']) == sorted(conceptnet_kg.dict()['fields'], key=lambda x: x['name']) |
There was a problem hiding this comment.
This is different from the master branch
square-core/datastore-api/tests/test_kgs.py
Lines 24 to 41 in d659549
Comment on lines
+46
to
+89
| def test_get_node_by_name(self, client: TestClient, conceptnet_kg: Datastore, test_node_lower: Document, token: str): | ||
| # Given: | ||
| url_post = f"/datastores/kg/{conceptnet_kg.name}/nodes/query_by_name" | ||
| expected_code_post = 200 | ||
|
|
||
| nodes_upper = ["Barack_Obama"] | ||
| nodes_lower = ["barack_obama"] | ||
|
|
||
| # When: | ||
| response_post_upper = client.post( | ||
| url_post, | ||
| json=nodes_upper, | ||
| headers={"Authorization": f"Bearer {token}"} | ||
| ) | ||
|
|
||
| response_post_lower = client.post( | ||
| url_post, | ||
| json=nodes_lower, | ||
| headers={"Authorization": f"Bearer {token}"} | ||
| ) | ||
|
|
||
| # Then: | ||
| assert response_post_lower.status_code == expected_code_post | ||
| assert response_post_upper.status_code == expected_code_post | ||
| assert response_post_lower.json()[test_node_lower.id] == test_node_lower | ||
| assert response_post_upper.json()[test_node_lower.id] == test_node_lower | ||
|
|
||
| def test_get_nodes_by_id(self, client: TestClient, conceptnet_kg: Datastore, test_node_lower: Document, token: str): | ||
| # Given: | ||
| url_post = f"/datastores/kg/{conceptnet_kg.name}/nodes/query_by_ids" | ||
| expected_code_post = 200 | ||
|
|
||
| nodes_upper = ["n1010", "n1234"] | ||
|
|
||
| # When: | ||
| response_post = client.post( | ||
| url_post, | ||
| json=nodes_upper, | ||
| headers={"Authorization": f"Bearer {token}"} | ||
| ) | ||
|
|
||
| # Then: | ||
| assert response_post.status_code == expected_code_post | ||
| assert response_post.json()[test_node_lower.id] == test_node_lower |
| def test_wikidata_get_node_by_id(self, client: TestClient, wikidata_kg_name: str): | ||
| # Given: | ||
| node_id = "Q42" | ||
| url = f"/datastores/kg/{wikidata_kg_name}/{node_id}" |
There was a problem hiding this comment.
Do we have such API in routers/kgs.py https://github.com/UKP-SQuARE/square-core/blob/master/datastore-api/app/routers/kgs.py?
| assert any("Turkey" == name["name"] for name in response.json().values()) | ||
|
|
||
| # Check wheter description is not null | ||
| assert all(len(name["description"]) > 2 for name in response.json().values()) |
There was a problem hiding this comment.
Suggested change
| assert all(len(name["description"]) > 2 for name in response.json().values()) | |
| assert all(len(name["description"]) for name in response.json().values()) |
datastore-api/tests/test_kgs.py
Outdated
Comment on lines
309
to
325
| assert response_get.json() == nodes No newline at end of file | ||
| assert response_get.json() == nodes | ||
|
|
||
| def test_get_subgraph(self, client: TestClient, kg_name: str, test_node_lower: Document, token: str): | ||
| # Given: | ||
| url_post = f"/datastores/kg/{kg_name}/subgraph/query_by_node_name" | ||
|
|
||
| # When: | ||
| response_post = client.post( | ||
| url_post, | ||
| json= {"nids": [test_node_lower["name"]], "hops": 2 }, | ||
| headers={"Authorization": f"Bearer {token}"} | ||
| ) | ||
|
|
||
| # Then: | ||
| assert response_post.status_code == 200 | ||
| assert len(response_post.json()) > 0 |
There was a problem hiding this comment.
Not related to this PR. Please creat another PR for this
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR is adding w WikiData-KG-Endpoint to the already existing KG-API. The main difference to a conventional KG, like "Conceptnet", is that we are using a middleware to connect to the Wikidata online Query service, which uses SPARQL code to retrieve the necessary information from Wikidata. Loading an offline version of WD into our Elasticsearch-cluster is not feasible.
This PR contains the main request functionalities, which are necessary for the working with KGs, except the Get-Subgraph request. This is part of a different PR.
Changes:
*
datastore-api/app/core/wikidata.py- new class for connecting to WikiData Query-service. The top class is handling the queries and has the necessary methods for searching edges/nodes. At the bottom you will find the routing for the middleware*
datastore-api/tests/conftest.py- Modifications on conftest.py were necessary to test the endpoint. Presaved answers and connections to the elasticsearch-cluster are handled here.*
datastore-api/tests/test_kgs.py- All new requests have accordingly tests. All requests are added to the KG-tests.Before submitting / marking as 'ready to review'