feat: Add SemanticScholarToolkits to integrate Semantic Scholar to camel #1493

renxinxing123 · 2025-01-23T12:14:13Z

Description

This PR introduces a new toolkit called SemanticScholarToolkits to integrate Semantic Scholar into CAMEL. It provides several functionalities, including searching for papers by paper ID, paper title, and keywords, and retrieving recommended papers based on a given paper ID as well as searching author by author ID. Howerver, although Semantic Scholar API is able to search dataset, this feature itself is currently non-responsive based on my testing.

Motivation and Context

Integrating Semantic Scholar into CAMEL enhances its ability to access and process academic papers and resources. This integration will make CAMEL more versatile for research-related tasks by leveraging the rich academic resources and powerful search capabilities of Semantic Scholar. This change addresses the feature request in issue #1032.
[ ] I have raised an issue to propose this change (#1032)

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds core functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Documentation (update in the documentation)
[ ] Example (update in the folder of example)

Implemented Tasks

[x] Implement search paper by paper ID
[x] Implement search paper by paper title
[x] Implement search papers by keywords
[x] Implement retrieve recommended papers by paper ID
[x] Implement search author by author ID

Checklist

[x] I have read the CONTRIBUTION guide. (required)
[ ] My change requires a change to the documentation.
[x] I have updated the tests accordingly. (required for a bug fix or a new feature)
[ ] I have updated the documentation accordingly.

harryeqs · 2025-01-23T23:59:04Z

camel/toolkits/semanticscholar_toolkit.py

+import json
+
+class SemanticScholarToolkit(BaseToolkit):
+    """A toolkit for interacting with the Semantic Scholar API to fetch paper and author data."""


Suggested change

"""A toolkit for interacting with the Semantic Scholar API to fetch paper and author data."""

r"""A toolkit for interacting with the Semantic Scholar API to fetch paper and author data."""

harryeqs · 2025-01-23T23:59:24Z

camel/toolkits/semanticscholar_toolkit.py

+    """A toolkit for interacting with the Semantic Scholar API to fetch paper and author data."""
+
+    def __init__(self):
+        """Initializes the SemanticScholarToolkit."""


Suggested change

"""Initializes the SemanticScholarToolkit."""

r"""Initializes the SemanticScholarToolkit."""

harryeqs · 2025-01-24T00:02:55Z

camel/toolkits/semanticscholar_toolkit.py

+            papers = response.json().get("recommendedPapers", [])
+            papers.sort(key=lambda paper: paper["citationCount"], reverse=True)
+            with open('recommended_papers_sorted.json', 'w') as output:
+                json.dump(papers, output)


Shall we make writing to a local JSON file optional?

harryeqs

Thanks @renxinxing123 ! Please run and fix the pre-commit test, and make sure that the code and docstring are formatted correctly. Could you also add the unit test and examples under the test and examples folder please? Thanks a lot!

renxinxing123 · 2025-01-24T12:37:49Z

Thanks @renxinxing123 ! Please run and fix the pre-commit test, and make sure that the code and docstring are formatted correctly. Could you also add the unit test and examples under the test and examples folder please? Thanks a lot!

Many thanks for you comment @harryeqs! I will fix the error.

harryeqs · 2025-01-24T14:31:40Z

Thanks @renxinxing123 ! Please run and fix the pre-commit test, and make sure that the code and docstring are formatted correctly. Could you also add the unit test and examples under the test and examples folder please? Thanks a lot!

Many thanks for you comment @harryeqs! I will fix the error.

Thanks! It seems there are some remaining formatting problems. You could run the pre-commit test locally using the commands.

# Install camel from source
poetry install --with dev,docs -E all  # (Suggested for developers, needed to pass all tests)

# The following command installs a pre-commit hook into the local git repo,
# so every commit gets auto-formatted and linted.
pre-commit install

# Run camel's pre-commit before push
pre-commit run --all-files

For other contributing guidelines please refer to: https://github.com/camel-ai/camel/blob/master/CONTRIBUTING.md

renxinxing123 · 2025-01-25T17:02:45Z

Thanks @renxinxing123 ! Please run and fix the pre-commit test, and make sure that the code and docstring are formatted correctly. Could you also add the unit test and examples under the test and examples folder please? Thanks a lot!

Many thanks for you comment @harryeqs! I will fix the error.

Thanks! It seems there are some remaining formatting problems. You could run the pre-commit test locally using the commands.
# Install camel from source
poetry install --with dev,docs -E all  # (Suggested for developers, needed to pass all tests)

# The following command installs a pre-commit hook into the local git repo,
# so every commit gets auto-formatted and linted.
pre-commit install

# Run camel's pre-commit before push
pre-commit run --all-files
For other contributing guidelines please refer to: https://github.com/camel-ai/camel/blob/master/CONTRIBUTING.md

Thank you, @harryeqs! I followed your suggestion, reformatted the Semantic Scholar toolkits, and added the related test and example files. All files passed the pre-commit tests on my local machine, but it seems that several tests didn’t pass during the PR. However, the error messages in these tests are unrelated to the newly added files.

harryeqs · 2025-01-31T02:30:19Z

Thanks @renxinxing123 ! Please run and fix the pre-commit test, and make sure that the code and docstring are formatted correctly. Could you also add the unit test and examples under the test and examples folder please? Thanks a lot!

Many thanks for you comment @harryeqs! I will fix the error.

Thanks! It seems there are some remaining formatting problems. You could run the pre-commit test locally using the commands.
# Install camel from source
poetry install --with dev,docs -E all  # (Suggested for developers, needed to pass all tests)

# The following command installs a pre-commit hook into the local git repo,
# so every commit gets auto-formatted and linted.
pre-commit install

# Run camel's pre-commit before push
pre-commit run --all-files
For other contributing guidelines please refer to: https://github.com/camel-ai/camel/blob/master/CONTRIBUTING.md
Thank you, @harryeqs! I followed your suggestion, reformatted the Semantic Scholar toolkits, and added the related test and example files. All files passed the pre-commit tests on my local machine, but it seems that several tests didn’t pass during the PR. However, the error messages in these tests are unrelated to the newly added files.

Happy Chinese New Year! Thank you very much for the contribution @renxinxing123 . Sorry for getting back quite late as I was working on different tasks in the past few days.
The only thing I can think of for improvement is to make the json file writing optional (or remove it) since the data is already present in the returned dictionary. All the other part looks good to me. Thanks!

AveryYay · 2025-01-31T06:15:51Z

camel/toolkits/semanticscholar_toolkit.py

+        """
+        url = f"{self.base_url}/paper/search"
+        query_params = {"query": paperTitle, "fields": fields}
+        response = requests.get(url, params=query_params)


It might be better if we implement error handling here

Hi @AveryYay ! Thanks for your suggestion! I noticed that the code already includes error handling for the case where the response status code is not 200, and it returns an error message accordingly.

Could you clarify if you're suggesting a different type of error handling?

Checking the status code works if requests.get() successfully returns. Adding try-except could prevents crashes if the request fails due to connectivity problems. There could also be some case where the response isn't a valid JSON.

Thank you for your explanation @AveryYay ! I've updated the SemanticScholarToolkit to improve error handling, adding support for request failures and invalid JSON responses. And the corresponding test file has been updated to correctly mock and validate error responses.

renxinxing123 · 2025-02-03T14:09:06Z

Thanks @renxinxing123 ! Please run and fix the pre-commit test, and make sure that the code and docstring are formatted correctly. Could you also add the unit test and examples under the test and examples folder please? Thanks a lot!

Many thanks for you comment @harryeqs! I will fix the error.

Thanks! It seems there are some remaining formatting problems. You could run the pre-commit test locally using the commands.
# Install camel from source
poetry install --with dev,docs -E all  # (Suggested for developers, needed to pass all tests)

# The following command installs a pre-commit hook into the local git repo,
# so every commit gets auto-formatted and linted.
pre-commit install

# Run camel's pre-commit before push
pre-commit run --all-files
For other contributing guidelines please refer to: https://github.com/camel-ai/camel/blob/master/CONTRIBUTING.md
Thank you, @harryeqs! I followed your suggestion, reformatted the Semantic Scholar toolkits, and added the related test and example files. All files passed the pre-commit tests on my local machine, but it seems that several tests didn’t pass during the PR. However, the error messages in these tests are unrelated to the newly added files.
Happy Chinese New Year! Thank you very much for the contribution @renxinxing123 . Sorry for getting back quite late as I was working on different tasks in the past few days. The only thing I can think of for improvement is to make the json file writing optional (or remove it) since the data is already present in the returned dictionary. All the other part looks good to me. Thanks!

Happy Chinese New Year @harryeqs ! It doesn't matter, hope you enjoyed a good time with your family and the ones you love! The generation of the json file has been revised as an option, where is set as False as default, while user can activate it using the promt such as 'Please search the information of 'author xx', and save it in a json file'.

…id JSON responses.

…lidate error responses

Wendong-Fan

Thanks, @renxinxing123! Overall, it looks great. The format could be improved further. I'll merge this PR and then create another enhancement PR for it. Next time, please create the branch directly in the camel repo for easier access by our core members. Thanks again for your contribution!

enhance PR based on review comment: https://github.com/camel-ai/camel/pull/1562/files feel free to check!

Wendong-Fan · 2025-02-06T19:24:58Z

camel/toolkits/semanticscholar_toolkit.py

+    """A toolkit for interacting with the Semantic Scholar
+    API to fetch paper and author data."""


docstring format

Suggested change

"""A toolkit for interacting with the Semantic Scholar

API to fetch paper and author data."""

r"""A toolkit for interacting with the Semantic Scholar

API to fetch paper and author data.

"""

Wendong-Fan · 2025-02-06T19:36:19Z

camel/toolkits/semanticscholar_toolkit.py

+        fields: str = """title,abstract,authors,year,citationCount,
+        publicationTypes,publicationDate,openAccessPdf""",


we could have better way for the fields input

Wendong-Fan · 2025-02-06T19:36:52Z

camel/toolkits/semanticscholar_toolkit.py

+            dict: The response data from the API or error information
+            if the request fails.


docstring format

Suggested change

dict: The response data from the API or error information

if the request fails.

dict: The response data from the API or error information

if the request fails.

lightaime

Thanks @renxinxing123! The PR looks great. Just leave a couple comments on the remove BaseMessage since we are depreciate it.

Also do we need an API key for this?

examples/toolkits/semanticscholar_toolkit.py

Wendong-Fan · 2025-02-06T21:23:40Z

Thanks @renxinxing123! The PR looks great. Just leave a couple comments on the remove BaseMessage since we are depreciate it.

Also do we need an API key for this?

hey @lightaime , it doesn't require api key

Add SemanticScholarToolkits to integrate Semantic Scholar to camel

d5ee024

Wendong-Fan assigned renxinxing123 Jan 23, 2025

Wendong-Fan added the New Feature label Jan 23, 2025

Wendong-Fan changed the title ~~Add SemanticScholarToolkits to integrate Semantic Scholar to camel~~ feat: Add SemanticScholarToolkits to integrate Semantic Scholar to camel Jan 23, 2025

Wendong-Fan added this to the Sprint 21 milestone Jan 23, 2025

Wendong-Fan linked an issue Jan 23, 2025 that may be closed by this pull request

[Feature Request] Integrate Semantic scholar #1032

Closed

2 tasks

Wendong-Fan requested review from harryeqs and AveryYay January 23, 2025 13:49

Import the class SemanticScholarToolkit to _init_.py of toolkits

ed6571c

harryeqs reviewed Jan 23, 2025

View reviewed changes

harryeqs reviewed Jan 24, 2025

View reviewed changes

任信行 added 2 commits January 24, 2025 20:39

Try to properly format the signature and docstring

96fe5b4

Try to properly format the signature and docstring

f760e39

harryeqs and others added 13 commits January 24, 2025 23:58

Merge branch 'master' into SemanticScholarToolkit

55c67cf

Re-formatted the toolkit file, add example and test file

e5ad6f0

Re-formatted the toolkit file

b475269

Re-foramtted the toolkits file

9ac017c

Re-foramtted the toolkits file

27afcef

Re-foramtted the toolkits file

0d52bf1

Re-foramtted the toolkits file

cfd7450

Re-foramtted the toolkits file

121b2e7

Re-foramtted the toolkits file

da00e4a

Re-foramtted the toolkits file

d8e7094

An example of semanticscholar_toolkit

e7c65b0

Re-formatted semanticscholar_toolkit

d81207d

feat: Integrate Semantic Scholar into Camel

5570dc9

Merge branch 'master' into SemanticScholarToolkit

c901858

AveryYay reviewed Jan 31, 2025

View reviewed changes

Merge branch 'master' into SemanticScholarToolkit

ab9f157

renxinxing123 added 2 commits February 5, 2025 22:02

Improve error handling, adding support for request failures and inval…

f655e24

…id JSON responses.

The corresponding test file has been updated to correctly mock and va…

fb96134

…lidate error responses

Wendong-Fan approved these changes Feb 6, 2025

View reviewed changes

Merge branch 'master' into SemanticScholarToolkit

55e4413

Wendong-Fan merged commit b35145a into camel-ai:master Feb 6, 2025
1 of 6 checks passed

lightaime reviewed Feb 6, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add SemanticScholarToolkits to integrate Semantic Scholar to camel #1493

feat: Add SemanticScholarToolkits to integrate Semantic Scholar to camel #1493

renxinxing123 commented Jan 23, 2025

harryeqs Jan 23, 2025

harryeqs Jan 23, 2025

harryeqs Jan 24, 2025

harryeqs left a comment •

edited

Loading

renxinxing123 commented Jan 24, 2025

harryeqs commented Jan 24, 2025

renxinxing123 commented Jan 25, 2025

harryeqs commented Jan 31, 2025

AveryYay Jan 31, 2025

renxinxing123 Feb 3, 2025

AveryYay Feb 3, 2025 •

edited

Loading

renxinxing123 Feb 5, 2025

renxinxing123 commented Feb 3, 2025

Wendong-Fan left a comment •

edited

Loading

Wendong-Fan Feb 6, 2025

Wendong-Fan Feb 6, 2025

Wendong-Fan Feb 6, 2025

lightaime left a comment

Wendong-Fan commented Feb 6, 2025

	"""A toolkit for interacting with the Semantic Scholar API to fetch paper and author data."""
	r"""A toolkit for interacting with the Semantic Scholar API to fetch paper and author data."""

	"""Initializes the SemanticScholarToolkit."""
	r"""Initializes the SemanticScholarToolkit."""

		"""A toolkit for interacting with the Semantic Scholar
		API to fetch paper and author data."""

		fields: str = """title,abstract,authors,year,citationCount,
		publicationTypes,publicationDate,openAccessPdf""",

		dict: The response data from the API or error information
		if the request fails.

feat: Add SemanticScholarToolkits to integrate Semantic Scholar to camel #1493

feat: Add SemanticScholarToolkits to integrate Semantic Scholar to camel #1493

Conversation

renxinxing123 commented Jan 23, 2025

Description

Motivation and Context

Types of changes

Implemented Tasks

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

harryeqs left a comment • edited Loading

Choose a reason for hiding this comment

renxinxing123 commented Jan 24, 2025

harryeqs commented Jan 24, 2025

renxinxing123 commented Jan 25, 2025

harryeqs commented Jan 31, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AveryYay Feb 3, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

renxinxing123 commented Feb 3, 2025

Wendong-Fan left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lightaime left a comment

Choose a reason for hiding this comment

Wendong-Fan commented Feb 6, 2025

harryeqs left a comment •

edited

Loading

AveryYay Feb 3, 2025 •

edited

Loading

Wendong-Fan left a comment •

edited

Loading