Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Duckduckgo,Brave,Startpage都失效 #82

Open
theLSA opened this issue Nov 9, 2024 · 1 comment
Open

Duckduckgo,Brave,Startpage都失效 #82

theLSA opened this issue Nov 9, 2024 · 1 comment

Comments

@theLSA
Copy link

theLSA commented Nov 9, 2024

这三个搜索引擎页面可能改变了.

pipenv install Search-Engines-Scraper-Tasos

duckduckgo报错如下:


from search_engines import Google,Bing,Duckduckgo,Brave,Startpage

>>> proxy="http://127.0.0.1:1080"
>>> engine=Duckduckgo(proxy)
>>> results=engine.search("test")
Searching Duckduckgo                                                           
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
  File "/home/user/.local/share/virtualenvs/scraper/lib/python3.11/site-packages/search_engines/engine.py", line 162, in search
    response = self._get_page(request['url'], request['data'])
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/share/virtualenvs/scraper/lib/python3.11/site-packages/search_engines/engines/duckduckgo.py", line 44, in _get_page
    response = self._http_client.get(page)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/share/virtualenvs/scraper/lib/python3.11/site-packages/search_engines/http_client.py", line 21, in get
    page = self._quote(page)
           ^^^^^^^^^^^^^^^^^
  File "/home/user/.local/share/virtualenvs/scraper/lib/python3.11/site-packages/search_engines/http_client.py", line 41, in _quote
    if utl.decode_bytes(utl.unquote_url(url)) == utl.decode_bytes(url):
                        ^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/share/virtualenvs/scraper/lib/python3.11/site-packages/search_engines/utils.py", line 15, in unquote_url
    return decode_bytes(requests.utils.unquote(url))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/parse.py", line 657, in unquote
    if '%' not in string:
       ^^^^^^^^^^^^^^^^^
TypeError: argument of type 'NoneType' is not iterable

brave报错::

results=engine.search("test")                                              
Searching Brave                                                                
ERROR Failed to decode response content.

Startpage无论搜索什么都返回空结果:

>>> engine3 = Startpage(proxy)
>>> results3 = engine3.search("test")
Searching Startpage                                                            
>>> links3 = results3.links()                             
>>> print(links3)
[]
@tasos-py
Copy link
Owner

Thanks for informing me about those issues. Actually about half of the engines don't work, and I'll have to update them when I have some time

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants