Got 0 tweets today. Scraping correctly until yesterday!

[matteoravani10]

No description provided.

[narain1]

yeah same problem

[dantechguy]

My network blocks proxys, so even with --disableproxy initially I got

requests.exceptions.ConnectionError: HTTPSConnectionPool(host='free-proxy-list.net', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x000002191C342A60>: Failed to establish a new connection: [WinError 10060]

After setting up a VPN to bypass the proxy block, am now getting

requests.exceptions.ConnectionError: ('Connection aborted.', OSError(0, 'Error'))

[lukaspistelak]

i think they shut down non-javascipt render at twitter search web page

[dantechguy]

If you need an urgent alternative twint seems to still be functional

[Legnus]

@dantechguy I was using Twint, and it stopped functioning as well today.
twintproject/twint#863

[abhi0420]

Same problem
Getting 0 tweets every time I run the query
It was working fine till yesterday
If anyone can suggest some alternatives It'd be great!

[sm-ghavami]

I just changed the code to solve the problem.
Check the changes at #337

[javad94]

i think they shut down non-javascipt render at twitter search web page

Maybe because they launched their new API:
https://blog.twitter.com/developer/en_us/topics/tools/2020/introducing_new_twitter_api.html

[snmishra]

A legacy feed is still available by submitting the form in the noscript tag. The scraping will be different though.

[reverse]

A legacy feed is still available by submitting the form in the noscript tag. The scraping will be different though.

how is this done?

[snmishra]

A legacy feed is still available by submitting the form in the noscript tag. The scraping will be different though.

how is this done?

Never mind my comment, I missed that it's already fixed in #337.

[someguy-2020]

Fix #337 did not completely fix the user search (keyword search seems good). Seems 1st return is JSON, but additional returns may not be. I added a "try: except:" after "try:" in line 107. It first trys response.text...than excepts: json_resp...

[blafaye1]

Fix #337 did not completely fix the user search (keyword search seems good). Seems 1st return is JSON, but additional returns may not be. I added a "try: except:" after "try:" in line 107. It first trys response.text...than excepts: json_resp...

I'm not following your fix for the user search. Can you share your code snippet for those lines?

[someguy-2020]

It did not fix the problem. I think the problem might be that the "INIT_URL_USER"/"RELOAD_URL_USER" have changed in query.py. The INIT_URL_USER seems to work OK, but says "has more items" is FALSE.

[devded]

the same problem got 0 tweets, maybe twitter launch their new API

[DiabetesLongTorsoMan]

Yeah it appears that #337 did not fully fix the issue. Before I made the changes in #337 I would get 0 tweets every time, but now, it will run for a while and then eventually start returning 0 tweets again.

[DiabetesLongTorsoMan]

It did not fix the problem. I think the problem might be that the "INIT_URL_USER"/"RELOAD_URL_USER" have changed in query.py. The INIT_URL_USER seems to work OK, but says "has more items" is FALSE.

Did the changes in #337 fix the problem temporarily for you too? With it I started getting tweets returned, but after 5-10 minutes it starting returning 0 tweets again.

[someguy-2020]

The #337 fix works for KW searches. #337 does not fix User searches. The INIT_URL_USER returns up to 20 (?) of a user tweets. The return is JSON, and processed OK. The initial returns seems to always say "has more items" = False, which is not true in most cases. My guess is that the URLs for User search need to be updated like they were for KW search in #337.

[DiabetesLongTorsoMan]

The #337 fix works for KW searches. #337 does not fix User searches. The INIT_URL_USER returns up to 20 (?) of a user tweets. The return is JSON, and processed OK. The initial returns seems to always say "has more items" = False, which is not true in most cases. My guess is that the URLs for User search need to be updated like they were for KW search in #337.

I don't think the #337 fix works fully for KW searches either. I am only doing KW searches and they work for a bit but after a while (it seems to be a different amount of time each time) 0 tweets is returned

[christiangfv]

How I can implement solution # 337 without modifying the module ?

[someguy-2020]

Without difficulty I don't think you can

[sspartzz257]

@christiangfv, You can edit your py file /opt/anaconda3/lib/python3.8/site-packages/twitterscrapper and go into the query.py script and change the code and this will solve your issue. I troubleshooter for a few hours last night and this was the best way to do it.

[wajonespress]

#337 worked a few times, and then I get this error message and only a fraction of my limit.

Failed to parse JSON while requesting "https://twitter.com/i/search/timeline?f=tweets&vertical=default&include_available_features=1&include_entities=1&reset_error_state=false&src=typd&max_position=thGAVUV0VFVBaAwKqhpsTgySEWgoCogZGk3sohEjUAFQAlAFUAFQAVARUAFQAA&q=arson%20OR%20firebug%20since%3A2019-12-12%20until%3A2019-12-26&l=english" Traceback (most recent call last): File "/opt/anaconda3/lib/python3.8/site-packages/twitterscraper/query.py", line 109, in query_single_page html = json_resp['items_html'] or '' KeyError: 'items_html'

I also get errors for line 104 and 182 which doesn't resolve until a reinstall twitterscraper and the commit request.

Can anyone help? I'm a first time coder. Very, very new to this stuff.

[GivenToFlyCoder]

I just changed the code to solve the problem.
Check the changes at #337

@snmishra
Thanks! I'm using Query Search with User Search, this work for a while, sometimes i get a lot of tweets and sometimes i get 0 tweets!
Sometimes i repeat the searches with the same keywords, Accounts and dates, and a i get different tweets.

[narain1]

anyone has any idea how to overcome: key error items_html

[mis-tech]

Same problem
Getting 0 tweets every time I run the query
It was working fine till yesterday
If anyone can suggest some alternatives It'd be great!

hey can uh get just user info? because it's giving syntax error

[ghost]

As of August 21, 0 tweets is still an issue. Hopefully, this can be addressed soon.

[EklipZgit]

Seriously, nobody fixed this in like 3 months? Just had to stop sending a header in the initial request, I'll try to get a pullrequest out...

[Altimis]

@EklipZgit
All twitter scrapers dont seem to work anymore. I tried to use selenium to simply scrap maximum tweets between two chosen dates for given quiries. Check my work on : Scweet. let me know if you need any clarification.

[amitchandnia]

the same problem got 0 tweets, maybe twitter launch their new API

have you solved this issue?