Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The site tik tok changes, so now the download does not work. #32

Open
MurloDin opened this issue Apr 21, 2020 · 36 comments
Open

The site tik tok changes, so now the download does not work. #32

MurloDin opened this issue Apr 21, 2020 · 36 comments
Labels
help wanted Extra attention is needed

Comments

@MurloDin
Copy link

No description provided.

@liner601
Copy link

liner601 commented Apr 22, 2020

No description provided.

This is incorrect, if you make too many requests to the site you can be blocked from making requests for a few hours or upto a day, If this fixes the issue please close the thread.

@werid
Copy link

werid commented Apr 22, 2020

i'm having the same problem, with my two different ip's (at home, and from a remote server), so since it's lasted for a day or two now, I tried from a new server, which have never connected to tiktok servers before, and it's stuck on preloading...

@MurloDin
Copy link
Author

MurloDin commented Apr 22, 2020

Описание не предоставлено.

Это неверно, если вы делаете слишком много запросов на сайт, вы можете быть заблокированы от отправки запросов в течение нескольких часов или до дня. Если это решит проблему, закройте ветку.

Can I get around the download limit?

@MurloDin
Copy link
Author

MurloDin commented Apr 22, 2020

Still, the problem is new links and site design

Resolving share link: https://vm.tiktok.com/72gvVk/
Error : encountered exception 'Uncaught' (0:2873)
Failed while scraping item:

@thenetguy
Copy link

Preloading...Error : context deadline exceeded
Failed while scraping item: tiktok_user

also the new windows zip doesn't have the extension on the exe file ?? whats that about

@bandicootdancing
Copy link

Having the same issue here today. They indeed changed their layout once again. I only download 10-20 videos a day so I'd be surprised if it was a limit issue.

@rabea98
Copy link

rabea98 commented Apr 22, 2020

@pikami TikTok changed the site layout, the script doesn't work at all. Please look into the matter

@pmjm
Copy link

pmjm commented Apr 23, 2020

Interestingly enough, it only works if you turn on debug mode.

@liner601
Copy link

Still, the problem is new links and site design

Resolving share link: https://vm.tiktok.com/72gvVk/
Error : encountered exception 'Uncaught' (0:2873)
Failed while scraping item:

Yep it seems the limit was unrelated in this case.

Preloading...Error : context deadline exceeded
Failed while scraping item: tiktok_user

also the new windows zip doesn't have the extension on the exe file ?? whats that about

Just change it to an exe and itll work.

Interestingly enough, it only works if you turn on debug mode.

How would you go about doing this?

@bandicootdancing
Copy link

How would you go about doing this?

By adding the -debug option
tiktok-dl -debug URL

Can confirm it works with the debug option on a desktop. Doesn't work on my headless machine.

@liner601
Copy link

How would you go about doing this?

By adding the -debug option
tiktok-dl -debug URL

Can confirm it works with the debug option on a desktop. Doesn't work on my headless machine.

Doesn't seem to work for me, the tool seems to vary between working and not working very often.

@rabea98
Copy link

rabea98 commented Apr 23, 2020

It doesn’t work for me also with the debug option

@AlphaSlayer1964
Copy link

How would you go about doing this?

By adding the -debug option
tiktok-dl -debug URL

Can confirm it works with the debug option on a desktop. Doesn't work on my headless machine.

When I add the debug mode a chrome tab opens and it goes to the profiles page but it won't load. The thing is on my regular user chrome the page loads just fine.

@bandicootdancing
Copy link

When I add the debug mode a chrome tab opens and it goes to the profiles page but it won't load. The thing is on my regular user chrome the page loads just fine.

You're right. I tested a single video without testing a full profile. Single videos still work but even with the debug option the profile page is blank in browser.

@AlphaSlayer1964
Copy link

When I add the debug mode a chrome tab opens and it goes to the profiles page but it won't load. The thing is on my regular user chrome the page loads just fine.

You're right. I tested a single video without testing a full profile. Single videos still work but even with the debug option the profile page is blank in browser.

Weird thing is if you inspect element the page it is "loaded" has the links to the first block of videos and everything don't know why when it creates a new user it breaks.

@pikami
Copy link
Owner

pikami commented Apr 23, 2020

Tiktok detects that the browser is being controlled by automation software, I don't have time to investigate crrently, but I will in the future, if anyone has free time to investigate and come up with a fix, fork this repo and create a PR with your fixes.

@pikami pikami added the help wanted Extra attention is needed label Apr 23, 2020
@AlphaSlayer1964
Copy link

Tiktok detects that the browser is being controlled by automation software, I don't have time to investigate crrently, but I will in the future, if anyone has free time to investigate and come up with a fix, fork this repo and create a PR with your fixes.

So i'm working on a python script that grabs the https://www.tiktok.com/@user/video/randomnumbers link to feed into yuotube-dl and two things I have found:
First using selenium with firefox has fixed the not loading part
Second some of the https://www.tiktok.com/@user/video/randomnumbers don't work with youtube-dl so I'm trying to figure out how to grab that direct link (from the user profile) that looks like v16.muscdn like you'rs seems to do. I have the regex to find it but getting it to pop up (have to hover over the video) is where I'm stuck at right now.
I should also state this is like my first time writing something like this and I have no idea what I'm doing.

@pikami
Copy link
Owner

pikami commented Apr 24, 2020

@AplhaSlayer1964 I didn't want to resort to using selenium because each selenium driver version requires a specific browser version and it would be a hassle to setup for users, but I might migrate my code to selenium if I can't figure out how their detection works.
If you want you can fork my project and try to migrate it to selenium, should be simple enough, the only file that needs to be changed is client/executeClientAction.go because it handles all the browser driving stuff.
Edit: I plan on investigating further thus weekend.

@DankMemeGuy
Copy link

@pikami

Have you tried bs4 or chromedriver.exe bundled with the script?

@pikami
Copy link
Owner

pikami commented Apr 27, 2020

@DankMemeGuy
bs4 wouldn't work because tiktok loads videos using JS
didn't use chromedriver because each driver version needs a specific version of chrome which would be a hastle to setup for the users

@DankMemeGuy
Copy link

@pikami

You sure about that? I used a script before that attached a chromedriver.exe with the .py script and it worked fine (to this day) without the person knowing what version of chrome I had, and I continue to update my real version of chrome but keep that chromedriver.exe that came with it in the folder of the .py

@tiaa4c
Copy link

tiaa4c commented May 2, 2020

is there anyway to save all of a users videos? or is everything broken as of now?

@pikami
Copy link
Owner

pikami commented May 2, 2020

@tiaa4c It's broken as of now

@mattcodez
Copy link

Screen Shot 2020-05-02 at 2 39 49 AM

Can it download directly from the API data? I grabbed this via curl after copying from chrome. The downloadAddr links directly to the file.

@pikami
Copy link
Owner

pikami commented May 2, 2020

@mattcodez No, it can't, this script scrapes the html of the page, no code was written to deal with their api

@rabea98
Copy link

rabea98 commented May 3, 2020

is there anyway to save all of a users videos? or is everything broken as of now?

Use this tool:
https://github.com/drawrowfly/tiktok-scraper

@rabea98
Copy link

rabea98 commented May 6, 2020

@pikami

You sure about that? I used a script before that attached a chromedriver.exe with the .py script and it worked fine (to this day) without the person knowing what version of chrome I had, and I continue to update my real version of chrome but keep that chromedriver.exe that came with it in the folder of the .py

Here's a script that uses selenium and chrome web driver
https://github.com/vashukarn/tiktok-dl
info:
It is a python automated script that downloads all videos of a particular TikTok user at once. As we know that web scraping doesn't work with BeautifulSoup for Dynamic pages in which that code go on updating. So, I used the selenium web browser to scrape videos from TikTok.

@liner601
Copy link

liner601 commented May 8, 2020

@pikami
You sure about that? I used a script before that attached a chromedriver.exe with the .py script and it worked fine (to this day) without the person knowing what version of chrome I had, and I continue to update my real version of chrome but keep that chromedriver.exe that came with it in the folder of the .py

Here's a script that uses selenium and chrome web driver
https://github.com/vashukarn/tiktok-dl
info:
It is a python automated script that downloads all videos of a particular TikTok user at once. As we know that web scraping doesn't work with BeautifulSoup for Dynamic pages in which that code go on updating. So, I used the selenium web browser to scrape videos from TikTok.

Seems to work as a temporary solution but is too slow to be considered a valid option.

@sukrosono
Copy link

here is my opinion
we copy-cat the basic workflow of youtube-dl, but necessarily we add --verbose.
without any option, the stdout is not helping, either for the user or developer.
in my assumption, this just for fetch all video, not just a single video.

that opinion is made without examining the code base, not go developer tho :)

@marcusawereally
Copy link

marcusawereally commented Feb 9, 2021

I am willing to put a bounty on this. $100 USD. Is this okay to do? A working copy in the next couple of weeks and I'll paypal/cashapp/payoneer the bounty.

@sukrosono
Copy link

interesting, if you put 100 USD I may digging trying to submit a PR to this.

@marcusawereally
Copy link

interesting, if you put 100 USD I may digging trying to submit a PR to this.

how do I do that?

@liamengland1
Copy link

I am willing to put a bounty on this. $100 USD. Is this okay to do? A working copy in the next couple of weeks and I'll paypal/cashapp/payoneer the bounty.

don't waste your money, just use youtube-dl while it stills works.

@marcusawereally
Copy link

I am willing to put a bounty on this. $100 USD. Is this okay to do? A working copy in the next couple of weeks and I'll paypal/cashapp/payoneer the bounty.

don't waste your money, just use youtube-dl while it stills works.

Can I do that to download everything from a single user? I am googling it now but coming up empty. Penny for your thoughts?

@marcusawereally
Copy link

FIGURED IT OUT! thanks

@marcusawereally
Copy link

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests