Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] YouTube sign In Bot Protection #335

Open
jared-bloomer opened this issue Sep 12, 2024 · 7 comments
Open

[BUG] YouTube sign In Bot Protection #335

jared-bloomer opened this issue Sep 12, 2024 · 7 comments
Assignees
@benoit74
Labels
bug upstream
Milestone
backlog

Comments

@jared-bloomer
Copy link

I cloned the repo and did a docker build of the image. I then ran the following command to try and get an entire channel on YouTube

docker run youtube2zim:latest youtube2zim --api-key <redacted for security> --type channel --name KM4ACK --id UCSQhXfGo_68Ta8-2wStAWkw --zim-file KM4ACK_20240910.zim --description "YouTube Channel for KM4ACK" --creator KM4ACK --tags "Amatuer Radio,youtube,km4ack" --format webm

Every single video it tries to grab logs a message like this

[youtube] Extracting URL: vYQVMYT3QKA
[youtube] vYQVMYT3QKA: Downloading webpage
[youtube] vYQVMYT3QKA: Downloading ios player API JSON
[youtube] vYQVMYT3QKA: Downloading web creator player API JSON
ERROR: [youtube] vYQVMYT3QKA: Sign in to confirm you’re not a bot. This helps protect our community. Learn more
[youtube2zim::2024-09-12 22:40:32,732] ERROR:Video file for vYQVMYT3QKA could not be downloaded

I am not sure if it is something I am doing, or if the YouTube Bot protection has essentially broken this tool. Looking at the help for youtube2zim I do not see a way to specify youtube login credentials, and I would assume the API key would work fine once the YouTube Data API v3 has been added to it. Is there more I need to add to my API key to make this work?
@jared-bloomer
Copy link
Author

So it appears, also adding YouTube Embedded Player API to my API Key may have resolved this. The process is still running after about an hour. I will follow up once it completes or fails.

@jared-bloomer
Copy link
Author

Guess I was wrong, still failing. Shortly after my last comment the log started being filled with

[youtube] Extracting URL: id65x7CsyEE
[youtube] id65x7CsyEE: Downloading webpage
[youtube] id65x7CsyEE: Downloading ios player API JSON
[youtube] id65x7CsyEE: Downloading web creator player API JSON
ERROR: [youtube] id65x7CsyEE: Sign in to confirm you’re not a bot. This helps protect our community. Learn more
[youtube2zim::2024-09-13 00:32:19,947] ERROR:Video file for id65x7CsyEE could not be downloaded

@jared-bloomer
Copy link
Author

Looking at the log I am thinking this maybe a rate limiting thing from downloading too many videos too quickly. It fails on several (5-10 usually) then gets 1 successfully.

@jared-bloomer
Copy link
Author 

[youtube] Extracting URL: XhDRCrSPwK4
[youtube] XhDRCrSPwK4: Downloading webpage
[youtube] XhDRCrSPwK4: Downloading ios player API JSON
[youtube] XhDRCrSPwK4: Downloading web creator player API JSON
[youtube] XhDRCrSPwK4: Downloading m3u8 information
[info] XhDRCrSPwK4: Downloading 1 format(s): 243+251
[download] Destination: /tmp/tmpkfm_1iwy/videos/XhDRCrSPwK4/video.f243.webm
[download] 100% of   12.52MiB in 00:00:03 at 3.62MiB/s
[download] Destination: /tmp/tmpkfm_1iwy/videos/XhDRCrSPwK4/video.f251.webm
[download] 100% of    6.90MiB in 00:00:02 at 3.35MiB/s
[Merger] Merging formats into "/tmp/tmpkfm_1iwy/videos/XhDRCrSPwK4/video.webm"
Deleting original file /tmp/tmpkfm_1iwy/videos/XhDRCrSPwK4/video.f243.webm (pass -k to keep)
Deleting original file /tmp/tmpkfm_1iwy/videos/XhDRCrSPwK4/video.f251.webm (pass -k to keep)
[youtube] Extracting URL: XhDRCrSPwK4
[youtube] XhDRCrSPwK4: Downloading webpage
[youtube] XhDRCrSPwK4: Downloading ios player API JSON
[youtube] XhDRCrSPwK4: Downloading web creator player API JSON
ERROR: [youtube] XhDRCrSPwK4: Sign in to confirm you’re not a bot. This helps protect our community. Learn more
[youtube2zim::2024-09-13 00:38:15,614] ERROR:Thumbnail for XhDRCrSPwK4 could not be downloaded
[youtube] Extracting URL: Z_nIp5v7UeI
[youtube] Z_nIp5v7UeI: Downloading webpage
[youtube] Z_nIp5v7UeI: Downloading ios player API JSON
[youtube] Z_nIp5v7UeI: Downloading web creator player API JSON
ERROR: [youtube] Z_nIp5v7UeI: Sign in to confirm you’re not a bot. This helps protect our community. Learn more
[youtube2zim::2024-09-13 00:38:16,268] ERROR:Video file for Z_nIp5v7UeI could not be downloaded
[youtube] Extracting URL: unal5dCKGXk
[youtube] unal5dCKGXk: Downloading webpage
[youtube] unal5dCKGXk: Downloading ios player API JSON
[youtube] unal5dCKGXk: Downloading web creator player API JSON
ERROR: [youtube] unal5dCKGXk: Sign in to confirm you’re not a bot. This helps protect our community. Learn more
[youtube2zim::2024-09-13 00:38:16,968] ERROR:Video file for unal5dCKGXk could not be downloaded
[youtube] Extracting URL: xXJsJKgU-mc
[youtube] xXJsJKgU-mc: Downloading webpage
[youtube] xXJsJKgU-mc: Downloading ios player API JSON
[youtube] xXJsJKgU-mc: Downloading web creator player API JSON
ERROR: [youtube] xXJsJKgU-mc: Sign in to confirm you’re not a bot. This helps protect our community. Learn more
[youtube2zim::2024-09-13 00:38:17,641] ERROR:Video file for xXJsJKgU-mc could not be downloaded
[youtube] Extracting URL: 0yEVdyvGqB8
[youtube] 0yEVdyvGqB8: Downloading webpage
[youtube] 0yEVdyvGqB8: Downloading ios player API JSON
[youtube] 0yEVdyvGqB8: Downloading web creator player API JSON
ERROR: [youtube] 0yEVdyvGqB8: Sign in to confirm you’re not a bot. This helps protect our community. Learn more
[youtube2zim::2024-09-13 00:38:18,295] ERROR:Video file for 0yEVdyvGqB8 could not be downloaded
[youtube] Extracting URL: Q1TjE11cqJg
[youtube] Q1TjE11cqJg: Downloading webpage
[youtube] Q1TjE11cqJg: Downloading ios player API JSON
[youtube] Q1TjE11cqJg: Downloading web creator player API JSON
ERROR: [youtube] Q1TjE11cqJg: Sign in to confirm you’re not a bot. This helps protect our community. Learn more
[youtube2zim::2024-09-13 00:38:18,973] ERROR:Video file for Q1TjE11cqJg could not be downloaded
[youtube] Extracting URL: 1NxYm-FCz6E
[youtube] 1NxYm-FCz6E: Downloading webpage
[youtube] 1NxYm-FCz6E: Downloading ios player API JSON
[youtube] 1NxYm-FCz6E: Downloading web creator player API JSON
ERROR: [youtube] 1NxYm-FCz6E: Sign in to confirm you’re not a bot. This helps protect our community. Learn more
[youtube2zim::2024-09-13 00:38:19,674] ERROR:Video file for 1NxYm-FCz6E could not be downloaded
[youtube] Extracting URL: YN2Aa4E1YtU
[youtube] YN2Aa4E1YtU: Downloading webpage
[youtube] YN2Aa4E1YtU: Downloading ios player API JSON
[youtube] YN2Aa4E1YtU: Downloading web creator player API JSON
ERROR: [youtube] YN2Aa4E1YtU: Sign in to confirm you’re not a bot. This helps protect our community. Learn more
[youtube2zim::2024-09-13 00:38:20,362] ERROR:Video file for YN2Aa4E1YtU could not be downloaded
[youtube] Extracting URL: uxheH96S_i4
[youtube] uxheH96S_i4: Downloading webpage
[youtube] uxheH96S_i4: Downloading ios player API JSON
[youtube] uxheH96S_i4: Downloading web creator player API JSON
[youtube] uxheH96S_i4: Downloading m3u8 information
ERROR: [youtube] uxheH96S_i4: Requested format is not available. Use --list-formats for a list of available formats
[youtube2zim::2024-09-13 00:38:21,109] ERROR:Video file for uxheH96S_i4 could not be downloaded
[youtube] Extracting URL: S6lX6CZuOF4
[youtube] S6lX6CZuOF4: Downloading webpage
[youtube] S6lX6CZuOF4: Downloading ios player API JSON
[youtube] S6lX6CZuOF4: Downloading web creator player API JSON
[youtube] S6lX6CZuOF4: Downloading m3u8 information
ERROR: [youtube] S6lX6CZuOF4: Requested format is not available. Use --list-formats for a list of available formats
[youtube2zim::2024-09-13 00:38:21,862] ERROR:Video file for S6lX6CZuOF4 could not be downloaded
[youtube] Extracting URL: CO-0qSpFE20
[youtube] CO-0qSpFE20: Downloading webpage
[youtube] CO-0qSpFE20: Downloading ios player API JSON
[youtube] CO-0qSpFE20: Downloading web creator player API JSON
ERROR: [youtube] CO-0qSpFE20: Sign in to confirm you’re not a bot. This helps protect our community. Learn more
[youtube2zim::2024-09-13 00:38:22,486] ERROR:Video file for CO-0qSpFE20 could not be downloaded
[youtube] Extracting URL: dhBZ5cfCv1c
[youtube] dhBZ5cfCv1c: Downloading webpage
[youtube] dhBZ5cfCv1c: Downloading ios player API JSON
[youtube] dhBZ5cfCv1c: Downloading web creator player API JSON
ERROR: [youtube] dhBZ5cfCv1c: Sign in to confirm you’re not a bot. This helps protect our community. Learn more
[youtube2zim::2024-09-13 00:38:23,092] ERROR:Video file for dhBZ5cfCv1c could not be downloaded
[youtube] Extracting URL: WuIvkKLs5XA
[youtube] WuIvkKLs5XA: Downloading webpage
[youtube] WuIvkKLs5XA: Downloading ios player API JSON
[youtube] WuIvkKLs5XA: Downloading web creator player API JSON
[youtube] WuIvkKLs5XA: Downloading m3u8 information
ERROR: [youtube] WuIvkKLs5XA: Requested format is not available. Use --list-formats for a list of available formats
[youtube2zim::2024-09-13 00:38:23,948] ERROR:Video file for WuIvkKLs5XA could not be downloaded
[youtube] Extracting URL: Dsq_9aE6fc0
[youtube] Dsq_9aE6fc0: Downloading webpage
[youtube] Dsq_9aE6fc0: Downloading ios player API JSON
[youtube] Dsq_9aE6fc0: Downloading web creator player API JSON
ERROR: [youtube] Dsq_9aE6fc0: Sign in to confirm you’re not a bot. This helps protect our community. Learn more
[youtube2zim::2024-09-13 00:38:24,508] ERROR:Video file for Dsq_9aE6fc0 could not be downloaded
[youtube] Extracting URL: iUOfsOnOmvQ
[youtube] iUOfsOnOmvQ: Downloading webpage
[youtube] iUOfsOnOmvQ: Downloading ios player API JSON
[youtube] iUOfsOnOmvQ: Downloading web creator player API JSON
ERROR: [youtube] iUOfsOnOmvQ: Sign in to confirm you’re not a bot. This helps protect our community. Learn more
[youtube2zim::2024-09-13 00:38:25,133] ERROR:Video file for iUOfsOnOmvQ could not be downloaded
[youtube] Extracting URL: 84OIKKt8eKM
[youtube] 84OIKKt8eKM: Downloading webpage
[youtube] 84OIKKt8eKM: Downloading ios player API JSON
[youtube] 84OIKKt8eKM: Downloading web creator player API JSON
ERROR: [youtube] 84OIKKt8eKM: Sign in to confirm you’re not a bot. This helps protect our community. Learn more
[youtube2zim::2024-09-13 00:38:25,740] ERROR:Video file for 84OIKKt8eKM could not be downloaded
[youtube] Extracting URL: pN8Q3G_SwqM
[youtube] pN8Q3G_SwqM: Downloading webpage
[youtube] pN8Q3G_SwqM: Downloading ios player API JSON
[youtube] pN8Q3G_SwqM: Downloading web creator player API JSON
ERROR: [youtube] pN8Q3G_SwqM: Sign in to confirm you’re not a bot. This helps protect our community. Learn more
[youtube2zim::2024-09-13 00:38:26,335] ERROR:Video file for pN8Q3G_SwqM could not be downloaded
[youtube] Extracting URL: Uy6xbC0Hiew
[youtube] Uy6xbC0Hiew: Downloading webpage
[youtube] Uy6xbC0Hiew: Downloading ios player API JSON
[youtube] Uy6xbC0Hiew: Downloading web creator player API JSON
[info] Uy6xbC0Hiew: Downloading 1 format(s): 243+251
[download] Destination: /tmp/tmpkfm_1iwy/videos/Uy6xbC0Hiew/video.f243.webm
[download] 100% of    8.09MiB in 00:00:02 at 3.83MiB/s
[download] Destination: /tmp/tmpkfm_1iwy/videos/Uy6xbC0Hiew/video.f251.webm
[download] 100% of   10.67MiB in 00:00:02 at 5.18MiB/s
[Merger] Merging formats into "/tmp/tmpkfm_1iwy/videos/Uy6xbC0Hiew/video.webm"
Deleting original file /tmp/tmpkfm_1iwy/videos/Uy6xbC0Hiew/video.f243.webm (pass -k to keep)
Deleting original file /tmp/tmpkfm_1iwy/videos/Uy6xbC0Hiew/video.f251.webm (pass -k to keep)
[youtube] Extracting URL: Uy6xbC0Hiew
[youtube] Uy6xbC0Hiew: Downloading webpage
[youtube] Uy6xbC0Hiew: Downloading ios player API JSON
[youtube] Uy6xbC0Hiew: Downloading web creator player API JSON
[info] Uy6xbC0Hiew: Downloading 1 format(s): 243+251
[info] Downloading video thumbnail 37 ...
[info] Writing video thumbnail 37 to: /tmp/tmpkfm_1iwy/videos/Uy6xbC0Hiew/video.webp

@benoit74 benoit74 self-assigned this Sep 13, 2024
@benoit74
Copy link
Collaborator

Thank you for reporting this!

This is a known issue with yt-dlp: yt-dlp/yt-dlp#10128, I forgot to mention it in this repo.

Unfortunately it looks like there is little to do, Youtube is becoming more aggressive with banning IPs which look illegitimate from their PoV. The ban can be temporary or permanent. And it looks like it is based on IP history + ASN, i.e. if you run the scraper from a residential IP you have less chances to be blocked than when you run it from a server on a hosting provider. But risking banning your own residential IP is probably not something you want to consider, might cause some trouble at home ^^

@jared-bloomer
Copy link
Author

No worries. I am thinking some logic change may be a way to get around this. Something like (obviously not working code, because it is not for a specific language but more of a flowchart idea).

for video in video_list; do
  if type == "channel"; then
    Download a video
    wait 10 seconds
  else
    Download video
  fi
done

@benoit74
Copy link
Collaborator

I don't think the problem happens only when type is channel. And from what I've read and what we've experienced, adding a delay does not help at all avoiding Youtube Bot protection, Google engineers have done something more clever than it might seem ^^

@benoit74 benoit74 mentioned this issue Sep 23, 2024
Open
@benoit74 benoit74 added this to the 3.3.0 milestone Oct 18, 2024
@benoit74 benoit74 modified the milestones: 3.3.0, backlog Nov 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@benoit74 benoit74

Labels
bug upstream
Projects
None yet
Milestone
backlog
Development

No branches or pull requests
2 participants
@benoit74 @jared-bloomer