Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

youtube channel page downloads broken #148

Closed
6 tasks done
bani6809 opened this issue Nov 9, 2020 · 29 comments · Fixed by #151 · May be fixed by #150
Closed
6 tasks done

youtube channel page downloads broken #148

bani6809 opened this issue Nov 9, 2020 · 29 comments · Fixed by #151 · May be fixed by #150

Comments

@bani6809
Copy link

bani6809 commented Nov 9, 2020

Checklist

  • I'm reporting a broken site support issue
  • I've verified that I'm running youtube-dlc version 2020.10.31
  • I've checked that all provided URLs are alive and playable in a browser
  • I've checked that all URLs and arguments with special characters are properly quoted or escaped
  • I've searched the bugtracker for similar bug reports including closed ones
  • I've read bugs section in FAQ

Verbose log

youtube-dlc.exe --ignore-config -i -o "%(uploader)s (%(uploader_id)s)/%(upload_date)s - %(title)s - (%(duration)ss) [%(resolution)s] [%(id)s].%(ext)s" --prefer-ffmpeg --merge-output-format mkv --write-sub --all-subs --convert-subs srt --add-metadata --write-description --write-annotations --write-all-thumbnails --write-info-json --cookies youtube-cookies.txt -v --proxy "" https://www.youtube.com/channel/UC7qPftDWPw9XuExpSgfkmJQ
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ignore-config', '-i', '-o', '%(uploader)s (%(uploader_id)s)/%(upload_date)s - %(title)s - (%(duration)ss) [%(resolution)s] [%(id)s].%(ext)s', '--prefer-ffmpeg', '--merge-output-format', 'mkv', '--write-sub', '--all-subs', '--convert-subs', 'srt', '--add-metadata', '--write-description', '--write-annotations', '--write-all-thumbnails', '--write-info-json', '--cookies', 'youtube-cookies.txt', '-v', '--proxy', '', 'https://www.youtube.com/channel/UC7qPftDWPw9XuExpSgfkmJQ']
[debug] Loading archive file None
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] youtube-dlc version 2020.11.07
[debug] Python version 3.9.0 (CPython) - Windows-10-10.0.19041-SP0
[debug] exe versions: ffmpeg git-2020-04-24-a501947, ffprobe git-2020-04-24-a501947
[debug] Proxy map: {}
[youtube:channel] UC7qPftDWPw9XuExpSgfkmJQ: Downloading channel page
[youtube:playlist] UU7qPftDWPw9XuExpSgfkmJQ: Downloading webpage
[download] Downloading playlist: UU7qPftDWPw9XuExpSgfkmJQ
[youtube:playlist] playlist UU7qPftDWPw9XuExpSgfkmJQ: Downloading 0 videos
[download] Finished downloading playlist: UU7qPftDWPw9XuExpSgfkmJQ

Description

youtube playlist downloads seem to have broken sometime within the last few hours. it was working earlier this morning, now all playlists result in "0 videos" from youtube-dlc. presumably youtube made a schema change which broke youtube-dlc.

Individual video downloads still work, for example this works perfectly:

youtube-dlc.exe --ignore-config -i -o "%(uploader)s (%(uploader_id)s)/%(upload_date)s - %(title)s - (%(duration)ss) [%(resolution)s] [%(id)s].%(ext)s" --prefer-ffmpeg --merge-output-format mkv --write-sub --all-subs --convert-subs srt --add-metadata --write-description --write-annotations --write-all-thumbnails --write-info-json --cookies youtube-cookies.txt -v --proxy "" https://www.youtube.com/watch?v=SXGAiTjFiMA

@someziggyman
Copy link

someziggyman commented Nov 9, 2020

@insaneracist @blackjack4494
I can confirm that this is broken for links like:
https://www.youtube.com/user/anjunadeep/videos
and "regular" playlists like this ID: PLOftnzGIKwJAfH33eTU2wDsBqwxOMRedV
Full link: https://www.youtube.com/watch?v=mhpfOQ6sf3s&list=PLOftnzGIKwJAfH33eTU2wDsBqwxOMRedV
or this:
https://www.youtube.com/playlist?list=RDCLAK5uy_m_h-nx7OCFaq9AlyXv78lG0AuloqW_NUA

Mix, search and user created playlists seem to be working just fine:
mix id to test: RDKR9wGi7gVLQ
search playlist to test: https://www.youtube.com/results?search_query=linkin+park+numb

@bani6809
Copy link
Author

bani6809 commented Nov 9, 2020

@ainsey11
Copy link

ainsey11 commented Nov 9, 2020

I too am seeing this issue, looks to be a global thing, same problem wherever I initiate the download from

[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'https://www.youtube.com/playlist?list=PLvgS71fU12Mbx-w18Chu_Sg9v6loipEFO', u'--verbose']
[debug] Loading archive file None
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dlc version 2020.11.07
[debug] Python version 2.7.17 (CPython) - Linux-4.19.104-microsoft-standard-x86_64-with-Ubuntu-18.04-bionic
[debug] exe versions: ffmpeg 3.4.8, ffprobe 3.4.8, phantomjs ., rtmpdump 2.4
[debug] Proxy map: {}
[youtube:playlist] PLvgS71fU12Mbx-w18Chu_Sg9v6loipEFO: Downloading webpage
[download] Downloading playlist: PLvgS71fU12Mbx-w18Chu_Sg9v6loipEFO
[youtube:playlist] playlist PLvgS71fU12Mbx-w18Chu_Sg9v6loipEFO: Downloading 0 videos
[download] Finished downloading playlist: PLvgS71fU12Mbx-w18Chu_Sg9v6loipEFO```

@ainsey11
Copy link

ainsey11 commented Nov 9, 2020

It would appear that youtube has disabled the "disable_polymer" feature that the YT extractor requires,

When visiting a playlist with the flag set, I now get unknown param

image

I can see the extractor is doing `

        query['disable_polymer'] = 'true'

so that is my hunch

@jbruchon
Copy link
Contributor

jbruchon commented Nov 9, 2020

@ainsey11 I can second what you've found there. Seems you're on the right track.

@etec-masterofsynapse
Copy link

Sorry if its a dumb question: Why does the extractor need the "old theme / non-polymer"?
Is it not currently able to pull the video IDs from the new theme?

@wlritchi
Copy link
Contributor

Sorry if its a dumb question: Why does the extractor need the "old theme / non-polymer"?
Is it not currently able to pull the video IDs from the new theme?

The extractor currently looks for HTML markup for each entry in a playlist. When the polymer frontend was introduced, YouTube changed from server-side rendering of HTML templates to client-side rendering. As a result, the HTML source received from the server no longer contains the markup the extractor looks for, and the only place that playlist items appear is in a JSON structure (and a fairly dense one at that). Now that the extractor can no longer opt out of polymer, it will need to be updated to pull data out of that structure.

@ZenulAbidin
Copy link
Contributor

It seems to be failing at youtube_dlc/extractor/youtube.py:_extract_playlist

https://github.com/blackjack4494/yt-dlc/blob/master/youtube_dlc/extractor/youtube.py#L2890-L2892

Every match after these lines isn't working.

@candrapersada
Copy link

candrapersada commented Nov 10, 2020

Individual video downloads still work, for example this works perfectly

how to use playlist track number from Individual video downloads? &index=2
Full link https://www.youtube.com/watch?v=QW7vi-LjzXI&list=PLOftnzGIKwJAfH33eTU2wDsBqwxOMRedV&index=2

@jbruchon
Copy link
Contributor

@candrapersada That's fine as a hack, but for all of us who do archival, it's a really bad hack to put in place. Using that method, there is no way to scan the archive for all of the playlist items and avoid downloading the page for every single one of those items. I archive over 150 channels regularly and dropping down to this method would be a little bit of a disaster.

@jbruchon
Copy link
Contributor

If no one has taken up the task of getting this fixed yet, I might give it a try tomorrow.

@blackjack4494
Copy link
Owner

Check network tab in browser. They likely completely changed over to api usage returning json instead of html events

@candrapersada
Copy link

what I mean is how to want youtube-dlc to name the files like 1-{name}, 2-{name}, ... n-{name} in order to be able to watch them in the same sequence as original Youtube playlist. In other words I need my downloaded videos to be prefixed with numbers. How can I do that?

@insaneracist
Copy link
Contributor

@candrapersada, check out the readme section called output template
if playlist downloading isn't broken, you can do
youtube-dlc -o "%(playlist_index)03d - %(title)s.%(ext)s" "https://www.youtube.com/playlist?list=..."

@arter97
Copy link

arter97 commented Nov 10, 2020

Hm, the playlist's title is still missing from these fixes @ZenulAbidin @insaneracist

Similar to #109.

python3.9 ./__main__.py --flat-playlist --add-metadata --all-subs -J -F 'https://youtube.com/playlist?list=PLq-W-cmkohAPAcxZWxh_ysJWPhqDxkETs'

Doesn't return a valid title key.

@insaneracist
Copy link
Contributor

@arter97 fixed

@blackjack4494
Copy link
Owner

Expect a release with a patch for this coming today (anything between 1-10 hrs depending on when I got time). Python version will be fixed then as well as win xp support is back then.

@ZenulAbidin
Copy link
Contributor

I was booked today, now that I have some free time I’ll see where in the JSON our variables have moved to and hopefully find a regex to match them.

@blackjack4494
Copy link
Owner

That's unfortunate
actions/upload-release-asset#68

Release is delayed till this issue is fixed.

@perry-mitchell
Copy link

@blackjack4494 Seems that the linked issue is fixed, according to comments. Would you have a potential ETA on release? Thank you 🙏

@candrapersada
Copy link

candrapersada commented Nov 11, 2020

why name folder broken %(playlist)s/ and %(playlist_title)s/?

@harrypotter0
Copy link

Hi everyone. I am trying to use this command to download playlist but not able to do so since past 2-3 days. Till now I can't see any fix for this. If anyone can tell me how to go past this issue it would be great help.

youtube-dl -c -i -f 18  https://www.youtube.com/playlist?list=PLv_5Gzr5T6wz0G1Ff3SO97ElLq1G0POWt
[youtube:playlist] PLv_5Gzr5T6wz0G1Ff3SO97ElLq1G0POWt: Downloading webpage
[download] Downloading playlist: PLv_5Gzr5T6wz0G1Ff3SO97ElLq1G0POWt
[youtube:playlist] playlist PLv_5Gzr5T6wz0G1Ff3SO97ElLq1G0POWt: Downloading 0 videos
[download] Finished downloading playlist: PLv_5Gzr5T6wz0G1Ff3SO97ElLq1G0POWt

@Fabian42
Copy link

@harrypotter0 Which system are you using? For Manjaro Linux, a package update was available when I checked 9 hours ago, since then it works again. Some other distributions might take longer. Debian for example sometimes takes months, so you might want to download it from Github and install it that way instead.

@someziggyman
Copy link

Hi everyone. I am trying to use this command to download playlist but not able to do so since past 2-3 days. Till now I can't see any fix for this. If anyone can tell me how to go past this issue it would be great help.

youtube-dl -c -i -f 18  https://www.youtube.com/playlist?list=PLv_5Gzr5T6wz0G1Ff3SO97ElLq1G0POWt
[youtube:playlist] PLv_5Gzr5T6wz0G1Ff3SO97ElLq1G0POWt: Downloading webpage
[download] Downloading playlist: PLv_5Gzr5T6wz0G1Ff3SO97ElLq1G0POWt
[youtube:playlist] playlist PLv_5Gzr5T6wz0G1Ff3SO97ElLq1G0POWt: Downloading 0 videos
[download] Finished downloading playlist: PLv_5Gzr5T6wz0G1Ff3SO97ElLq1G0POWt

This particular playlist is working on the latest release.
try also using --rm-cache-dir --no-cache-dir params in your command.

@cgerrie
Copy link

cgerrie commented Nov 13, 2020

When I download my Youtube watch later playlist, it only downloads the first page. Any videos beyond the first 100 don't get downloaded. This happens using both youtube-dlc --cookies cookies.txt :ytwatchlater and youtube-dlc --cookies cookies.txt https://www.youtube.com/playlist?list=WL. I am using version 2020.11.11-3, and I've cleared my cache directory. I've tried messing with the --playlist-start and --playlist-end flags, but those don't seem to help either. Could this be because the watch later playlist has some unique polymer structure that got missed in the patch?

@theAkito
Copy link

theAkito commented Dec 2, 2020

I updated to today's youtube-dl version through pip. Still, none of the auxiliary options like --rm-cache-dir --no-cache-dir seem to work. Tried to download several completely different channels' videos and yet, it always tells me, that there are only 0 videos to download, which is absolutely false.

What am I missing here? Is this truly fixed?

@blackjack4494
Copy link
Owner

Try updating to master with git. Search for git in readme :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
17 participants