batch download / batch dl from list

[SevensRequiem]

I want to dl alot of anime that I watch / have watched. So maybe having support to batch dl from a file or from anilist/myanimelist as either sub or dub or both, would be really nice, or even able to download all seasons from a specific anime. I have over 600 + in my lists so batch dl would be nice :pp

[FireHead90544]

That's a really great idea. I started this project to be atleast a bare minimum downloader that looked cool and actually works lol. I'm currently busy in some other project, but surely will implement bulk downloading (i.e, through reading from file, mal, anilist etc.) in upcoming future. Marked it as enchancement request for now :)

[Arctic4161]

@FireHead90544 I will work on this in my spare time and push to the project. I also want to work on the back button. I'll create a pull request when I get it done.

I have a couple commits for this project optimizing the code, but I'll push when I do the other two.

[FireHead90544]

I see, thank you. Sure, take your time. I am planning to write a custom video player maybe using mpd and stuff later on to enable tracking and a sync with mal later too. Dunno when I'll start, but yeah it'll be later.

[Arctic4161]

@SevensRequiem In what form is this list stored as? It is bookmarked under your gogo account. I'd need to know what I am reading in order to have a start point.

[FireHead90544]

Let's adapt a simpler convention. A text file with anime-id, a single anime per line. We'll expose a wrapper or a simple utility later on to pull the list from mal and get it in the desired format. What's your take on this?

[Arctic4161]

@FireHead90544 Works for me. I have changed around some stuff and allowed the user to navigate update config. Can you go to my fork and run it and see if it works for you? I don't have aria2 so can't fully test it but I think its the only way to put in a back button for the update config options. I still have yet to work out the download anime back buttons.

[FireHead90544]

Alright, let me setup a virtual env and test it up.

[FireHead90544]

Just tested it out. Awesome work. It isn't breaking or anything, the downloader's working fine.

Just a suggestion, instead of increasing the memory usage/messy structure by creating many variables for updating config, you can simply update the dictionary variable which is loaded when the program is started, in the main.py file it can be accessed as client.config.loaded_config. After the updates are done, just write it out to the client.config.config_path. Using this, you probably don't even need to restart the program (need to try before being sure) as the config variables used by program are themselves being updated.

And this will help keeping the program structure the way it is, short and precise.

[Arctic4161]

I'll give it a look when I have time, thanks for the suggestion!

[Arctic4161]

@FireHead90544 Just finished adding in back buttons. I also took your suggestion and added in the writing of the config file into client.config. I decided to keep the client.config.loaded_config variable alone until the user decides to write it to file. That way the client.config.loaded_config variable from the file is used when they go to download the videos. I did make the Keys in the new dictionary mirror the keys in client.config.loaded_config that way the keys in client.config.loaded_config can be easily updated once they wirte the file without restarting the application. Downloading anime option is a little slower this way since it grabs the cookies and uses client.config.loaded_config instead of at the start of the application. I hope this makes sense. I created a pull request with these changes and will work on the bulk download option next.

[FireHead90544]

Awesomee, I'll check that out in a while and merge. Btw I might make some changes too so make sure to fetch the upstream before pushing anything again to prevent any rebasing issue later.

[Arctic4161]

@FireHead90544 alright cool, I'll wait till you do what you have to and pull down before moving onto the bulk download. I think what you have going on here is a great

[FireHead90544]

Thanks buddy! You really helped implementing half of the brain-eating stuffs too. Just released v1.1.0. Quite late ik, just got free with my other stuffs. Let's work on this one now.

[Arctic4161]

@SevensRequiem Can you posts an example of what you have in your list? I'd like to go about it as an opportunistic search with a list and use that to build out a file of dub or sub anime URLs that can be passed to aria2.

[FireHead90544]

Hey @Arctic4161,
I think providing a separate utility class for this purpose would be awesome, exposing one method for one type of format. I really liked the idea of building out a dump file that can be read and act as a universal format and build absolute urls.
For example,

class BatchDumpExporter:
   def parse_list(*args, **kwargs):
      ... # write to the dump file
   def parse_text_file(*args, **kwargs):
      ... # write to the dump file
   ...

The dump file could be of certain format, maybe like either each line containing {animeid}-{ep} or {animeid},{ep} (csv kinda thing), or anything else. What's your take on this?

Otherwise, as you said, set a specified format for batch exports file and use that to generate a file that could be passed to aria2 for bulk downloads.

[FireHead90544]

The advantage of first approach would be that we could write a parser to read data from someone's MAL profile or so

[Arctic4161]

@FireHead90544 I like it, I don't use MAL but that would make it 10x easier on us if we could parse something that's a known format.

So if we read in something from MAL with a known format. Build out a list or dictionary of anime names and IDs. Fetch the URls and then build out a URL text file that can be fed into aria2 I think that would work.

Aria2 can read files for download with the -I or "--input-file=" flag. I dabbled in that with gogo Downloader.
Format needs to be for example:

http://example.com/foo
dir=C:\pdfs
out=0.pdf
http://example.com/bar
dir=C:\pdfs
out=0.pdf
I don't think we need to specify the file out name though.

Since this may be a long list of items to download we should also use the "--save-session=" flag and then iterate over that when aria2 finishes in case some fail. The save session flag will save the URls that it failed on in a file.

[FireHead90544]

@Arctic4161 agreed, this works. Let's finish and merge your PR first before starting to work on this. If you'd like, I can add you as a Contributor to this repo with write access so that you won't need to generate a PR each time.

[Arctic4161]

@Arctic4161 agreed, this works. Let's finish and merge your PR first before starting to work on this. If you'd like, I can add you as a Contributor to this repo with write access so that you won't need to generate a PR each time.

I think that'd be great. I don't think I'd ever merge without discussing it first or having it tested out tbh.

[FireHead90544]

@Arctic4161 agreed, this works. Let's finish and merge your PR first before starting to work on this. If you'd like, I can add you as a Contributor to this repo with write access so that you won't need to generate a PR each time.

I think that'd be great. I don't think I'd ever merge without discussing it first or having it tested out tbh.

Yeah, it works. We can always do the tests on a separate branch before merging the changes to main.
Sure, let's merge your PR first, I'll add you as a contributor afterwards so this pr doesn't create any merge conflicts.

[FireHead90544]

Finalized it, let's start working on this.

[Arctic4161]

Finalized it, let's start working on this.

Awesome, I pulled and my fork is up to date. I'll start dabbling and working some ideas on Monday importing MAL

[FireHead90544]

Finalized it, let's start working on this.

Awesome, I pulled and my fork is up to date. I'll start dabbling and working some ideas on Monday importing MAL

Sure sure, lemme know if you need any help or want me to write some certain portion in the meantime.

[Arctic4161]

Finalized it, let's start working on this.

Awesome, I pulled and my fork is up to date. I'll start dabbling and working some ideas on Monday importing MAL

Sure sure, lemme know if you need any help or want me to write some certain portion in the meantime.

If you have the time and have an idea of what you want just let me know what portion you want to work on. I was planning on adding in a batch download selection on the menu and figuring out how to parse MAL

[FireHead90544]

Finalized it, let's start working on this.

Awesome, I pulled and my fork is up to date. I'll start dabbling and working some ideas on Monday importing MAL

Sure sure, lemme know if you need any help or want me to write some certain portion in the meantime.

If you have the time and have an idea of what you want just let me know what portion you want to work on. I was planning on adding in a batch download selection on the menu and figuring out how to parse MAL

I'll probably refactor the code, work on format that can be inputted to aria2 for batch downloads and plan on rewriting some portions to be more pythonic. For example, currently interaction with aria2 is just a system call to shell to execute aria2, there's no direct way of monitoring it's output/interact with aria2. Writing a wrapper around it would be cool.

Just made a Github Project (something similar to a trello board), for jotting down the ideas. Added you as a collaborator too, feel free to add anything that comes up to your mind.

[Arctic4161]

Finalized it, let's start working on this.

Awesome, I pulled and my fork is up to date. I'll start dabbling and working some ideas on Monday importing MAL

Sure sure, lemme know if you need any help or want me to write some certain portion in the meantime.

If you have the time and have an idea of what you want just let me know what portion you want to work on. I was planning on adding in a batch download selection on the menu and figuring out how to parse MAL

I'll probably refactor the code, work on format that can be inputted to aria2 for batch downloads and plan on rewriting some portions to be more pythonic. For example, currently interaction with aria2 is just a system call to shell to execute aria2, there's no direct way of monitoring it's output/interact with aria2. Writing a wrapper around it would be cool.

Just made a Github Project (something similar to a trello board), for jotting down the ideas. Added you as a collaborator too, feel free to add anything that comes up to your mind.

A wrapper would be cool. I'm surprised there's not already one. If you want to make one create a repository and I'll help with that one as well.

I also want to implement the bookmark parser that was in Gogo into the batch download option. That way you can either do the bookmarks from Anitaku or you can do the MAL.

We should also add in an option to set dub or not in the config for the batch downloads. Mal doesn't differentiate between dub or sub

[FireHead90544]

Yeah ikr, there is a popular one aria2p but it doesn't get frequent updates and it breaks most of the time. So implementing one ourselves would be better. Sure, I'll create a repo and add you as a collaborator there as well.

Yeah bookmark parser would be a good addition, and yeah setting a conf option for sub/sub would be required.

[FireHead90544]

BTW, the convos here are too cluttered, do you use discord?

[Arctic4161]

BTW, the convos here are too cluttered, do you use discord?

Haha I was just about to ask you that. What's a secure way to exchange that information?

[FireHead90544]

BTW, the convos here are too cluttered, do you use discord?

Haha I was just about to ask you that. What's a secure way to exchange that information?

I guess just drop it in the contact section of my website. Since the backend's written by me, I'll simply collect it from the database.

[FireHead90544]

Under progress on branch SenPY/Batch-Download