What to add to config to get it to grab .htm file of a page?

[Jeddostotle7]

OOOOOOK, after a long laborious process that involved much trial and error, I have now figured out how to make the program run as needed; actually utilizing the function to grab a description of a work and put it in a separate .txt file, how to make it put works from different artists in different directories, how to make it continue where it left off. Almost all my previous questions that I'd posited in this question post are now moot. I think I now have only one main question remaining:

What would be an example of some code to add to the config file (specifically under the extractors for Deviantart, though would like something that could apply to other sites too) to get it to also simply nab the .htm file for the page of each image? Maybe put it in its own directory like with the descriptions?

There is also one secondary thing, small: how exactly would I modify this:

"favorite": {
                "directory": ["{category}", "{username}", "Favourites", "{author[username]}"]
            },

to first split the Favourites directory into directories based on the collections within that user's Favourites, and then within those directories into the further directories separating out by author? Basically what specific word do I add between '"Favourites",' and '"{author[username]}"'?

[mikf]

What would be an example of some code to add to the config file (specifically under the extractors for Deviantart, though would like something that could apply to other sites too) to get it to also simply nab the .htm file for the page of each image? Maybe put it in its own directory like with the descriptions?

I guess you could use an exec post processor to download the HTML with curl and such, but gallery-dl cannot really do this on its own. The only data it has is from the API, which does not include any HTML except descriptions.

to first split the Favourites directory into directories based on the collections within that user's Favourites, and then within those directories into the further directories separating out by author? Basically what specific word do I add between '"Favourites",' and '"{author[username]}"'?

collection[title]

You can get a list of all available names with -K:

$ gallery-dl -K https://www.deviantart.com/aldenm750/favourites/90576767/featured
...
collection[title]
  Featured
...

[Jeddostotle7]

I guess you could use an exec post processor to download the HTML with curl and such, but gallery-dl cannot really do this on its own. The only data it has is from the API, which does not include any HTML except descriptions.

Hmm, so there is a way to get it to call on curl to do it? I guess I'll have to look into curl after I wake up then. I'll try to figure out a functioning exec postprocessor after installing curl and figuring it out and then looking around the documentation here for exec, and if I can't manage it, I'll ask here for an example bit of code.

collection[title]

You can get a list of all available names with -K:

$ gallery-dl -K https://www.deviantart.com/aldenm750/favourites/90576767/featured
...
collection[title]
  Featured
...

I was having trouble with this because for some reason I was expecting it to be able to differentiate and identify multiple collections from the main "favourites" page or "favourites/all" and so tried modifying the config and running the same command as before to no avail, and then ran the -K command on my test URL and didn't see any collection stuff among the names it brought up, but then I looked again at the example command you used there and realized it was for a specific collection, and so tried it again with a specific collection in my test URL and it worked like a charm. Thanks for the help! Will mean I'll have to run multiple instances of the command, one for each folder, but that'll probably help get through it quicker anyway.

EDIT: I actually just decided to test the multiple instances thing first before I went to bed to see if and how it would work, and it did go smoothly for all of a second before one instance quit; I had put one deviation in both of the test collections because in the favourites gallery I actually want to download I have a fair few deviations in multiple collections and so I wanted to see how things would handle it. Specifically, it's the most recent image in the favourites and so first up in both collections. One stopped and threw up a deviantart: Unable to download data: PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: '/tmp/.download/deviantart_[deviationid]_[deviationtitle].jpg.part', while the other went through everything after just fine.

I don't know how often this occurence would come up while going through the much larger favourites gallery I'm actually trying to download, because the number of deviations in both categories is in the minority and the number of deviations in between instances of the same deviation being shared between both would vary between collections, meaning many of the times the same deviation would be downloaded twice wouldn't be simultaneous and thus wouldn't have this same problem, but I can't help but imagine it's far from impossible for it to happen somewhere in the middle of my ~6000-total deviations download and stop one side dead in its tracks.

Aaaand.... after typing all that out I just remembered I already discovered gallery-dl totally has the capability to pick up where it left off, so it can just start again if that happens, and this whole point is nearly moot. I once again think I'll be fine multiple instances of gallery-dl each focusing on different collections.

[mikf]

Hmm, so there is a way to get it to call on curl to do it? I guess I'll have to look into curl after I wake up then. I'll try to figure out a functioning exec postprocessor after installing curl and figuring it out and then looking around the documentation here for exec, and if I can't manage it, I'll ask here for an example bit of code.

    "postprocessors": [
        {
            "name": "exec",
            "command": [
                "curl",
                "-o", "{_directory}{title}-{index}.htm",
                "https://www.deviantart.com/{author[username]}/art/{index}"
            ]
        }
    ]

One stopped and threw up a deviantart: Unable to download data: PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: '/tmp/.download/deviantart_[deviationid]_[deviationtitle].jpg.part', while the other went through everything after just fine.

Yeah, having multiple gallery-dl processes trying to download the same file at the same time doesn't work. I'd recommend splitting the work into separate parts before starting to download, like putting different favorite folder URLs into separate plain-text files and loading them with --input-file, or by using --chapter-range with https://www.deviantart.com/USER/favourites as URL.

[Jeddostotle7]

    "postprocessors": [
        {
            "name": "exec",
            "command": [
                "curl",
                "-o", "{_directory}{title}-{index}.htm",
                "https://www.deviantart.com/{author[username]}/art/{index}"
            ]
        }
    ]

Hmm, ok I've installed curl and added this to the config under the deviantart extractor's postprocessors, so in context it now looks like:

     "postprocessors": [
         {
             "name": "metadata",
             "mode": "custom",
             "directory"       : "Descriptions",
             "content-format"  : "{description}\n",
             "extension-format": "descr.txt"
         },
         {
             "name": "exec",
             "command": [
                 "C:\\dir\\ec\\to\\ry\\curl\\curl-7.86.0_2-win64-mingw\\bin\\curl.exe",
                 "-o", "{_directory}{title}-{index}.htm",
                 "https://www.deviantart.com/{author[username]}/art/{index}"
             ]
         }
     ]

So I ran it to test, and at first it threw up a red postprocessor: 'exec' initialization failed: KeyError: 'command', but then continued to work almost just fine and started downloading the images, metadata .jsons, description .txts, and the .htm files, except for one deviation, for which it threw up postprocessor.exec: '['C:\\dir\\ec\\to\\ry\\curl\\curl-7.86.0_2-win64-mingw\\bin\\curl.exe', '-o', '\\\\?\\C:\\dir\\ec\\to\\ry\\gallery-dl\\deviantart\\(favourites'ownerusername)\\Favourites\\korblborp\\MJOLNIR Gen 2/ARC-354079667.htm', 'https://www.deviantart.com/korblborp/art/354079667']' returned with non-zero exit status (23), and downloaded the image, metadata .json, description .txt, but not the .htm file, before moving onto the rest of the works and doing all four files/filetypes for each just fine. I think it might have to do with the / in the title of the work?

Also, what should I add to this exec postprocessor to get it to put the .htm files into their own separate directory like the metadata postprocessor does for description .txts? I tried this:

{
            "name": "exec",
            "command": [
                "C:\\dir\\ec\\to\\ry\\curl\\curl-7.86.0_2-win64-mingw\\bin\\curl.exe",
                "-o", "{_directory}{title}-{index}.htm",
                "https://www.deviantart.com/{author[username]}/art/{index}"
            ],
            "directory"  : "HTML files"
        }

but it just dropped the .htms right alongside the images and .jsons.

Yeah, having multiple gallery-dl processes trying to download the same file at the same time doesn't work. I'd recommend splitting the work into separate parts before starting to download, like putting different favorite folder URLs into separate plain-text files and loading them with --input-file, or by using --chapter-range with https://www.deviantart.com/USER/favourites as URL.

That's probably a good idea, yeah. Thanks again.