Continuous scraping

[autaut03]

As of right now, scrapers are basically a lookup DB. They return scene/performer details based on an existing identifier - a name or a URL. The problem with that is you have to have at least some kind of set of performers/scenes/photos before you can scrape any more data.

What would be great to have is the ability for a scraper to specify actions that are regularly ran: one that returns a list of perfomers, another one returning scenes, another one for photos. They would be defined as follows:

Scrape type	Input	Output
performers	{"lastRan": "2000-10-31T01:30:00.000Z"}	Array of JSON-encoded performer fragments
scenes	{"lastRan": "2000-10-31T01:30:00.000Z"}	Array of JSON-encoded scene fragments
photos	{"lastRan": "2000-10-31T01:30:00.000Z"}	Array of JSON-encoded photo fragments

All of those actions would be required to only return new items that have been added or changed since the last time the scraper was ran. The interval at which those are ran should be a configuration option.

[SmallCoccinelle]

In the spirit of clarity: what's an immediate use case? Having that background information is going to be valuable for a discussion.

The main pain-point of aggressive scraping is that it can be quite taxing on web sites if many people do them, especially in a regular period, which has to be factored into a continuous scraping solution.

[philpw99]

I did scrape all my performers by hand. I just set a filter "is missing: gender" then bookmark that link. Any time I click on that link, the unprocessed performers will pop up. Saved me quite a lot of time in looking for unscraped performers.
The big problem of auto-scraping is that the quality of result has no guarantee. When scrape a performer by name, there might be 10 entities show up, and the 2nd or the 3rd one is the one you are actually looking for. Without human inspection, scraping will produce quite a lot of errors.

[autaut03]

Well.. there are two kind of porn websites, and there's a use case for both:

1. the regular pornhub-like type that display a list of scenes with no particular performer. Let's say I have an account on a couple of different websites of this kind and they all have a feed of recommendations based on my likings. Then it would be great to have those merged into one feed (tag) on my local stashapp, so that: I can properly tag those scenes; I can use ratings; I can find/scrape more scenes of those performers on other websites; all of the scenes are in one place, one format.
2. "studio" kind of websites that display a list of performers (a good example is xuk.net). There are two problems with that: first, if you have multiple of those, you have the same problem as with first type of websites; second, the UX of those websites is sometimes just awful. For example, when you click on a performer on xuk.net on a mobile device, it opens an image slider. A model usually has 30-50 photos, so clicking "next" 30 times just to get down to the last photo is not ideal. Again, having those performers on a standartized (and possibly customized) stashapp instance is much better.

[philpw99]

I think you have the misconception about what scraper actually is. Scrapers are not plug-ins or running tasks. They are the ones which retrieve the information for you, and ask you for the judgement: is this info OK, or not? It saves you lots of time because you just need to provide a name or URL, and it will do the rest, but still you need to be present to approve the info.

Otherwise, you just write an AutoIt script in windows, or python script in other platforms for automation.
So the two cases:

1. You need to write the script yourself to automate the whole process. Scrapers can help you by giving you all the XPath info, but still you need you own script to make the feeds merge together in a uniform way.
2. Scrapers are not supposed to get all 50 images for you either. They will get you the details, performers, and studio..etc, but not actually retrieve the whole content. It's not a download manager.

[autaut03]

@philpw99 Call it "crawling" if you wish, but technically, nothing in the term "scraping" specifies what happens with scraped information. Regardless of it's name, the functionality I'm describing has it's use cases, and I don't believe that extracting it outside of stash is a good idea.

Each of those scrapers (crawlers) has the same logic: every once in a while check for new content, then find existing and update or create new records of data on stash. Obviously, if some logic is used more than once, it's a great candidate for standartization.

But let's say we solve this by extracting all of that logic into an external library. There are even more problems with that:

1. stash's API has to be very stable for the lib not to break every month. I doubt that's the case with a beta product
2. it's either gonna be a one language library or a whole service with JSON IO, which increases complexity even more
3. this is much worse for the end user because of the increased complexity of the whole thing. Installing and properly configuring such crawler is orders of magnitudes harder than dropping a .yml file into configs folder

I'm not saying it's impossible to implement as of right now. Stash has a good GraphQL API so it's not a question of whether it can be implemented, but how hard it is.

[7dJx1qP]

You describe the output of continuous scraping as arrays of JSON encoded fragments, but then what does stash do with that output?

It seems like your idea for continuous scraping is based on the idea of stash being able to aggregate and store metadata for content that you haven't downloaded. However, the way stash is now, that kind of scraping for metadata can only apply to performers. Every scene and image in stash is still tied to a file.

I think there's plans to decouple scenes and images from files, but that needs to be done first before something like continuous scraping can be considered. Even the idea of just scraping everything from a single site and not just a specific scene would need to be considered first before the idea of continuous scraping.

[autaut03]

Well, the idea was that the content should be downloaded too, regardless of whether that happens before creating a row or after. But I got your point.