Review IndexNow

[fils]

Need to review https://www.indexnow.org/faq

If we want PUSH notification, this would be the way to do it. However, this is a bit at odds with the current structured data on the web approach and diffing sitemaps is not that hard (I mean we are doing it).

So I worry that at the scale Gleaner normally works at this is a bit of a digression (at best) or regression (at worst).

Needs to be reviewed though and would be a neat side tool that could then invoke Gleaner with a "set of URLs". This bypassing of the sitemaps et al and providing a URL or set of URLs to index has been something I felt might be nice for testing purposes or for developers / publishers to have. That aspect would not be hard to implement in the Gleaner code.

Then the "indexnow" server would be separate from that and be responsible for calling Gleaner then.