Web scrape "automatic refresh" option

[KastanDay]

Start implementing the ‘re-scrape’ feature.

Note: We want to fully crawl the website again, not just refresh the content of the pages we already managed to crawl. Imagine a news website with new pages all the time, users will expect us to capture these brand new pages (not just update the old pages we already scraped).

Functions

1. Delete scrape results from {base_url} on {course_name}. See the existing delete functions for how this should work.
2. Add a boolean flag for “only scrape this website vs include other websites that are linked” where website == base_url.

[KastanDay]

Maybe use this (on front-end side...): https://upstash.com/blog/qstash-periodic-data-updates

or Github Actions for a cron job, equally good.
Github Actions will NOT work for a cron job because we need to create a SEPARATE new cron job for every single website ingested. So we do need Upstash to easily create many cron jobs programmatically, and via GUI.
OR
We just have ONE cron job that refreshes all websites at the same time every day. Manual refresh also available to end users on website GUI button click.