Add advanced usage docs

Co-authored-by: wvengen <[email protected]>

docs/advanced_usage.md

@@ -2,11 +2,31 @@
 
 ## Crawling
 
+### Skipping specific requests
+
+The `wacz_crawl_skip` flag is applied to requests that should be ignored by the crawler. When this flag is present, the middleware intercepts the request and prevents it from being processed further, skipping both download and parsing. This is useful in scenarios where the request should not be collected during a scraping session. Usage:
+
+```python
+yield Request(url, callback=cb_func, flags=["wacz_crawl_skip"])
+```
+
+When this happens, the statistic `webarchive/crawl_skip` is increased.
+
+### Disallowing archived URLs
+
+If the spider has the attribute `archive_disallow_regexp`, all requests returned from the spider that match this regular expression, are ignored. For example, when a product page was returned in `start_requests`, but the product page disappeared and redirected to its category page, the category page can be disallowed, so as to avoid crawling the whole category, which would take much more time and could lead to unknown URLs (e.g. the spider's requested pagination size could be different from the website default).
+
+When this happens, the statistic `wacz/crawl_skip/disallowed` is increased.
+
 ### Iterating a WACZ archive index
 
+When using a WACZ file that is not generated by your own spiders, it might be that the spider for crawling is not in place. In order to crawl this WACZ you need to tailor a spider to work with this specific WACZ file. This will require building the spider different to what it is supposed to look like with a live resource.
+
 Going around the default behaviour of the spider, the `WaczCrawlMiddleware` spider middleware will, when enabled, replace the crawl by an iteration through all the entries in the WACZ archive index.
 
-To use this strategy, enable both middlewares in the spider settings like so:
+#### Configuration
+
+To use this strategy, enable both the spider- and the downloadermiddleware in the spider settings like so:
 
 ```python
 DOWNLOADER_MIDDLEWARES = {
@@ -24,3 +44,33 @@ Then define the location of the WACZ archive with `SW_WACZ_SOURCE_URI` setting:
 SW_WACZ_SOURCE_URI = "s3://scrapy-webarchive/archive.wacz"
 SW_WACZ_CRAWL = True
 ```
+
+#### Controlling the crawl
+
+Not all URLs will be interesting for the crawl since your WACZ will most likely contain static files such as fonts, JavaScript (website and external), stylesheets, etc. In order to improve the performance of the spider by not reading all the irrelevant request/response entries, you can configure the following atrribute in your spider, `archive_regex`:
+
+```python
+class MyWaczSpider(Spider):
+    name = "myspider"
+    archive_regex = r"^/tag/[\w-]+/$"
+```
+
+If the spider has an `archive_regexp` attribute, only response URLs matching this regexp are presented in `start_requests`. To visualise that, the spider above will only crawl the indented cdxj records below:
+
+```
+com,toscrape,quotes)/favicon.ico 20241007081411465 {...}
+com,gstatic,fonts)/s/raleway/v34/1ptug8zys_skggpnyc0it4ttdfa.woff2 {...}
+com,googleapis,fonts)/css?family=raleway%3A400%2C700 20241007081525229 {...}
+com,toscrape,quotes)/static/bootstrap.min.css 20241007081525202 {...}
+com,toscrape,quotes)/static/main.css 20241007081525074 {...}
+> com,toscrape,quotes)/tag/books/ 20241007081513898 {...}
+> com,toscrape,quotes)/tag/friends/ 20241007081520928 {...}
+> com,toscrape,quotes)/tag/friendship/ 20241007081519648 {...}
+> com,toscrape,quotes)/tag/humor/ 20241007081512594 {...}
+> com,toscrape,quotes)/tag/inspirational/ 20241007081506990 {...}
+> com,toscrape,quotes)/tag/life/ 20241007081510349 {...}
+> com,toscrape,quotes)/tag/love/ 20241007081503814 {...}
+> com,toscrape,quotes)/tag/reading/ 20241007081516781 {...}
+> com,toscrape,quotes)/tag/simile/ 20241007081524944 {...}
+> com,toscrape,quotes)/tag/truth/ 20241007081523804 {...}
+```