WebCrawler: use disk to store status if configured #2307

Job	Run time
CLI End to End tests	3m 3s
End to End tests (Python, python)	2m 58s
Unit tests (Runtime)	2m 55s
End to End tests (Other, other)	2m 58s
Unit tests (Runtime IT)	2m 56s
Unit tests (Deployer)	3m 4s
Unit tests (Api Gateway)	2m 59s
Unit tests (Control plane)	2m 57s
Unit tests (Other)	2m 55s
Unit tests (Style)	2m 55s
	29m 40s

Provide feedback