WebCrawler: use disk to store status if configured #2312

Job	Run time
CLI End to End tests	14m 38s
End to End tests (Python, python)	20m 3s
End to End tests (Other, other)	17m 32s
Unit tests (Runtime)	6m 9s
Unit tests (Runtime IT)	17m 27s
Unit tests (Deployer)	8m 18s
Unit tests (Api Gateway)	7m 0s
Unit tests (Control plane)	3m 51s
Unit tests (Other)	10m 51s
Unit tests (Style)	3m 55s
	1h 49m 44s

Provide feedback