Adapt CaseSearchProfiler for more general use #16577

Job	Run time
tests (python, 05)	17m 58s
tests (python, 6a)	15m 41s
tests (python, bf)	14m 46s
tests (python-sharded-and-javascript)	16m 31s
	1h 4m 56s

Provide feedback