Unit tests are too slow

[julienh-ssv]

Hey team,

Issue

I've noticed that our unit tests are running a bit slower than we'd ideally want. Since fast unit tests are key for quick feedback and smooth development cycles—as Uncle Bob and Martin Fowler often highlight—a slow suite can really slow us down. Improving the speed of these tests will not only help catch issues faster but also boost our overall productivity and make our day-to-day coding more efficient.

I'm happy to brainstorm and work on some optimisations with anyone interested.

Culprits (non-exhaustive list)

1. At first glance it seems that the use of time.After and time.Sleep really slows the tests down and leads to inconsistent/flaky tests. For instance this recent issue Flaky TestExecQueue_Stop #2013.
2. Possible misuse of the for-select idiom: for-select allow to block until channel produces value, but adding a default case with time.Sleep defeats its purpose by introducing busy waiting instead, which is a very different pattern: L142

   		for {
   			select {
   			case <-ctx.Done():
   				return
   			case <-stopChan:
   				return
   			default:
   				time.Sleep(100 * time.Millisecond)
   			}
   		}
   	}()```

3. Possible mix of integration tests and unit-tests: unit-tests should not depend on external dependencies like topics, services or servers.

Target execution time

A good first target would be 5s to 10s total run time for all unit-tests.

Current execution time

So far this is what I have on local:

make unit-test

Package	Duration (s)	Coverage	Result
github.com/ssvlabs/ssv/api	1.847	21.8%	Success
github.com/ssvlabs/ssv/api/handlers	1.426	9.1%	Success
github.com/ssvlabs/ssv/beacon/goclient	9.269	20.9%	Success
github.com/ssvlabs/ssv/cli/operator	1.827	2.5%	Success
github.com/ssvlabs/ssv/ekm	2.123	74.7%	Success
github.com/ssvlabs/ssv/eth/ethtest	9.801	[no statements]	Success
github.com/ssvlabs/ssv/eth/eventhandler	5.279	71.1%	Success
github.com/ssvlabs/ssv/eth/eventparser	1.403	73.3%	Success
github.com/ssvlabs/ssv/eth/eventsyncer	3.483	57.0%	Success
github.com/ssvlabs/ssv/eth/executionclient	2.643	66.1%	Success
github.com/ssvlabs/ssv/eth/localevents	1.387	35.8%	Success
github.com/ssvlabs/ssv/exporter/api	7.076	69.7%	Success
github.com/ssvlabs/ssv/ibft/storage	3.675	71.2%	Success
github.com/ssvlabs/ssv/identity	1.592	81.5%	Success
github.com/ssvlabs/ssv/logging	1.384	54.0%	Success
github.com/ssvlabs/ssv/message/validation	1.861	79.3%	Success
github.com/ssvlabs/ssv/migrations	1.445	10.4%	Success
github.com/ssvlabs/ssv/network/commons	1.493	26.6%	Success
github.com/ssvlabs/ssv/network/discovery	2.196	61.1%	Success
github.com/ssvlabs/ssv/network/p2p	114.939	50.7%	Success
github.com/ssvlabs/ssv/network/peers	5.136	52.4%	Success
github.com/ssvlabs/ssv/network/peers/connections	1.741	22.1%	Success
github.com/ssvlabs/ssv/network/records	1.514	61.8%	Success
github.com/ssvlabs/ssv/network/streams	7.570	80.0%	Success
github.com/ssvlabs/ssv/network/topics	1200.584	9.3%	Fail
github.com/ssvlabs/ssv/network/topics/params	2.074	81.5%	Success
github.com/ssvlabs/ssv/nodeprobe	1.430	65.2%	Success
github.com/ssvlabs/ssv/observability	1.387	11.4%	Success
github.com/ssvlabs/ssv/operator/datastore	1.414	96.6%	Success
github.com/ssvlabs/ssv/operator/duties	28.788	86.2%	Success
github.com/ssvlabs/ssv/operator/fee_recipient	4.583	92.2%	Success
github.com/ssvlabs/ssv/operator/keys	2.536	75.6%	Success
github.com/ssvlabs/ssv/operator/keystore	2.113	84.2%	Success
github.com/ssvlabs/ssv/operator/slotticker	11.079	100.0%	Success
github.com/ssvlabs/ssv/operator/storage	1.565	79.7%	Success
github.com/ssvlabs/ssv/operator/validator	1.868	50.1%	Success
github.com/ssvlabs/ssv/operator/validator/metadata	1.634	92.2%	Success
github.com/ssvlabs/ssv/protocol/v2/blockchain/beacon	1.396	3.7%	Success
github.com/ssvlabs/ssv/protocol/v2/message	2.458	9.5%	Success
github.com/ssvlabs/ssv/protocol/v2/qbft/controller	1.717	27.8%	Success
github.com/ssvlabs/ssv/protocol/v2/qbft/instance	1.755	4.2%	Success
github.com/ssvlabs/ssv/protocol/v2/qbft/roundtimer	5.019	72.1%	Success
github.com/ssvlabs/ssv/protocol/v2/queue/worker	7.668	57.1%	Success
github.com/ssvlabs/ssv/protocol/v2/ssv/queue	1.933	69.1%	Success
github.com/ssvlabs/ssv/protocol/v2/ssv/validator	1.688	3.6%	Success
github.com/ssvlabs/ssv/protocol/v2/types	1.532	20.0%	Success
github.com/ssvlabs/ssv/registry/storage	4.829	81.7%	Success
github.com/ssvlabs/ssv/storage/kv	6.570	53.7%	Success
github.com/ssvlabs/ssv/utils/async	1.437	100.0%	Success
github.com/ssvlabs/ssv/utils/blskeygen	1.395	100.0%	Success
github.com/ssvlabs/ssv/utils/format	1.394	70.8%	Success
github.com/ssvlabs/ssv/utils/hashmap	5.425	100.0%	Success
github.com/ssvlabs/ssv/utils/rsaencryption	2.552	56.8%	Success
github.com/ssvlabs/ssv/utils/tasks	4.137	82.2%	Success
github.com/ssvlabs/ssv/utils/threshold	1.399	87.5%	Success

Total run time: 1507.47 seconds (approximately 25 minutes and 7 seconds)