Refactor and parallelize integration tests

[2opremio]

What

Parallelize and refactor integration tests.

Why

Tests look way cleaner and run 10x faster (at least in my local machine)

Followup to #207

Known limitations

The dynamic port allocation makes the code a little less readable

[2opremio]

@tamirms the PR is pretty big, but I have created self-container commits which should be reviewable incrementally.

[2opremio]

@tamirms It's still a draft until we check how it performs and I can address a couple of TODOs.

[2opremio]

It turns out that the tests are around 3 times faster in CI (I will try to play with the parallelism level to see if it improves)

[2opremio]

I get spurious port clashes:

Error response from daemon: driver failed programming external connectivity on endpoint testgetnetworksucceeds-core-1 (b49f38b68105bcc50def3377640097ea93dac08deb115d612624fd4896bace26): Error starting userland proxy: listen tcp4 0.0.0.0:35855: bind: address already in use

Probably the issue is that you cannot really reserve ports with 100% accuracy (the only way to make sure you reserve them is by using them right away)

[tamirms]

@2opremio you can use this approach to let docker-compose select open ports on the host for the exposed ports on the containers:

https://stackoverflow.com/questions/60109556/expose-random-port-to-docker-compose-yml

there is a docker-compose command you can use to find the port mapping between host and container

[2opremio]

Ahh, interesting. I will look into that. That should reduce the clashes (or even completely remove them)

[2opremio]

@tamirms I applied your suggestion. The only preallocated port is Captive Core's peer port, because there doesn't seem to be a way to make Core allocate it dynamically.

We can even get rpc to use dynamic ports! :)

[2opremio]

Now the problem is I cannot get an ordered output of the failing tests (since tests are running in parallel, golang outputs a mixed output) which makes it hard to see what went wrong.

[2opremio]

The logging should be fixed. This blogpost was pretty useful: https://brandur.org/t-parallel

The trick was to use testing.T.Log() for everything

[2opremio]

Tests are now failing some times because of stellar/go#5350

[2opremio]

I fixed the race at stellar/go#5352

[Shaptic]

This PR is a lot to take in so my review isn't super comprehensive but I do have a main question: can you control "how parallel" the tests are or does it run them all at once? Gonna try to run them locally and I'm worried it won't work on older machines.

Edit: ah, your linked post explains you can use -count=N for it.

[2opremio]

@Shaptic PTAL

[2opremio]

This PR is a lot to take in so my review isn't super comprehensive but I do have a main question: can you control "how parallel" the tests are or does it run them all at once? Gonna try to run them locally and I'm worried it won't work on older machines.

Edit: ah, your linked post explains you can use -count=N for it.

It’s -parallel, for which Go chooses a reasonable default. See https://pkg.go.dev/cmd/go/internal/test

By default, -parallel is set to the value of GOMAXPROCS.