e2e test for pod completion and next pod start

[tardieu]

We test that we do not ungate more pods than we can fit on available gpus by launching 8 long-running 1g pods and checking exactly 7 are running (in a single gpu setup). We should extend such a test to:

1. confirm that when one of the running pods completes, the pending pod starts running;
2. verify the transition latency, i.e., that the pending pod starts running without delay.

[asm582]

Thanks, this test case on KinD does what we ask in 1st sub-bullet point across two GPUs; one pod remains in scheduling gated:

instaslice-operator/test/e2e/e2e_test.go

Line 692 in ce1f522

It("should verify all 1g profiles of GPUs are consumed", func() {

[tardieu]

AFAIK this test only addresses point 0, i.e., one pod remains gated, not point 1, i.e., the gated pod eventually runs.