ingest/ledgerbackend: getLogLineWriter() leaks goroutines and pipes

[2opremio]

getLogLineWriter() creates a pipe and a goroutine, but the goroutine sometimes doesn't exit (and the pipe is never released).

Every time captive core is restarted this causes a leak.

The easiest fix is probably returning a CloseWriter and making sure that the writer is closed (releasing the pipe and the goroutine).

[2opremio]

Discovered while investigating failing tests at stellar/stellar-rpc#207

[2opremio]

It turns out the goroutines are closed in many places but something must be amiss, since they are still left behind in some cases. All the (*pipe).read goroutines in this dump are from Closed() captive core backends (from different soroban-rpc tests):

[2opremio]

I think stellarCoreRunner.handleExit() may not be closing the pipes in some cases

[2opremio]

Here's another dump:

[tamirms]

It looks like there are a few createCmd calls where we do not close the log line writers:

go/ingest/ledgerbackend/stellar_core_runner.go

Line 311 in a4e5a3f

cmd, err := r.createCmd("new-db")

go/ingest/ledgerbackend/stellar_core_runner.go

Lines 400 to 404 in a4e5a3f

	if from > 2 {
	cmd, err = r.createCmd("catchup", fmt.Sprintf("%d/0", from-1))
	} else {
	cmd, err = r.createCmd("catchup", "2/0")
	}

go/ingest/ledgerbackend/stellar_core_runner.go

Line 311 in a4e5a3f

cmd, err := r.createCmd("new-db")