[8.11](backport #3765) Fix index pattern when querying ES and conditi…

…on when searching logs (#3829)

Two issues were causing flakiness on integration tests and are fixed by this PR:
1. The pattern used to query ES was not working on serverless, this commit updates it to a pattern that works on both stateful and serverless as well as make it more specific to the indexes/data streams we want to query
2. `findESDocs` did not wait for the data to be indexed, only to a successful query on ES. In some cases the documents the test wanted were not indexed yet, leading to 0 documents being returned and the test failing with no error in the logs/diagnostics. This is fixed by waiting for a document count > 0 and no error.

(cherry picked from commit a03aa9c)

# Conflicts:
#	pkg/testing/tools/estools/elasticsearch.go
#	testing/integration/logs_ingestion_test.go

The `add_cloud_metadata` and some other `add_*_metadata` processors
are expected to log some errors if they cannot fetch the necessary
information. It is normal to find their error logs in pretty much any
deployment, some examples:
 - When Docker is not installed/running `add_docker_metadata` will log
 some errors
 - When the Elastic-Agent is deployed in a non-cloud VM
 `add_cloud_metadata` will log some errors.

This commit removes all those processors from the queries for log
errors, as they're expected.

Add a 2h timeout for `go test` when run on remote hosts.

---------

Co-authored-by: Tiago Queiroz <[email protected]>

magefile.go

@@ -1601,7 +1601,7 @@ func (Integration) TestOnRemote(ctx context.Context) error {
 			extraFlags = append(extraFlags, goTestFlags...)
 		}
 		extraFlags = append(extraFlags, "-test.shuffle", "on",
-			"-test.timeout", "0", "-test.run", "^("+strings.Join(packageTests, "|")+")$")
+			"-test.timeout", "2h", "-test.run", "^("+strings.Join(packageTests, "|")+")$")
 		params := mage.GoTestArgs{
 			LogName:         testName,
 			OutputFile:      fileName + ".out",