[TT-590] logwatch with multiple targets and retries

.gitignore

@@ -56,6 +56,9 @@ dist/
 **/remote_runner_config.yaml
 logs/
 
+.test_summary/
+.run.id
+
 env/cmd/chaos
 env/bin/
 # temp manifest for deployment and validation

Makefile

@@ -73,7 +73,7 @@ compile_contracts:
 	python3 ./utils/compile_contracts.py
 
 test_unit: install_gotestfmt
-	go test -json -cover -covermode=count -coverprofile=unit-test-coverage.out ./client ./gauntlet ./testreporters ./docker/test_env ./k8s/config 2>&1 | tee /tmp/gotest.log | gotestfmt
+	go test -json -cover -covermode=count -coverprofile=unit-test-coverage.out ./client ./gauntlet ./testreporters ./docker/test_env ./k8s/config ./logstream 2>&1 | tee /tmp/gotest.log | gotestfmt
 
 
 #######################

README.md

@@ -93,3 +93,79 @@ Modified packages or new packages get added and pushed to the `gh-pages` branch
 Removed charts do not trigger a re-publish, the packages have to be removed and the index file regenerated in the `gh-pages` branch of the [qa-charts](https://github.com/smartcontractkit/qa-charts) repository.
 
 Note: The qa-charts repository is scheduled to look for changes to the charts once every hour. This can be expedited by going to that repo and running the cd action via github UI.
+
+# Using LogStream
+
+LogStream is a package that allows to connect to a Docker container and then flush logs to configured targets. Currently 3 targets are supported:
+* `file` - saves logs to a file in `./logs` folder
+* `loki` - sends logs to Loki
+* `in-memory` - stores logs in memory
+
+It can be configured to use multiple targets at once. If no target is specified, it becomes a no-op.
+
+Targets can be set in two ways:
+* using `LOGSTREAM_LOG_TARGETS` environment variable, e.g. `Loki,in-MemOry` (case insensitive)
+* using programmatic functional option `WithLogTarget()`
+
+Functional option has higher priority than environment variable.
+
+When you connect a contaier LogStream will create a new consumer and start a detached goroutine that listens to logs emitted by that container and which reconnects and re-requests logs if listening fails for whatever reason. Retry limit and timeout can both be configured using functional options. In most cases one container should have one consumer, but it's possible to have multiple consumers for one container.
+
+LogStream stores all logs in gob temporary file. To actually send/save them, you need to flush them. When you do it, LogStream will decode the file and send logs to configured targets. If log handling results in an error it won't be retried and processing of logs for given consumer will stop (if you think we should add a retry mechanism please let us know).
+
+*Important:* Flushing and accepting logs is blocking operation. That's because they both share the same cursor to temporary file and otherwise it's position would be racey and could result in mixed up logs.
+
+When using `in-memory` or `file` target no other environment variables are required. When using `loki` target, following environment variables are required:
+* `LOKI_TENTANT_ID` - tenant ID
+* `LOKI_URL` - Loki URL to which logs will be pushed
+* `LOKI_BASIC_AUTH` -- only needed when running in CI and using public endpoint
+
+Also, do remember that different `LOKI_URL` should be used when running in CI and everywhere else. In CI it should be a public endpoint, while in local environment it should be a private one.
+
+You can print log location for each target using this function: `(m *LogStream) PrintLogTargetsLocations()`. For `file` target it will print relative folder path, for `loki` it will print URL of a Grafana Dashboard scoped to current execution and container ids. For `in-memory` target it's no-op.
+
+It is recommended to shutdown LogStream at the end of your tests. Here's an example:
+```go
+
+t.Cleanup(func() {
+    l.Warn().Msg("Shutting down Log Stream")
+
+    if t.Failed() || os.Getenv("TEST_LOG_COLLECT") == "true" {
+        // we can't do much if this fails, so we just log the error
+        _ = logStream.FlushLogsToTargets()
+        logStream.PrintLogTargetsLocations()
+        logStream.SaveLogLocationInTestSummary()
+    }
+
+    // we can't do much if this fails, so we just log the error
+    _ = logStream.Shutdown(testcontext.Get(b.t))
+    })
+```
+
+## Grouping test execution
+
+When running tests in CI you're probably interested in grouping logs by test execution, so that you can easily find the logs in Loki. To do that your job should set `RUN_ID` environment variable. In GHA it's recommended to set it to workflow id. If that variable is not set, then a run id will be automatically generated and saved in `.run.id` file, so that it can be shared by tests that are part of the same execution, but are running in different processes.
+
+## Test Summary
+
+In order to facilitate displaying information in GH's step summary `testsummary` package was added. It exposes a single function `AddEntry(testName, key string, value interface{}) `. When you call it, it either creates a test summary JSON file or appends to it. The result is is a map of keys with values.
+
+Example:
+```JSON
+{
+   "file":[
+      {
+         "test_name":"TestOCRv2Basic",
+         "value":"./logs/TestOCRv2Basic-2023-12-01T18-00-59-TestOCRv2Basic-38ac1e52-d0a6-48"
+      }
+   ],
+   "loki":[
+      {
+         "test_name":"TestOCRv2Basic",
+         "value":"https://grafana.ops.prod.cldev.sh/d/ddf75041-1e39-42af-aa46-361fe4c36e9e/ci-e2e-tests-logs?orgId=1\u0026var-run_id=TestOCRv2Basic-38ac1e52-d0a6-48\u0026var-container_id=cl-node-a179ca7d\u0026var-container_id=cl-node-76798f87\u0026var-container_id=cl-node-9ff7c3ae\u0026var-container_id=cl-node-43409b09\u0026var-container_id=cl-node-3b6810bd\u0026var-container_id=cl-node-69fed256\u0026from=1701449851165\u0026to=1701450124925"
+      }
+   ]
+}
+```
+
+In GHA after tests have ended we can use tools like `jq` to extract the information we need and display it in step summary.

docker/test_env/env_component_test.go

@@ -27,7 +27,7 @@ func followLogs(t *testing.T, c testcontainers.Container) *TestLogConsumer {
 	}
 	go func() {
 		c.FollowOutput(consumer)
-		err := c.StartLogProducer(testcontext.Get(t))
+		err := c.StartLogProducer(testcontext.Get(t), time.Duration(5*time.Second))
 		require.NoError(t, err)
 	}()
 	return consumer

go.mod

@@ -5,7 +5,6 @@ go 1.21
 require (
 	github.com/avast/retry-go v3.0.0+incompatible
 	github.com/chaos-mesh/chaos-mesh/api/v1alpha1 v0.0.0-20220226050744-799408773657
-	github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc
 	github.com/docker/go-connections v0.4.0
 	github.com/ethereum/go-ethereum v1.12.0
 	github.com/go-resty/resty/v2 v2.7.0
@@ -20,7 +19,7 @@ require (
 	github.com/prometheus/common v0.44.0
 	github.com/rs/zerolog v1.30.0
 	github.com/slack-go/slack v0.12.2
-	github.com/smartcontractkit/wasp v0.3.6
+	github.com/smartcontractkit/wasp v0.3.7
 	github.com/stretchr/testify v1.8.4
 	github.com/testcontainers/testcontainers-go v0.23.0
 	go.uber.org/atomic v1.11.0
@@ -29,25 +28,36 @@ require (
 )
 
 require (
+	github.com/Microsoft/hcsshim v0.11.1 // indirect
 	github.com/allegro/bigcache v1.2.1 // indirect
 	github.com/c9s/goprocinfo v0.0.0-20210130143923-c95fcf8c64a8 // indirect
 	github.com/cespare/cp v1.1.1 // indirect
+	github.com/containerd/log v0.1.0 // indirect
+	github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
 	github.com/gballet/go-libpcsclite v0.0.0-20191108122812-4678299bea08 // indirect
 	github.com/google/pprof v0.0.0-20230705174524-200ffdc848b8 // indirect
 	github.com/hashicorp/go-version v1.6.0 // indirect
+	github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect
 	github.com/mattn/go-sqlite3 v2.0.3+incompatible // indirect
 	github.com/pbnjay/memory v0.0.0-20210728143218-7b4eea64cf58 // indirect
+	github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c // indirect
 	github.com/pyroscope-io/client v0.7.1 // indirect
 	github.com/rs/cors v1.8.3 // indirect
 	github.com/sercand/kuberesolver/v4 v4.0.0 // indirect
 	github.com/sergi/go-diff v1.2.0 // indirect
+	github.com/shirou/gopsutil/v3 v3.23.9 // indirect
+	github.com/shoenig/go-m1cpu v0.1.6 // indirect
 	github.com/yusufpapurcu/wmi v1.2.3 // indirect
 )
 
+// avoids ambigious imports of indirect dependencies
+exclude github.com/hashicorp/consul v1.2.1
+
 // K8s versions are infuriatingly inconsistent, so we pin them here.
 replace (
 	// replicating the replace directive on cosmos SDK
 	github.com/gogo/protobuf => github.com/regen-network/protobuf v1.3.3-alpha.regen.1
+	github.com/testcontainers/testcontainers-go => github.com/Tofel/testcontainers-go v0.0.0-20231130110817-e6fbf9498b56
 
 	k8s.io/api => k8s.io/api v0.25.11
 	k8s.io/apimachinery => k8s.io/apimachinery v0.25.11
@@ -88,15 +98,15 @@ require (
 	github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b // indirect
 	github.com/cockroachdb/pebble v0.0.0-20230209160836-829675f94811 // indirect
 	github.com/cockroachdb/redact v1.1.3 // indirect
-	github.com/containerd/containerd v1.7.3 // indirect
+	github.com/containerd/containerd v1.7.7 // indirect
 	github.com/coreos/go-semver v0.3.0 // indirect
 	github.com/coreos/go-systemd/v22 v22.5.0 // indirect
 	github.com/cpuguy83/dockercfg v0.3.1 // indirect
 	github.com/deckarep/golang-set/v2 v2.3.0 // indirect
 	github.com/decred/dcrd/dcrec/secp256k1/v4 v4.2.0 // indirect
 	github.com/dennwc/varint v1.0.0 // indirect
 	github.com/docker/distribution v2.8.2+incompatible // indirect
-	github.com/docker/docker v24.0.5+incompatible // indirect
+	github.com/docker/docker v24.0.6+incompatible // indirect
 	github.com/docker/go-units v0.5.0 // indirect
 	github.com/dustin/go-humanize v1.0.1 // indirect
 	github.com/edsrzf/mmap-go v1.1.0 // indirect
@@ -190,7 +200,7 @@ require (
 	github.com/mitchellh/go-homedir v1.1.0 // indirect
 	github.com/mitchellh/go-wordwrap v1.0.1 // indirect
 	github.com/mitchellh/mapstructure v1.5.0 // indirect
-	github.com/moby/patternmatcher v0.5.0 // indirect
+	github.com/moby/patternmatcher v0.6.0 // indirect
 	github.com/moby/spdystream v0.2.0 // indirect
 	github.com/moby/sys/sequential v0.5.0 // indirect
 	github.com/moby/term v0.5.0 // indirect
@@ -203,7 +213,7 @@ require (
 	github.com/oklog/ulid v1.3.1 // indirect
 	github.com/olekukonko/tablewriter v0.0.5 // indirect
 	github.com/opencontainers/go-digest v1.0.0 // indirect
-	github.com/opencontainers/image-spec v1.1.0-rc4 // indirect
+	github.com/opencontainers/image-spec v1.1.0-rc5 // indirect
 	github.com/opencontainers/runc v1.1.7 // indirect
 	github.com/opentracing-contrib/go-grpc v0.0.0-20210225150812-73cb765af46e // indirect
 	github.com/opentracing-contrib/go-stdlib v1.0.0 // indirect
@@ -225,7 +235,7 @@ require (
 	github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529 // indirect
 	github.com/shirou/gopsutil v3.21.11+incompatible // indirect
 	github.com/sirupsen/logrus v1.9.3 // indirect
-	github.com/spf13/cobra v1.6.1 // indirect
+	github.com/spf13/cobra v1.6.1
 	github.com/spf13/pflag v1.0.5 // indirect
 	github.com/stretchr/objx v0.5.0 // indirect
 	github.com/syndtr/goleveldb v1.0.1-0.20220721030215-126854af5e6d // indirect