drenv: Improve gather logs

We used to drop gather info logs. Watch the gather process and log the
gather logs as they are written to stderr.

Example logs with this change:

    % drenv gather envs/regional-dr.yaml
    2025-01-14 14:28:57,539 INFO    [rdr] Gathering environment
    2025-01-14 14:28:57,787 INFO    [rdr] 2025-01-14T14:28:57.786+0200	INFO	gather	Using kubeconfig "/Users/nir/.kube/config"
    2025-01-14 14:28:57,789 INFO    [rdr] 2025-01-14T14:28:57.789+0200	INFO	gather	Gathering from all namespaces
    2025-01-14 14:28:57,789 INFO    [rdr] 2025-01-14T14:28:57.789+0200	INFO	gather	Using all addons
    2025-01-14 14:28:57,789 INFO    [rdr] 2025-01-14T14:28:57.789+0200	INFO	gather	Storing data in "gather.20250114142857"
    2025-01-14 14:28:57,789 INFO    [rdr] 2025-01-14T14:28:57.789+0200	INFO	gather	Gathering from cluster "dr1"
    2025-01-14 14:28:57,789 INFO    [rdr] 2025-01-14T14:28:57.789+0200	INFO	gather	Gathering from cluster "dr2"
    2025-01-14 14:28:57,789 INFO    [rdr] 2025-01-14T14:28:57.789+0200	INFO	gather	Gathering from cluster "hub"
    2025-01-14 14:28:59,082 INFO    [rdr] 2025-01-14T14:28:59.082+0200	INFO	gather	Gathered 1551 resources from cluster "hub" in 1.293 seconds
    2025-01-14 14:29:06,687 INFO    [rdr] 2025-01-14T14:29:06.687+0200	INFO	gather	Gathered 1992 resources from cluster "dr2" in 8.899 seconds
    2025-01-14 14:29:08,989 INFO    [rdr] 2025-01-14T14:29:08.989+0200	INFO	gather	Gathered 1982 resources from cluster "dr1" in 11.200 seconds
    2025-01-14 14:29:08,989 INFO    [rdr] 2025-01-14T14:29:08.989+0200	INFO	gather	Gathered 5525 resources from 3 clusters in 11.200 seconds
    2025-01-14 14:29:08,996 INFO    [rdr] Environment gathered in 11.46 seconds

Issues:

- Gather logs include duplicate timestamps and log level names. We will
  improve this later by adding json logs to kubectl gather, similar to
  lima.

Signed-off-by: Nir Soffer <[email protected]>

test/drenv/__main__.py

@@ -285,6 +285,7 @@ def do_gather(args):
         [p["name"] for p in env["profiles"]],
         directory=args.directory,
         namespaces=args.namespaces,
+        name=env["name"],
     )
     logging.info(
         "[%s] Environment gathered in %.2f seconds",

test/drenv/kubectl.py

@@ -1,6 +1,9 @@
 # SPDX-FileCopyrightText: The RamenDR authors
 # SPDX-License-Identifier: Apache-2.0
 
+import subprocess
+import logging
+
 from . import commands
 
 JSONPATH_NEWLINE = '{"\\n"}'
@@ -188,16 +191,20 @@ def watch(
     return commands.watch(*cmd, timeout=timeout)
 
 
-def gather(contexts, namespaces=None, directory=None):
+def gather(contexts, namespaces=None, directory=None, name="gather"):
     """
-    Run kubectl gather plugin.
+    Run kubectl gather plugin, logging gather logs.
     """
     cmd = ["kubectl", "gather", "--contexts", ",".join(contexts)]
     if namespaces:
         cmd.extend(("--namespaces", ",".join(namespaces)))
     if directory:
         cmd.extend(("--directory", directory))
-    commands.run(*cmd)
+
+    # Redirecting stderr to stdout to get the logs. kubectl gather does not
+    # output anytihng to stdout.
+    for line in commands.watch(*cmd, stderr=subprocess.STDOUT):
+        logging.info("[%s] %s", name, line)
 
 
 def _run(cmd, *args, env=None, context=None):