GH 675 fs-mode run report (#692)

docs/sections/user_guide/cli/tools/config.rst

@@ -90,6 +90,7 @@ The examples that follow use identical namelist files ``a.nml`` and ``b.nml`` wi
 In ``uw`` terminology, to realize a configuration file is to transform it from its raw form into its final, usable state. The ``realize`` action can build a complete config file from two or more separate files.
 
 .. literalinclude:: config/realize-help.cmd
+   :language: text
    :emphasize-lines: 1
 .. literalinclude:: config/realize-help.out
    :language: text

docs/sections/user_guide/cli/tools/fs.rst

@@ -6,6 +6,7 @@
 The ``uw`` mode for handling filesystem items (files and directories).
 
 .. literalinclude:: fs/help.cmd
+   :language: text
    :emphasize-lines: 1
 .. literalinclude:: fs/help.out
    :language: text
@@ -18,6 +19,7 @@ The ``copy`` action stages files in a target directory by copying files. Any ``K
 Source paths prefixed with ``http://`` or ``https://`` will be copied from their upstream network locations to the local filesystem.
 
 .. literalinclude:: fs/copy-help.cmd
+   :language: text
    :emphasize-lines: 1
 .. literalinclude:: fs/copy-help.out
    :language: text
@@ -30,6 +32,7 @@ Given ``copy-config.yaml`` containing a mapping from local-filesystem destinatio
 .. literalinclude:: fs/copy-config.yaml
    :language: yaml
 .. literalinclude:: fs/copy-exec.cmd
+   :language: text
    :emphasize-lines: 2
 .. literalinclude:: fs/copy-exec.out
    :language: text
@@ -41,6 +44,7 @@ The ``--cycle`` and ``--leadtime`` options can be used to make Python ``datetime
 .. literalinclude:: fs/copy-config-timedep.yaml
    :language: yaml
 .. literalinclude:: fs/copy-exec-timedep.cmd
+   :language: text
    :emphasize-lines: 2
 .. literalinclude:: fs/copy-exec-timedep.out
    :language: text
@@ -50,16 +54,36 @@ The ``--target-dir`` option need not be specified when all destination paths are
 .. literalinclude:: fs/copy-config.yaml
    :language: yaml
 .. literalinclude:: fs/copy-exec-no-target-dir-err.cmd
+   :language: text
    :emphasize-lines: 1
 .. literalinclude:: fs/copy-exec-no-target-dir-err.out
    :language: text
 
+When the ``--report`` option is specified, a report of files not copied ("not-ready") and copied ("ready") will be printed to ``stdout`` as machine-readable JSON. For example, using a config specifying both available and unavailable source files:
+
+.. literalinclude:: fs/copy-config-report.yaml
+   :language: yaml
+.. literalinclude:: fs/copy-exec-report.cmd
+   :language: text
+   :emphasize-lines: 2
+.. literalinclude:: fs/copy-exec-report.out
+   :language: text
+
+Since ``uwtools`` logs to ``stderr``, log and report output can be separated and the latter processed with a tool like ``jq``:
+
+.. literalinclude:: fs/copy-exec-report-jq.cmd
+   :language: text
+   :emphasize-lines: 2
+.. literalinclude:: fs/copy-exec-report-jq.out
+   :language: text
+
 ``link``
 --------
 
 The ``link`` action stages files in a target directory by linking files, directories, or other symbolic links. Any ``KEY`` positional arguments are used to navigate, in the order given, from the top of the config to the :ref:`file block <files_yaml>`.
 
 .. literalinclude:: fs/link-help.cmd
+   :language: text
    :emphasize-lines: 1
 .. literalinclude:: fs/link-help.out
    :language: text
@@ -72,6 +96,7 @@ Given ``link-config.yaml`` containing
 .. literalinclude:: fs/link-config.yaml
    :language: yaml
 .. literalinclude:: fs/link-exec.cmd
+   :language: text
    :emphasize-lines: 2
 .. literalinclude:: fs/link-exec.out
    :language: text
@@ -83,6 +108,7 @@ The ``--cycle`` and ``--leadtime`` options can be used to make Python ``datetime
 .. literalinclude:: fs/link-config-timedep.yaml
    :language: yaml
 .. literalinclude:: fs/link-exec-timedep.cmd
+   :language: text
    :emphasize-lines: 2
 .. literalinclude:: fs/link-exec-timedep.out
    :language: text
@@ -92,16 +118,36 @@ The ``--target-dir`` option need not be specified when all linkname paths are ab
 .. literalinclude:: fs/link-config.yaml
    :language: yaml
 .. literalinclude:: fs/link-exec-no-target-dir-err.cmd
+   :language: text
    :emphasize-lines: 1
 .. literalinclude:: fs/link-exec-no-target-dir-err.out
    :language: text
 
+When the ``--report`` option is specified, a report of files not linked ("not-ready") and linked ("ready") will be printed to ``stdout`` as machine-readable JSON. For example, using a config specifying both available and unavailable source files:
+
+.. literalinclude:: fs/link-config-report.yaml
+   :language: yaml
+.. literalinclude:: fs/link-exec-report.cmd
+   :language: text
+   :emphasize-lines: 2
+.. literalinclude:: fs/link-exec-report.out
+   :language: text
+
+Since ``uwtools`` logs to ``stderr``, log and report output can be separated and the latter processed with a tool like ``jq``:
+
+.. literalinclude:: fs/link-exec-report-jq.cmd
+   :language: text
+   :emphasize-lines: 2
+.. literalinclude:: fs/link-exec-report-jq.out
+   :language: text
+
 ``makedirs``
 ------------
 
 The ``makedirs`` action creates directories. Any ``KEY`` positional arguments are used to navigate, in the order given, from the top of the config to the :ref:`makedirs block <makedirs_yaml>`.
 
 .. literalinclude:: fs/makedirs-help.cmd
+   :language: text
    :emphasize-lines: 1
 .. literalinclude:: fs/makedirs-help.out
    :language: text
@@ -114,6 +160,7 @@ Given ``makedirs-config.yaml`` containing
 .. literalinclude:: fs/makedirs-config.yaml
    :language: yaml
 .. literalinclude:: fs/makedirs-exec.cmd
+   :language: text
    :emphasize-lines: 2
 .. literalinclude:: fs/makedirs-exec.out
    :language: text
@@ -123,6 +170,7 @@ The ``--cycle`` and ``--leadtime`` options can be used to make Python ``datetime
 .. literalinclude:: fs/makedirs-config-timedep.yaml
    :language: yaml
 .. literalinclude:: fs/makedirs-exec-timedep.cmd
+   :language: text
    :emphasize-lines: 2
 .. literalinclude:: fs/makedirs-exec-timedep.out
    :language: text
@@ -132,6 +180,25 @@ The ``--target-dir`` option need not be specified when all directory paths are a
 .. literalinclude:: fs/makedirs-config.yaml
    :language: yaml
 .. literalinclude:: fs/makedirs-exec-no-target-dir-err.cmd
+   :language: text
    :emphasize-lines: 1
 .. literalinclude:: fs/makedirs-exec-no-target-dir-err.out
    :language: text
+
+When the ``--report`` option is specified, a report of directories not created ("not-ready") and created ("ready") will be printed to ``stdout`` as machine-readable JSON. For example, using a config specifying both available and unavailable source files:
+
+.. literalinclude:: fs/makedirs-config-report.yaml
+   :language: yaml
+.. literalinclude:: fs/makedirs-exec-report.cmd
+   :language: text
+   :emphasize-lines: 5
+.. literalinclude:: fs/makedirs-exec-report.out
+   :language: text
+
+Since ``uwtools`` logs to ``stderr``, log and report output can be separated and the latter processed with a tool like ``jq``:
+
+.. literalinclude:: fs/makedirs-exec-report-jq.cmd
+   :language: text
+   :emphasize-lines: 5
+.. literalinclude:: fs/makedirs-exec-report-jq.out
+   :language: text

docs/sections/user_guide/cli/tools/fs/.gitignore

@@ -1,4 +1,7 @@
 copy-dst
+copy-dst-report-jq
 copy-dst-timedep
 link-dst
+link-dst-report
+link-dst-report-jq
 link-dst-timedep

docs/sections/user_guide/cli/tools/fs/copy-config-report.yaml

@@ -0,0 +1,2 @@
+foo: src/foo
+qux: src/qux

docs/sections/user_guide/cli/tools/fs/copy-exec-report-jq.cmd

@@ -0,0 +1,2 @@
+rm -rf copy-dst
+uw fs copy --report --target-dir copy-dst --config-file copy-config-report.yaml 2>/dev/null | jq -r .ready[]

docs/sections/user_guide/cli/tools/fs/copy-exec-report-jq.out

@@ -0,0 +1 @@
+copy-dst/foo

docs/sections/user_guide/cli/tools/fs/copy-exec-report.cmd

@@ -0,0 +1,2 @@
+rm -rf copy-dst-report-jq
+uw fs copy --report --target-dir copy-dst-report-jq --config-file copy-config-report.yaml

docs/sections/user_guide/cli/tools/fs/copy-exec-report.out

@@ -0,0 +1,21 @@
+[2025-01-30T06:36:44]     INFO Validating config against internal schema: files-to-stage
+[2025-01-30T06:36:44]     INFO 0 schema-validation errors found in fs config
+[2025-01-30T06:36:48]     INFO File src/foo: Ready
+[2025-01-30T06:36:48]  WARNING File src/qux: Not ready [external asset]
+[2025-01-30T06:36:48]     INFO Copy src/foo -> copy-dst-report-jq/foo: Executing
+[2025-01-30T06:36:48]     INFO Copy src/foo -> copy-dst-report-jq/foo: Ready
+[2025-01-30T06:36:48]  WARNING Copy src/qux -> copy-dst-report-jq/qux: Not ready
+[2025-01-30T06:36:48]  WARNING Copy src/qux -> copy-dst-report-jq/qux: Requires:
+[2025-01-30T06:36:48]  WARNING Copy src/qux -> copy-dst-report-jq/qux: ✖ File src/qux
+[2025-01-30T06:36:48]  WARNING File copies: Not ready
+[2025-01-30T06:36:48]  WARNING File copies: Requires:
+[2025-01-30T06:36:48]  WARNING File copies: ✔ Copy src/foo -> copy-dst-report-jq/foo
+[2025-01-30T06:36:48]  WARNING File copies: ✖ Copy src/qux -> copy-dst-report-jq/qux
+{
+  "not-ready": [
+    "copy-dst-report-jq/qux"
+  ],
+  "ready": [
+    "copy-dst-report-jq/foo"
+  ]
+}

docs/sections/user_guide/cli/tools/fs/copy-help.out

@@ -1,6 +1,6 @@
 usage: uw fs copy [-h] [--version] [--config-file PATH] [--target-dir PATH]
                   [--cycle CYCLE] [--leadtime LEADTIME] [--dry-run]
-                  [--key-path KEY[.KEY...]] [--quiet] [--verbose]
+                  [--key-path KEY[.KEY...]] [--report] [--quiet] [--verbose]
 
 Copy files
 
@@ -21,6 +21,8 @@ Optional arguments:
       Only log info, making no changes
   --key-path KEY[.KEY...]
       Dot-separated path of keys to config block to use
+  --report
+      Show JSON report on [non]ready assets
   --quiet, -q
       Print no logging messages
   --verbose, -v

docs/sections/user_guide/cli/tools/fs/link-config-report.yaml

@@ -0,0 +1,2 @@
+foo: src/foo
+qux: src/qux

docs/sections/user_guide/cli/tools/fs/link-exec-report-jq.cmd

@@ -0,0 +1,2 @@
+rm -rf link-dst-report-jq
+uw fs link --report --target-dir link-dst-report-jq --config-file link-config-report.yaml 2>/dev/null | jq -r .ready[]

docs/sections/user_guide/cli/tools/fs/link-exec-report-jq.out

@@ -0,0 +1 @@
+link-dst-report-jq/foo

docs/sections/user_guide/cli/tools/fs/link-exec-report.cmd

@@ -0,0 +1,2 @@
+rm -rf link-dst-report
+uw fs link --report --target-dir link-dst-report --config-file link-config-report.yaml

docs/sections/user_guide/cli/tools/fs/link-exec-report.out

@@ -0,0 +1,21 @@
+[2025-01-30T06:36:37]     INFO Validating config against internal schema: files-to-stage
+[2025-01-30T06:36:38]     INFO 0 schema-validation errors found in fs config
+[2025-01-30T06:36:40]     INFO Filesystem item src/foo: Ready
+[2025-01-30T06:36:40]  WARNING Filesystem item src/qux: Not ready [external asset]
+[2025-01-30T06:36:40]     INFO Link link-dst-report/foo -> src/foo: Executing
+[2025-01-30T06:36:40]     INFO Link link-dst-report/foo -> src/foo: Ready
+[2025-01-30T06:36:41]  WARNING Link link-dst-report/qux -> src/qux: Not ready
+[2025-01-30T06:36:41]  WARNING Link link-dst-report/qux -> src/qux: Requires:
+[2025-01-30T06:36:41]  WARNING Link link-dst-report/qux -> src/qux: ✖ Filesystem item src/qux
+[2025-01-30T06:36:41]  WARNING File links: Not ready
+[2025-01-30T06:36:41]  WARNING File links: Requires:
+[2025-01-30T06:36:41]  WARNING File links: ✔ Link link-dst-report/foo -> src/foo
+[2025-01-30T06:36:41]  WARNING File links: ✖ Link link-dst-report/qux -> src/qux
+{
+  "not-ready": [
+    "link-dst-report/qux"
+  ],
+  "ready": [
+    "link-dst-report/foo"
+  ]
+}

docs/sections/user_guide/cli/tools/fs/link-help.out

@@ -1,6 +1,6 @@
 usage: uw fs link [-h] [--version] [--config-file PATH] [--target-dir PATH]
                   [--cycle CYCLE] [--leadtime LEADTIME] [--dry-run]
-                  [--key-path KEY[.KEY...]] [--quiet] [--verbose]
+                  [--key-path KEY[.KEY...]] [--report] [--quiet] [--verbose]
 
 Link files
 
@@ -21,6 +21,8 @@ Optional arguments:
       Only log info, making no changes
   --key-path KEY[.KEY...]
       Dot-separated path of keys to config block to use
+  --report
+      Show JSON report on [non]ready assets
   --quiet, -q
       Print no logging messages
   --verbose, -v

docs/sections/user_guide/cli/tools/fs/makedirs-config-report.yaml

@@ -0,0 +1,3 @@
+makedirs:
+  - foo
+  - subdir/bar

docs/sections/user_guide/cli/tools/fs/makedirs-exec-report-jq.cmd

@@ -0,0 +1,6 @@
+d=makedirs-parent-report-jq
+rm -rf $d
+mkdir -p $d/subdir
+chmod 550 $d/subdir # read-only
+uw fs makedirs --report --target-dir $d --config-file makedirs-config-report.yaml 2>/dev/null | jq -r .ready[]
+chmod 750 $d/subdir # read-write

docs/sections/user_guide/cli/tools/fs/makedirs-exec-report-jq.out

@@ -0,0 +1 @@
+makedirs-parent-report-jq/foo

docs/sections/user_guide/cli/tools/fs/makedirs-exec-report.cmd

@@ -0,0 +1,6 @@
+d=makedirs-parent-report
+rm -rf $d
+mkdir -p $d/subdir
+chmod 550 $d/subdir # read-only
+uw fs makedirs --report --target-dir $d --config-file makedirs-config-report.yaml
+chmod 750 $d/subdir # read-write

docs/sections/user_guide/cli/tools/fs/makedirs-exec-report.out

@@ -0,0 +1,18 @@
+[2025-01-30T06:32:58]     INFO Validating config against internal schema: makedirs
+[2025-01-30T06:32:58]     INFO 0 schema-validation errors found in fs config
+[2025-01-30T06:33:00]     INFO Directory makedirs-parent-report/foo: Executing
+[2025-01-30T06:33:00]     INFO Directory makedirs-parent-report/foo: Ready
+[2025-01-30T06:33:01]     INFO Directory makedirs-parent-report/subdir/bar: Executing
+[2025-01-30T06:33:01]  WARNING Directory makedirs-parent-report/subdir/bar: Not ready
+[2025-01-30T06:33:01]  WARNING Directories: Not ready
+[2025-01-30T06:33:01]  WARNING Directories: Requires:
+[2025-01-30T06:33:01]  WARNING Directories: ✔ Directory makedirs-parent-report/foo
+[2025-01-30T06:33:01]  WARNING Directories: ✖ Directory makedirs-parent-report/subdir/bar
+{
+  "not-ready": [
+    "makedirs-parent-report/subdir/bar"
+  ],
+  "ready": [
+    "makedirs-parent-report/foo"
+  ]
+}

docs/sections/user_guide/cli/tools/fs/makedirs-help.out

@@ -1,7 +1,8 @@
 usage: uw fs makedirs [-h] [--version] [--config-file PATH]
                       [--target-dir PATH] [--cycle CYCLE]
                       [--leadtime LEADTIME] [--dry-run]
-                      [--key-path KEY[.KEY...]] [--quiet] [--verbose]
+                      [--key-path KEY[.KEY...]] [--report] [--quiet]
+                      [--verbose]
 
 Make directories
 
@@ -22,6 +23,8 @@ Optional arguments:
       Only log info, making no changes
   --key-path KEY[.KEY...]
       Dot-separated path of keys to config block to use
+  --report
+      Show JSON report on [non]ready assets
   --quiet, -q
       Print no logging messages
   --verbose, -v

notebooks/exp-config-cb.ipynb

@@ -350,17 +350,17 @@
      "name": "stderr",
      "output_type": "stream",
      "text": [
-      "[2024-11-19T23:14:15]     INFO Validating config against internal schema: chgres-cube\n",
-      "[2024-11-19T23:14:15]     INFO 0 schema-validation errors found in chgres_cube config\n",
-      "[2024-11-19T23:14:15]     INFO Validating config against internal schema: platform\n",
-      "[2024-11-19T23:14:15]     INFO 0 schema-validation errors found in platform config\n",
-      "[2024-11-19T23:14:15]     INFO 20241120 05:14:15 chgres_cube valid schema: State: Ready\n"
+      "[2025-01-30T06:42:11]     INFO Validating config against internal schema: chgres-cube\n",
+      "[2025-01-30T06:42:11]     INFO 0 schema-validation errors found in chgres_cube config\n",
+      "[2025-01-30T06:42:11]     INFO Validating config against internal schema: platform\n",
+      "[2025-01-30T06:42:11]     INFO 0 schema-validation errors found in platform config\n",
+      "[2025-01-30T06:42:11]     INFO 20250130 12:42:11 chgres_cube valid schema: Ready\n"
      ]
     },
     {
      "data": {
       "text/plain": [
-       "Asset(ref=None, ready=<function Assets.validate.<locals>.<lambda> at 0xffff685ad6c0>)"
+       "20250130 12:42:11 chgres_cube valid schema <281473276425904>"
       ]
      },
      "execution_count": 8,