Merge branch 'topic/timw/revert-bro-pkg'

* topic/timw/revert-bro-pkg:
  Update some documentation about bro-pkg.meta
  Revert "Remove references to bro-pkg.meta and bro-pkg.index"

CHANGES

@@ -1,3 +1,11 @@
+3.0.0-3 | 2024-02-20 10:42:47 -0700
+
+  * Update some documentation about bro-pkg.meta (Tim Wojtulewicz, Corelight)
+
+  * Revert "Remove references to bro-pkg.meta and bro-pkg.index" (Tim Wojtulewicz, Corelight)
+
+    This reverts commit 964b0e7235febd17e414134737513d600673e77c.
+
 3.0.0 | 2024-02-15 09:21:49 -0500
 
   * Release 3.0.0.

VERSION

@@ -1 +1 @@
-3.0.0
+3.0.0-3

doc/man/zkg.1

@@ -27,7 +27,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
 .in \\n[rst2man-indent\\n[rst2man-indent-level]]u
 ..
-.TH "ZKG" "1" "Feb 15, 2024" "3.0.0" "Zeek Package Manager"
+.TH "ZKG" "1" "Feb 20, 2024" "3.0.0-3" "Zeek Package Manager"
 .SH NAME
 zkg \- Zeek Package Manager
 .sp
@@ -316,7 +316,7 @@ usage: zkg refresh [\-h] [\-\-aggregate] [\-\-fail\-on\-aggregate\-problems] [\-
 .INDENT 7.0
 .TP
 .B  \-\-aggregate=False
-Crawls the urls listed in package source zkg.index files and aggregates the metadata found in their zkg.meta files.  The aggregated metadata is stored in the local clone of the package source that zkg uses internally for locating package metadata. For each package, the metadata is taken from the highest available git version tag or the default branch, like "main" or "master", if no version tags exist
+Crawls the urls listed in package source zkg.index files and aggregates the metadata found in their zkg.meta (or legacy bro\-pkg.meta) files.  The aggregated metadata is stored in the local clone of the package source that zkg uses internally for locating package metadata. For each package, the metadata is taken from the highest available git version tag or the default branch, like "main" or "master", if no version tags exist
 .TP
 .B  \-\-fail\-on\-aggregate\-problems=False
 When using \-\-aggregate, exit with error when any packages trigger metadata problems. Normally such problems only cause a warning.
@@ -773,7 +773,7 @@ state_dir =
 # A subdirectory named "packages" is always created within the
 # specified path and the package manager will copy the directory
 # specified by the "script_dir" option of each package\(aqs zkg.meta
-# file there.
+# (or legacy bro\-pkg.meta) file there.
 # If left blank or with \-\-user this defaults to <state_dir>/script_dir.
 # In Zeek\-bundled installations, it defaults to
 # <zeek_install_prefix>/share/zeek/site.
@@ -786,7 +786,7 @@ script_dir =
 # A subdirectory named "packages" is always created within the
 # specified path and the package manager will copy the directory
 # specified by the "plugin_dir" option of each package\(aqs zkg.meta
-# file there.
+# (or legacy bro\-pkg.meta) file there.
 # If left blank or with \-\-user this defaults to <state_dir>/plugin_dir.
 # In Zeek\-bundled installations, it defaults to
 # <zeek_install_prefix>/lib/zeek/plugins.
@@ -817,13 +817,13 @@ default = https://github.com/zeek/package\-template
 
 [user_vars]
 
-# For any key in this section that is matched for value interpolation in a
-# package\(aqs zkg.meta file, the corresponding value is substituted during
-# execution of the package\(aqs \(gabuild_command\(ga.  This section is typically
-# automatically populated with the the answers supplied during package
-# installation prompts and, as a convenience feature, used to recall the
-# last\-used settings during subsequent operations (e.g. upgrades) on the same
-# package.
+# For any key in this section that is matched for value interpolation
+# in a package\(aqs zkg.meta (or legacy bro\-pkg.meta) file, the corresponding
+# value is substituted during execution of the package\(aqs \(gabuild_command\(ga.
+# This section is typically automatically populated with the
+# the answers supplied during package installation prompts
+# and, as a convenience feature, used to recall the last\-used settings
+# during subsequent operations (e.g. upgrades) on the same package.
 
 .ft P
 .fi

doc/package.rst

@@ -22,6 +22,12 @@ This is the package's metadata file in INI file format and may contain
 :ref:`additional fields <metadata-fields>` that describe the package as well
 as how it inter-operates with Zeek, the package manager, or other packages.
 
+.. note::
+
+   :file:`zkg.meta` is the canonical metadata file name used :program:`since
+   zkg v2.0`.  The previous metadata file name of :file:`bro-pkg.meta` is also
+   accepted when no :file:`zkg.meta` exists.
+
 .. _package-shorthand-name:
 
 Note that the shorthand name for your package that may be used by :ref:`zkg

doc/source.rst

@@ -22,6 +22,12 @@ usernames or some unique way of identifying the organization/person that
 maintains Zeek packages.  However, a source is free to use a flat organization
 with a single, top-level :file:`zkg.index`.
 
+.. note::
+
+   The magic index file name of :file:`zkg.index` is available :program:`since
+   zkg v2.0`.  For compatibility purposes, the old index file name of
+   :file:`bro-pkg.index` is also still supported.
+
 After creating a git repo for the package source and adding package index files
 to it, it's ready to be used by :ref:`zkg <zkg>`.
 
@@ -30,9 +36,10 @@ to it, it's ready to be used by :ref:`zkg <zkg>`.
 Package Index Files
 -------------------
 
-Files named :file:`zkg.index` are used to describe the :doc:`Zeek Packages
-<package>` found within the package source.  They are simply a list of git URLs
-pointing to the git repositories of packages.  For example::
+Files named :file:`zkg.index` (or the legacy :file:`bro-pkg.index`) are used to
+describe the :doc:`Zeek Packages <package>` found within the package source.
+They are simply a list of git URLs pointing to the git repositories of
+packages.  For example::
 
   https://github.com/zeek/foo
   https://github.com/zeek/bar
@@ -68,8 +75,9 @@ Aggregating Metadata
 --------------------
 
 The maintainer/operator of a package source may choose to periodically aggregate
-the metadata contained in its packages' :file:`zkg.meta` files.  The :ref:`zkg
-refresh <refresh-command>` is used to perform the task. For example:
+the metadata contained in its packages' :file:`zkg.meta` (and legacy
+:file:`bro-pkg.meta`) files.  The :ref:`zkg refresh <refresh-command>`
+is used to perform the task.  For example:
 
 .. code-block:: console
 

testing/baselines/tests.install-bro-pkg-warning/stderr

@@ -0,0 +1,2 @@
+### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
+XXXX-XX-XX XX:XX:XX WARNING  Package one/alice/foo is using the legacy bro-pkg.meta metadata file. While bro-pkg.meta still functions, it is recommended to use zkg.meta instead for future-proofing. Please report this to the package maintainers.

testing/baselines/tests.refresh/agg.errout

@@ -1,8 +1,8 @@
 ### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
 XXXX-XX-XX XX:XX:XX DEBUG    found source clone of "one" at <...>/one
 XXXX-XX-XX XX:XX:XX DEBUG    refresh "one": pulling <...>/one
-XXXX-XX-XX XX:XX:XX WARNING  <...>/zkg.meta: missing metadata file
-XXXX-XX-XX XX:XX:XX WARNING  skipping aggregation of <...>/bad_pkg: bad metadata: missing zkg.meta metadata file
+XXXX-XX-XX XX:XX:XX WARNING  <...>/bro-pkg.meta: missing metadata file
+XXXX-XX-XX XX:XX:XX WARNING  skipping aggregation of <...>/bad_pkg: bad metadata: missing zkg.meta (or bro-pkg.meta) metadata file
 XXXX-XX-XX XX:XX:XX DEBUG    metadata refresh: 8 additions (alice/bar, alice/baz, alice/foo, alice/i-have-no-scripts, alice/new_pkg, alice/qux, bob/corge, bob/grault), 0 changes, 0 removals
 XXXX-XX-XX XX:XX:XX INFO     committed package source "one" metadata update
 XXXX-XX-XX XX:XX:XX DEBUG    fetch package one<...>/foo

testing/baselines/tests.refresh/agg.fail.errout

@@ -1,7 +1,7 @@
 ### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
 XXXX-XX-XX XX:XX:XX DEBUG    found source clone of "one" at <...>/one
 XXXX-XX-XX XX:XX:XX DEBUG    refresh "one": pulling <...>/one
-XXXX-XX-XX XX:XX:XX WARNING  <...>/zkg.meta: missing metadata file
-XXXX-XX-XX XX:XX:XX WARNING  skipping aggregation of <...>/bad_pkg: bad metadata: missing zkg.meta metadata file
+XXXX-XX-XX XX:XX:XX WARNING  <...>/bro-pkg.meta: missing metadata file
+XXXX-XX-XX XX:XX:XX WARNING  skipping aggregation of <...>/bad_pkg: bad metadata: missing zkg.meta (or bro-pkg.meta) metadata file
 XXXX-XX-XX XX:XX:XX DEBUG    metadata refresh: 8 additions (alice/bar, alice/baz, alice/foo, alice/i-have-no-scripts, alice/new_pkg, alice/qux, bob/corge, bob/grault), 0 changes, 0 removals
 XXXX-XX-XX XX:XX:XX DEBUG    fetch package one<...>/foo

testing/baselines/tests.refresh/agg.fail.out

@@ -2,6 +2,6 @@
 Refresh package source: one
 	No membership changes
 	WARNING: Metadata aggregated, but excludes the following packages due to described problems:
-		<...>/bad_pkg: missing zkg.meta metadata file
+		<...>/bad_pkg: missing zkg.meta (or bro-pkg.meta) metadata file
 Refresh installed packages
 	No new outdated packages

testing/baselines/tests.refresh/agg.out

@@ -2,7 +2,7 @@
 Refresh package source: one
 	No membership changes
 	WARNING: Metadata aggregated, but excludes the following packages due to described problems:
-		<...>/bad_pkg: missing zkg.meta metadata file
+		<...>/bad_pkg: missing zkg.meta (or bro-pkg.meta) metadata file
 	Pushed aggregated metadata
 Refresh installed packages
 	No new outdated packages

testing/tests/install-bro-pkg-warning

@@ -0,0 +1,3 @@
+# @TEST-EXEC: (cd packages/foo; git mv zkg.meta bro-pkg.meta; git commit -m 'Use bro-pkg.meta')
+# @TEST-EXEC: zkg install foo 2> stderr
+# @TEST-EXEC: TEST_DIFF_CANONIFIER='sed -r "s/[0-9]{2}/XX/g"' btest-diff stderr

zeekpkg/__init__.py

@@ -9,7 +9,7 @@
 
 import logging
 
-__version__ = "3.0.0"
+__version__ = "3.0.0-3"
 __all__ = ["manager", "package", "source", "template", "uservar"]
 
 LOG = logging.getLogger(__name__)

zeekpkg/manager.py

@@ -50,6 +50,7 @@
     BUILTIN_SOURCE,
     BUILTIN_SCHEME,
     METADATA_FILENAME,
+    LEGACY_METADATA_FILENAME,
     TRACKING_METHOD_VERSION,
     TRACKING_METHOD_BRANCH,
     TRACKING_METHOD_COMMIT,
@@ -192,19 +193,20 @@ class Manager:
         scratch_dir (str): a directory where the package manager performs
             miscellaneous/temporary file operations
 
-        script_dir (str): the directory where the package manager will copy each
-            installed package's `script_dir` (as given by its :file:`zkg.meta`).
-            Each package gets a subdirectory within `script_dir` associated with
-            its name.
+        script_dir (str): the directory where the package manager will
+            copy each installed package's `script_dir` (as given by its
+            :file:`zkg.meta` or :file:`bro-pkg.meta`).  Each package gets a
+            subdirectory within `script_dir` associated with its name.
 
-        plugin_dir (str): the directory where the package manager will copy each
-            installed package's `plugin_dir` (as given by its :file:`zkg.meta`).
-            Each package gets a subdirectory within `plugin_dir` associated with
-            its name.
+        plugin_dir (str): the directory where the package manager will
+            copy each installed package's `plugin_dir` (as given by its
+            :file:`zkg.meta` or :file:`bro-pkg.meta`).  Each package gets a
+            subdirectory within `plugin_dir` associated with its name.
 
         bin_dir (str): the directory where the package manager will link
             executables into that are provided by an installed package through
-            `executables` (as given by its :file:`zkg.meta`)
+            `executables` (as given by its :file:`zkg.meta` or
+            :file:`bro-pkg.meta`)
 
         source_clonedir (str): the directory where the package manager
             will clone package sources.  Each source gets a subdirectory
@@ -231,7 +233,6 @@ class Manager:
             in a directory named :file:`packages`, so as long as
             :envvar:`ZEEKPATH` is configured correctly, ``@load packages`` will
             load all installed packages that have been marked as loaded.
-
     """
 
     def __init__(
@@ -1150,7 +1151,7 @@ def _refresh_source(self, name, aggregate=False, push=False):
                         aggregation_issues.append((url, msg))
                         continue
 
-                    metadata_file = os.path.join(clone.working_dir, METADATA_FILENAME)
+                    metadata_file = _pick_metadata_file(clone.working_dir)
                     metadata_parser = configparser.ConfigParser(interpolation=None)
                     invalid_reason = _parse_package_metadata(
                         metadata_parser, metadata_file
@@ -2732,7 +2733,7 @@ def _stage(self, package, version, clone, stage, env=None):
 
         """
         LOG.debug('staging "%s": version %s', package, version)
-        metadata_file = os.path.join(clone.working_dir, METADATA_FILENAME)
+        metadata_file = _pick_metadata_file(clone.working_dir)
         metadata_parser = configparser.ConfigParser(interpolation=None)
         invalid_reason = _parse_package_metadata(metadata_parser, metadata_file)
         if invalid_reason:
@@ -3071,7 +3072,7 @@ def _install(self, package, version, use_existing_clone=False):
         status.current_hash = clone.head.object.hexsha
         status.is_outdated = _is_clone_outdated(clone, version, status.tracking_method)
 
-        metadata_file = os.path.join(clone.working_dir, METADATA_FILENAME)
+        metadata_file = _pick_metadata_file(clone.working_dir)
         metadata_parser = configparser.ConfigParser(interpolation=None)
         invalid_reason = _parse_package_metadata(metadata_parser, metadata_file)
 
@@ -3252,7 +3253,7 @@ def ignore(_, files):
         rval = []
 
         for f in files:
-            if f in {".git", "zkg.meta"}:
+            if f in {".git", "bro-pkg.meta", "zkg.meta"}:
                 rval.append(f)
 
         return rval
@@ -3302,11 +3303,22 @@ def _get_package_metadata(parser):
     return metadata
 
 
+def _pick_metadata_file(directory):
+    rval = os.path.join(directory, METADATA_FILENAME)
+
+    if os.path.exists(rval):
+        return rval
+
+    return os.path.join(directory, LEGACY_METADATA_FILENAME)
+
+
 def _parse_package_metadata(parser, metadata_file):
     """Return string explaining why metadata is invalid, or '' if valid."""
     if not parser.read(metadata_file):
         LOG.warning("%s: missing metadata file", metadata_file)
-        return f"missing {METADATA_FILENAME} metadata file"
+        return "missing {} (or {}) metadata file".format(
+            METADATA_FILENAME, LEGACY_METADATA_FILENAME
+        )
 
     if not parser.has_section("package"):
         LOG.warning("%s: metadata missing [package]", metadata_file)
@@ -3319,6 +3331,9 @@ def _parse_package_metadata(parser, metadata_file):
     return ""
 
 
+_legacy_metadata_warnings = set()
+
+
 def _info_from_clone(clone, package, status, version):
     """Retrieves information about a package.
 
@@ -3335,7 +3350,7 @@ def _info_from_clone(clone, package, status, version):
     else:
         version_type = TRACKING_METHOD_BRANCH
 
-    metadata_file = os.path.join(clone.working_dir, METADATA_FILENAME)
+    metadata_file = _pick_metadata_file(clone.working_dir)
     metadata_parser = configparser.ConfigParser(interpolation=None)
     invalid_reason = _parse_package_metadata(metadata_parser, metadata_file)
 
@@ -3351,6 +3366,19 @@ def _info_from_clone(clone, package, status, version):
             default_branch=default_branch,
         )
 
+    if (
+        os.path.basename(metadata_file) == LEGACY_METADATA_FILENAME
+        and package.qualified_name() not in _legacy_metadata_warnings
+    ):
+        LOG.warning(
+            "Package %s is using the legacy bro-pkg.meta metadata file. "
+            "While bro-pkg.meta still functions, it is recommended to "
+            "use zkg.meta instead for future-proofing. Please report this "
+            "to the package maintainers.",
+            package.qualified_name(),
+        )
+        _legacy_metadata_warnings.add(package.qualified_name())
+
     metadata = _get_package_metadata(metadata_parser)
 
     return PackageInfo(

zeekpkg/package.py

@@ -15,6 +15,7 @@
 
 #: The name of files used by packages to store their metadata.
 METADATA_FILENAME = "zkg.meta"
+LEGACY_METADATA_FILENAME = "bro-pkg.meta"
 
 TRACKING_METHOD_VERSION = "version"
 TRACKING_METHOD_BRANCH = "branch"
@@ -306,7 +307,7 @@ class PackageInfo:
             packages
 
         metadata (dict of str -> str): the contents of the package's
-            :file:`zkg.meta`
+            :file:`zkg.meta` or :file:`bro-pkg.meta`
 
         versions (list of str): a list of the package's availabe git version
             tags
@@ -320,9 +321,10 @@ class PackageInfo:
         invalid_reason (str): this attribute is set when there is a problem
             with gathering package information and explains what went wrong.
 
-        metadata_file: the absolute path to the :file:`zkg.meta` or for this
-            package. Use this if you'd like to parse the metadata yourself.
-            May not be defined, in which case the value is None.
+        metadata_file: the absolute path to the :file:`zkg.meta` or
+            :file:`bro-pkg.meta` for this package.  Use this if you'd like to
+            parse the metadata yourself. May not be defined, in which case the
+            value is None.
     """
 
     def __init__(
@@ -435,10 +437,10 @@ class Package:
             or if it's located in a top-level :file:`zkg.index` file.
 
         metadata (dict of str -> str): the contents of the package's
-            :file:`zkg.meta` file.  If the package has not been installed then
-            this information may come from the last aggregation of the source's
-            :file:`aggregate.meta` file (it may not be accurate/up-to-date).
-
+            :file:`zkg.meta` or :file:`bro-pkg.meta` file.  If the package has
+            not been installed then this information may come from the last
+            aggregation of the source's :file:`aggregate.meta` file (it may not
+            be accurate/up-to-date).
     """
 
     def __init__(