cleanup, bugfix, document usage

README.md

@@ -21,6 +21,7 @@ It includes automatic parsing of the `Puppetfile`, `environment.conf` and others
     - [Puppetfile](#puppetfile)
   - [Spec testing](#spec-testing)
     - [Adding your own spec tests](#adding-your-own-spec-tests)
+    - [Vendored Modules](#vendored-modules)
   - [Using Workarounds](#using-workarounds)
   - [Extra tooling](#extra-tooling)
     - [Plugins](#plugins)
@@ -33,6 +34,7 @@ It includes automatic parsing of the `Puppetfile`, `environment.conf` and others
     - [Ruby Warnings](#ruby-warnings)
     - [Rake tasks](#rake-tasks)
       - [generate_fixtures](#generate_fixtures)
+      - [generate_vendor_cache](#generate_vendor_cache)
 
 ## Overview
 
@@ -605,6 +607,30 @@ If you want to see Puppet's output, you can set the `SHOW_PUPPET_OUTPUT` environ
 
 `SHOW_PUPPET_OUTPUT=true onceover run spec`
 
+### Vendored Modules
+
+As of Puppet 6.0 some resource types were removed from Puppet and repackaged as individual modules. These supported type modules are still included in the `puppet-agent` package, so you don't have to download them from the Forge. However, this does not apply to the `puppet` gem used when spec testing. This frequently results in users wondering why their Puppet manifests apply just fine on a node, but their tests fail with messages like `Unknown resource type: cron_core` for example. A common workaround for this problem was to add said modules into your Puppetfile, thus requiring manual management.
+
+Onceover now has the ability to remove that manual process for you by querying Github's API to determine which versions are in use by the version of the [puppet-agent package](https://github.com/puppetlabs/puppet-agent/tree/main/configs/components) you are testing against.
+
+This functionality is opt in, so to use it configure the following:
+
+```yaml
+# onceover.yaml
+opts:
+  auto_vendored: true
+```
+
+or on the cli:
+
+```shell
+bundle exec onceover run spec --auto_vendored=true
+```
+
+Essentially what this is doing is resolving any of these [supported type modules](https://www.puppet.com/docs/puppet/8/type#supported-type-modules-in-puppet-agent) that are not already specified in your Puppetfile, and adding them to the copy Onceover uses to deploy into its working directory structure.
+
+CI/CD pipeline users are encouraged to provide Onceover with a cache of the module versions to test against in order to avoid hitting Githubs API ratelimit. To do so, the [generate_vendor_cache](#generate_vendor_cache) rake task can be used to populate the cache into your `spec/vendored_modules` directory.
+
 ## Using workarounds
 
 There may be situations where you cannot test everything that is in your puppet code, some common reasons for this include:
@@ -879,6 +905,12 @@ fixtures:
 
 Notice that the symlinks are not the ones that we provided in `environment.conf`? This is because the rake task will go into each of directories, find the modules and create a symlink for each of them (This is what rspec expects).
 
+#### generate_vendor_cache
+
+`bundle exec rake generate_vendor_cache`
+
+This task will query Github's API to determine the versions of the vendored modules in use by the version of the puppet agent you are testing against, and cache that information in `control-repo/spec/vendored_modules`. This way your pipelines won't need to reach out for this information each time Onceover is ran with `auto_vendored` enabled.
+
 ## Developing Onceover
 
 Install gem dependencies:

features/auto_vendored.feature

@@ -1,4 +1,4 @@
-@vendored
+@vendored @puppet6
 Feature: Automatically resolve modules vendored with puppet-agent package
   Onceover should optionally attempt to resolve these vendored modules so that
   users do not need to maintain these in their Puppetfile's unless they have a reason

lib/onceover/vendored_modules.rb

@@ -26,35 +26,37 @@ class VendoredModules
     attr_reader :vendored_references, :missing_vendored
 
     def initialize(opts = {})
-#    def initialize(repo = Onceover::Controlrepo.new, cachedir = nil)
       @repo = opts[:repo] || Onceover::Controlrepo.new
       @cachedir = opts[:cachedir] || File.join(@repo.tempdir, 'vendored_modules')
       @puppet_version = Gem::Version.new(Puppet.version)
-      @puppet_major_version = Gem::Version.new(@puppet_version).segments[0]
+      @puppet_major_version = Gem::Version.new(@puppet_version.segments[0])
       @force_update = opts[:force_update] || false
 
       @missing_vendored = []
 
+      # This only applies to puppet >= 6 so bail early
+      raise 'Auto resolving vendored modules only applies to puppet versions >= 6' unless @puppet_major_version >= Gem::Version.new('6')
+
       # Create cachedir
       unless File.directory?(@cachedir)
         logger.debug "Creating #{@cachedir}"
         FileUtils.mkdir_p(@cachedir)
       end
 
-      # location of user provided caches:
+      # Location of user provided caches:
       #   control-repo/spec/vendored_modules/<component>-puppet_agent-<agent version>.json
       @manual_vendored_dir = File.join(@repo.spec_dir, 'vendored_modules')
 
-      # get the entire file tree of the puppetlabs/puppet-agent repository
+      # Get the entire file tree of the puppetlabs/puppet-agent repository
       # https://docs.github.com/en/rest/git/trees?apiVersion=2022-11-28#get-a-tree
       puppet_agent_tree = query_or_cache(
         "https://api.github.com/repos/puppetlabs/puppet-agent/git/trees/#{@puppet_version}",
         { :recursive => true },
         component_cache('repo_tree')
       )
-      # get only the module-puppetlabs-<something>_core.json component files
+      # Get only the module-puppetlabs-<something>_core.json component files
       vendored_components =  puppet_agent_tree['tree'].select { |file| /configs\/components\/module-puppetlabs-\w+\.json/.match(file['path']) }
-      # get the contents of each component file
+      # Get the contents of each component file
       # https://docs.github.com/en/rest/git/blobs?apiVersion=2022-11-28#get-a-blob
       @vendored_references = vendored_components.map do |component|
         mod_slug = component['path'].match(/.*(puppetlabs-\w+).json$/)[1]
@@ -73,26 +75,34 @@ def component_cache(component)
       # By default look for any caches created during previous runs
       cache_file = File.join(@cachedir, desired_name)
 
-      # If the user provides their own cache
-      if File.directory?(@manual_vendored_dir)
-        # Check for any '<component>-puppet_agent-<puppet version>.json' files
-        dg = Dir.glob(File.join(@manual_vendored_dir, "#{component}-puppet_agent*"))
-        # Check if there are multiple versions of the component cache
-        if dg.size > 1
-          # If there is the same version supplied as whats being tested against use that
-          if dg.any? { |s| s[desired_name] }
-            cache_file = File.join(@manual_vendored_dir, desired_name)
-          # If there are any with the same major version, use the latest supplied
-          elsif dg.any? { |s| s["#{component}-puppet_agent-#{@puppet_major_version}"] }
-            maj_match = dg.select { |f| /#{component}-puppet_agent-#{@puppet_major_version}.\d+\.\d+\.json/.match(f) }
-            maj_match.each { |f| cache_file = f if version_from_file(f) >= version_from_file(cache_file) }
-          # otherwise just use the latest supplied
-          else
-            dg.each { |f| cache_file = f if version_from_file(f) >= version_from_file(cache_file) }
+      unless @force_update
+        # If the user provides their own cache
+        if File.directory?(@manual_vendored_dir)
+          # Check for any '<component>-puppet_agent-<puppet version>.json' files
+          dg = Dir.glob(File.join(@manual_vendored_dir, "#{component}-puppet_agent*"))
+          # Check if there are multiple versions of the component cache
+          if dg.size > 1
+            # If there is the same version supplied as whats being tested against use that
+            if dg.any? { |s| s[desired_name] }
+              cache_file = File.join(@manual_vendored_dir, desired_name)
+            # If there are any with the same major version, use the latest supplied
+            elsif dg.any? { |s| s["#{component}-puppet_agent-#{@puppet_major_version}"] }
+              maj_match = dg.select { |f| /#{component}-puppet_agent-#{@puppet_major_version}.\d+\.\d+\.json/.match(f) }
+              maj_match.each do |f|
+                if (version_from_file(cache_file) == version_from_file(desired_name)) || (version_from_file(f) >= version_from_file(cache_file))
+                  # if the current cache version matches the desired version, use the first matching major version in user cache
+                  # if there are multiple major version matches in user cache, use the latest
+                  cache_file = f
+                end
+              end
+            # Otherwise just use the latest supplied
+            else
+              dg.each { |f| cache_file = f if version_from_file(f) >= version_from_file(cache_file) }
+            end
+          # If there is only one use that
+          elsif dg.size == 1
+            cache_file = dg[0]
           end
-        # if there is only one use that
-        elsif dg.size == 1
-          cache_file = dg[0]
         end
       end
 
@@ -110,15 +120,15 @@ def version_from_file(cache_file)
       Gem::Version.new(version_regex.match(cache_file)[1])
     end
 
-    # currently expects to be passed a R10K::Puppetfile object.
+    # Currently expects to be passed a R10K::Puppetfile object.
     # ex: R10K::ModuleLoader::Puppetfile.new(basedir: '.')
     def puppetfile_missing_vendored(puppetfile)
       puppetfile.load
       @vendored_references.each do |mod|
-        # extract name and slug from url
+        # Extract name and slug from url
         mod_slug = mod['url'].match(/.*(puppetlabs-\w+)\.git/)[1]
         mod_name = mod_slug.match(/^puppetlabs-(\w+)$/)[1]
-        # array of modules whos names match
+        # Array of modules whos names match
         existing = puppetfile.modules.select { |e_mod| e_mod.name == mod_name }
         if existing.empty?
           # Change url to https instead of ssh to allow anonymous git clones
@@ -132,7 +142,7 @@ def puppetfile_missing_vendored(puppetfile)
       end
     end
 
-    # return json from a query whom caches, or from the cache to avoid spamming github
+    # Return json from a query whom caches, or from the cache to avoid spamming github
     def query_or_cache(url, params, filepath)
       if (File.exist? filepath) && (@force_update == false)
         logger.debug "Using cache: #{filepath}"
@@ -146,7 +156,7 @@ def query_or_cache(url, params, filepath)
       json
     end
 
-    # given a github url and optional query parameters, return the parsed json body
+    # Given a github url and optional query parameters, return the parsed json body
     def github_get(url, params)
       uri = URI.parse(url)
       uri.query = URI.encode_www_form(params) if params
@@ -168,12 +178,12 @@ def github_get(url, params)
       end
     end
 
-    # returns parsed json of file
+    # Returns parsed json of file
     def read_json_dump(filepath)
       MultiJson.load(File.read(filepath))
     end
 
-    # writes json to a file
+    # Writes json to a file
     def write_json_dump(filepath, json_data)
       File.write(filepath, MultiJson.dump(json_data))
     end