test_frameworks profile hides source directories with names like 'spec' or 'features'

[dmolesUC]

Summary

I'm the author of a gem called ruby-marc-spec that for domain-nomenclature reasons, has a main module called MARC::Spec, with source in a corresponding directory lib/marc/spec.

I recently noticed that SimpleCov was only covering the root file, lib/marc/spec.rb, and ignoring all the files under lib/marc/spec.

Steps to reproduce

1. Clone the linked repo and check out tag 0.1.1.
2. In the project root, run bundle install.
3. Run bundle exec rake coverage.

Expected

• Coverage task fails with Coverage (99.87%) is below the expected minimum coverage (100.00%).
• Generated HTML report in artifacts/rcov includes all source files under lib, including lib/marc/spec/queries/query_executor.rb, which has an uncovered line 47.

Actual

• Coverage task suceeds.
• Generated HTML report shows only lib/marc/spec.rb, with 100% coverage.

Discussion

The issue is with the filters in test_frameworks.rb, plain strings which each produce a StringFilter that will check for the presence of the substring anywhere in the file path. A regex filter, e.g. %r{^/spec/}, would instead exclude these directories only at the root of the project, which seems to be the intent.

(I also wonder about the similar bundler_filter.rb, but that one seems less likely to produce accidental false positives.)

Workarounds

In my case, the test_frameworks profile was being pulled in by the rails profile, which I didn't need (lazy cut-and-paste from a Rails project). I dropped back to the default profile and added a custom regex filter for the /spec directory to my .simplecov:

require 'simplecov-rcov'

SimpleCov.start do
  add_filter %r{^/spec/}
  add_filter 'module_info.rb'

  coverage_dir 'artifacts'
  formatter SimpleCov::Formatter::RcovFormatter
  minimum_coverage 100
end

Another workaround for a Rails user would be to explicitly remove the "/spec/" filter (or whichever other problem filter):

SimpleCov.start 'rails' do
  filters.reject! { |f| f.filter_argument == '/spec/' }
  add_filter %r{^/spec/}

  # ...etc.
end

System info

• SimpleCov version: 0.16.1 (also tested with 0.21.2, with same results)
• Ruby version: ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [arm64-darwin21]
• OS: macOS 12.2 (Monterey)