Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unexpected coverage calculation after adding specs #1120

Open
tgentry-bamboo opened this issue Dec 9, 2024 · 0 comments
Open

Unexpected coverage calculation after adding specs #1120

tgentry-bamboo opened this issue Dec 9, 2024 · 0 comments

Comments

@tgentry-bamboo
Copy link

tgentry-bamboo commented Dec 9, 2024

Hello, I'm working on a fairly large Rails application utilizing SimpleCov for test coverage reporting.

We are using SimpleCov version 0.22 and Rails 6.1.7.7.

Here is the SimpleCov configuration we are using as seen in our config/spec_helper.rb file:

SimpleCov.start :rails do
    add_filter ['/vendor/', '/lib/pmp_credentials/']
    command_name  "RSpec #{rand(1000000)}"
    merge_timeout 3600   # increase default because suite takes long time
  end

We are also using a .simplecov configuration file whose contents are such:

class LineFilter < SimpleCov::Filter
  def matches?(source_file)
    source_file.lines.count < filter_argument
  end
end

SimpleCov.start :rails do
  use_merging true
  merge_timeout 3600
  add_filter '/.gem/'
  add_filter '/vendor/'
  add_filter '/config/'
  add_filter '/lib/tasks/'
  add_filter '/script/'
  add_filter '/lib/*.rb'
  add_group 'Services', 'app/services'
  add_group 'Workers', 'app/workers'
  add_group 'Trailblazer', 'app/concepts'
  add_group 'Long files' do |src_file|
    src_file.lines.count > 100
  end
  add_group "Short files", LineFilter.new(5) # Using the LineFilter class defined in Filters section above
end

SimpleCov.minimum_coverage 80 if ENV['CHECK_COVERAGE']

The problem we are experiencing is that after adding ~100 new specs and making additions to an existing FactoryBot factory, using ~1000 new lines of spec code to do so, the coverage calculation has changed dropping us below our minimum coverage enforcement by 10%.

Here are the SImpleCov calculations before adding the new specs:

All Files ( 80.34% covered at 2594.63 hits/line )
1803 files in total.
52115 relevant lines, 41867 lines covered and 10248 lines missed. ( 80.34% )

and here are the SimpleCov calculations after adding the new specs:

All Files ( 71.91% covered at 3880.83 hits/line )
1803 files in total.
53406 relevant lines, 38406 lines covered and 15000 lines missed. ( 71.91% )

No new app code was added in this time, only the spec code mentioned previously. The amount of files has not changed but SimpleCov is picking up 1291 new lines of code and 3461 less lines covered with the simple addition of test code. This for us is unexpected behavior. I've searched for similar issues but was unable to find any. Any ideas off the top of the head for what the underlying issue may be? I'm willing to devote some time digging into SimpleCov's source code should the issue be an actual bug in the library but will need some guidance.

Thanks for all your hard work!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant