-
-
Notifications
You must be signed in to change notification settings - Fork 641
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Scoverage work - add scoverage to OSS pants #8064
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Only looked at where you can use Python 3. Also did this from my phone so please message me on Slack if any of this is confusing.
# Copyright 2015 Pants project contributors (see CONTRIBUTORS.md). | ||
# Licensed under the Apache License, Version 2.0 (see LICENSE). | ||
|
||
from __future__ import absolute_import, division, print_function, unicode_literals |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don’t need this anymore.
@@ -0,0 +1,94 @@ | |||
# coding=utf-8 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don’t need this because we don’t care about Python 2 support.
src/python/pants/backend/jvm/subsystems/scala_coverage_platform.py
Outdated
Show resolved
Hide resolved
@property | ||
def injectables_spec_mapping(self): | ||
return { | ||
'scoverage': ['{}'.format(self.get_options().scoverage_target_path)], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can use an f string
`f”{self.get_options...}”
""" | ||
Adds 'scoverage' to scalac_plugins in case scoverage is enabled for that [target]. | ||
:return: modified scalac_plugins | ||
:rtype: list of strings |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rather than putting this in docstring, can use a type hint. On line 40 put -> List[str]
between the param list and colon. See strutil.py for an example.
You’ll also need to add from typing import List
""" | ||
Adds 'scoverage' to scalac_plugins_args in case scoverage is enabled for that [target]. | ||
:return: modified scalac_plugins_args | ||
:rtype: map from string to list of strings. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Return type is Dict[str, List[str]]
scalac_plugin_args = target.payload.scalac_plugin_args | ||
if scalac_plugin_args: | ||
scalac_plugin_args.update( | ||
{"scoverage": ["writeToClasspath:true", "dataDir:{}".format(target.identifier)]}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
F strings here and a few lines below
Thanks @Eric-Arellano 💯 for python3 help. The latest commit removes scoverage target from BUILD.tools and creates an "injectable" for the same in ScalaCoveragePlatform subsystem. |
d11fd6c
to
fc48b64
Compare
…m.py Co-Authored-By: Eric Arellano <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! Looks like a great start.
I'd like to see (at least) a draft of the second half of the change before we land this though.
3rdparty/jvm/org/scoverage/BUILD
Outdated
@@ -0,0 +1,3 @@ | |||
target( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe that this target can be removed now.
pants.ini
Outdated
@@ -220,6 +220,13 @@ no_warning_args: [ | |||
'-S-nowarn', | |||
] | |||
|
|||
compiler_option_sets_enabled_args: { | |||
# Needed to make scoverage CodeGrid highlighting work | |||
'scoverage': [ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rather than having to manually include this, we should automatically include it when scoverage
is enabled. Probably somewhere around here?
pants/src/python/pants/backend/jvm/tasks/jvm_compile/zinc/zinc_compile.py
Lines 324 to 325 in 4435b8e
compiler_option_sets_args = self.get_merged_args_for_compiler_option_sets(compiler_option_sets) | |
zinc_args.extend(compiler_option_sets_args) |
blacklist_file = 'new_blacklist_scoverage' | ||
|
||
|
||
class ScalaCoveragePlatform(InjectablesMixin, Subsystem): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Naming nit: rather than calling this ScalaCoverage
, we should probably call it Scoverage
, as that seems to be the project's actual name.
class ScalaCoveragePlatform(InjectablesMixin, Subsystem): | ||
"""The scala coverage platform.""" | ||
|
||
options_scope = 'scala-coverage' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ditto naming.
register('--enable-scoverage', | ||
default=False, | ||
type=bool, | ||
help='Specifies whether to generate scoverage reports for scala test targets.' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should be consumed from this subsystem instead:
pants/src/python/pants/backend/jvm/tasks/coverage/manager.py
Lines 73 to 74 in 4435b8e
register('--coverage-open', type=bool, fingerprint=True, | |
help='Open the generated HTML coverage report in a browser. Implies --coverage.') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If I understand correctly, manager.py
is initialized by junit_run.py
at runtime only. However, I also need this option (enable-scoverage
) at compile time to determine whether to instrument the targets or not. That is why I kept it here. Am I thinking about this right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I see what is happening here: Even though the CodeCoverage subsystem gets initialized, none of its options are registered. This is because the subsystem is using def register_junit_options(register, register_jvm_tool):
instead of def register_options(cls, register):
while the options under the latter (register_options
) gets automatically registered, the former (register_junit_options
) requires to be called by some other code, e.g, this is how it is called in junit_run.py
:
pants/src/python/pants/backend/jvm/tasks/junit_run.py
Lines 99 to 102 in 35be511
# TODO(jtrobec): Remove direct register when coverage steps are moved to their own subsystem. | |
CodeCoverage.register_junit_options(register, cls.register_jvm_tool) | |
And when it is called by some other class,( junit
in this example), the scope of that class gets prepended to the scope of coverage
, thus registering the options as --test-junit-coverage
Consequently, setting --scoverage-coverage=True
(scope at compile time) does not automatically set --test-junit-coverage=True
(scope at runtime) (and vice-versa)
:return: See constructor. | ||
:rtype: list of strings. | ||
""" | ||
if self._scoverage: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rather than fully overriding the set of plugins, this should merge both lists. Ditto the plugin args and options.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the function get_scalac_plugins
in scoverage_platform.py does that. It merges the list if initially non empty (otherwise creates a new one) Similarly, for plugin args and options, the merging happens inside the scoverage subsystem.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok. That's a bit confusing, because if someone were to change line 102 here, they would need to remember to change the subsystem code as well. IMO, the merging code should live in here to avoid that issue.
Scoverage (2/2) - Adding report generator Files Changesbackend/jvm/tasks/coverage/manager.py: Adding support for scoverage New Files Addedbackend/jvm/tasks/coverage/scoverage.py: scoverage runtime subsystem |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! This looks really good.
As mentioned below: please break out a separate PR for the scoverage report generation JVM code. We'll have to land that first.
:return: See constructor. | ||
:rtype: list of strings. | ||
""" | ||
if self._scoverage: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok. That's a bit confusing, because if someone were to change line 102 here, they would need to remember to change the subsystem code as well. IMO, the merging code should live in here to avoid that issue.
if self._settings.coverage_open: | ||
return os.path.join(base_report_dir, 'html', 'index.html') | ||
|
||
def _execute_scoverage_report_gen(self, measurements_dir, report_dir, target_filter): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Another PR will need to be broken out to land the report generation JVM code so that we can publish it to maven central and then consume it here as a register_jvm_tool
.
dependencies = [ | ||
'src/python/pants/option', | ||
'src/python/pants/subsystem', | ||
'src/python/pants/java/jar', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: extra space?
compiler_option_sets = [SCOVERAGE] | ||
return tuple(compiler_option_sets) | ||
|
||
def is_blacklisted(self, target) -> bool: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The blacklist should be cached in-memory, currently the file is read per target.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you switch this to a list option (as above) that will be handled automatically.
f'"{report_dir}"]', | ||
] | ||
|
||
# if target_filter: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
dead code? can be removed
src/scala/org/pantsbuild/scoverage/report/ScoverageReport.scala
Outdated
Show resolved
Hide resolved
…blacklisted files
…nto scoverageWork
Scoverage Outline Major files added
Major files changed
Other changes
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
default=False, | ||
type=bool, | ||
help='Specifies whether to generate scoverage reports for scala test targets.' | ||
'Default value is False. If True,' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These lines need spaces in them in order to avoid running together.
'Default value is False. If True,' | ||
'implies --test-junit-coverage-processor=scoverage.') | ||
|
||
register('--blacklist-file', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pants supports @file
options natively, so you should make this option a list option:
register('--blacklist-targets',
type=list,
member_type=target_option,
..
)
And then someone who wants to put the list of targets in a file (in python list syntax) can pass --blacklist-targets=@some-blacklist-file
.
You could also pass in a (string) list of regexes to match against target names.
compiler_option_sets = [SCOVERAGE] | ||
return tuple(compiler_option_sets) | ||
|
||
def is_blacklisted(self, target) -> bool: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you switch this to a list option (as above) that will be handled automatically.
return False | ||
|
||
if target.address.spec in self._blacklist_file_contents: | ||
logger.warning(f"{target.address.spec} found in blacklist, not instrumented.") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should probably be at debug
or info
rather than warn
.
@@ -47,6 +51,8 @@ def __init__(self, java_sources=None, payload=None, **kwargs): | |||
}) | |||
super().__init__(payload=payload, **kwargs) | |||
|
|||
self._scoverage_instance = ScoveragePlatform.global_instance() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The target itself doesn't need to hold a reference to _scoverage_instance
: can get it lazily from the singleton.
] | ||
) | ||
|
||
register('--target-filters', type=list, default=[], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This feels redundant with the ScoveragePlatform.blacklist option. Do we need both?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This target filtering is for report generation, i.e helps in generating reports only for the specified targets. The blacklist
option is more for overcoming the issue that if some targets cannot be instrumented at all (due to exceeding JVM code size limits), they should be passed into the blacklist.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this option helps in preventing the need to compile again if the report is needed only for a handful of targets, i.e, say I compiled 500 targets and got the report generated. But now I need a report for all targets containing the class "merlin"(~ 10 targets), or all targets in package "report", then I can get those reports without needing to compile everything again.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hm. But in reality, are people going to know that? From a user's perspective, having just one option would likely be easier to understand.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed. People just need to use this target-filter
option most of the time. In case they encounter this error of JVM code size limits, they can overcome that with the help of blacklist
option as well. More like a Troubleshooting mechanism. I'll make sure to document this.
def slf4j_jar(name): | ||
return JarDependency(org='org.slf4j', name=name, rev='1.7.5') | ||
|
||
def scopt_jar(): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is only used one place, so maybe just inline it.
def _iter_datafiles(self, output_dir): | ||
for root, _, files in safe_walk(output_dir): | ||
for f in files: | ||
if f.startswith("scoverage"): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should move these comments into python method docs:
def _iter_datafiles(self, output_dir):
"""This is a docstring for a method.
With some other stuff in it.
"""
..
@@ -322,6 +322,11 @@ def relative_to_exec_root(path): | |||
zinc_args.append('-transactional') | |||
|
|||
compiler_option_sets_args = self.get_merged_args_for_compiler_option_sets(compiler_option_sets) | |||
|
|||
# Needed to make scoverage CodeGrid highlighting work |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should be incorporated in ScalaLibrary
rather than here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How can I set compiler_option_set_enabled_args
in ScalaLibrary. I looked at the compiler_option_sets_mixin.py
here : https://github.com/pantsbuild/pants/blob/14e0bd6b27910310e225ff9586c2410459153fd9/src/python/pants/option/compiler_option_sets_mixin.py But there is no API for adding extra arguments. That is I why added it manually here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, fine with this as is I guess.
help='Path to files containing targets not to be instrumented.') | ||
|
||
register('--scoverage-target-path', | ||
default='//:scoverage', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks like this target is bootstrapped by default. If a user overrides this, the default //:scoverage target is still injected into the build graph.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks Sameer for doing this awesome work, based off my previous review and assuming you address Stu's concerns/changes. Everything looks good to me.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great! Thanks.
else: | ||
compiler_option_sets = [SCOVERAGE] | ||
|
||
super().__init__(payload=payload, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Having two super calls is pretty weird, and in general this has resulted in a lot of nesting.
An alternative might be to do something like:
scalac_plugins = ... or []
scalac_plugin_args = ... or []
compiler_option_sets = ... or []
if ScoveragePlatform.global_instance().get_options().enable_scoverage:
scalac_plugins.append(..)
scalac_plugin_args.update(..)
compiler_option_sets.update(..)
super(...)
] | ||
) | ||
|
||
register('--target-filters', type=list, default=[], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hm. But in reality, are people going to know that? From a user's perspective, having just one option would likely be easier to understand.
@@ -322,6 +322,11 @@ def relative_to_exec_root(path): | |||
zinc_args.append('-transactional') | |||
|
|||
compiler_option_sets_args = self.get_merged_args_for_compiler_option_sets(compiler_option_sets) | |||
|
|||
# Needed to make scoverage CodeGrid highlighting work |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, fine with this as is I guess.
@sammy-1234: re: the lint errors: are you using the commit hooks? https://www.pantsbuild.org/howto_contribute.html#getting-pants-source-code |
Thanks for that. But even then, it is still giving me this weird error: and then failing with:
Any clue how I can resolve this? Thanks :) |
@sammy-1234 : If you have an |
Thanks @stuhood . The hooks pass now 💯 |
Problem
This PR is in response to issue #8026.
Solution
It attempts to add scoverge by creating a new subsystem scoverage_platform.py and modifies the scala_library target according to the subsystem in case scoverage is enabled.
Furthermore, it adds a scoverage.py runtime support required for report generation.
Result
Users can run scoverage by running the following command:
./pants --scoverage-enable-scoverage=True test path/to/<scala-target>
Tests
I have added tests for the new subsystem in test_scoverage_platform.py. Run them as follows:
./pants test tests/python/pants_test/backend/jvm/subsystems:test_scoverage_platform
Files changed
Please take a look at my comment below for a list of all the modified files.