Should we expose internal (python) tools as targets?

[thejcannon]

OK my last big request/idea for 2.15.x I promise (but also not really 😉)
Sometimes I find myself wanting to run a tool which Pants is currently wrapping.

• pex - For hopefully obvious reasons.
• also tools like black or pytest or similar
  I could of course declare a pex_binary and run that but it requires me ensuring the version and configuration is similar to that of Pants (boo double-config).

One big reason for this that I find myself wanting is knowing what CLI args are possible (especially for pytest). There's no easy way invoke pytest --help and get all the CLI knobs/levers including third-party and first-party ones. A second reason is wanting to run pex lock ... using the exact same version of pex Pants is (See other discussion)

So what would it look like to have explicit/implicit targets for each tool? Then I could run them. AND export wouldn't be exporting them by default if they had addresses 😉

Thoughts? My bikeshed name was interal_tool or more specifically internal_python_tool (to mirror external_tool from #17277)

[Eric-Arellano]

Thoughts on what the address might look like? Some prefix like __pants_tools:black? This brave new world with synthetic target addresses still makes me nervous, e.g. #17343

[thejcannon]

FWIW I'm OK with this being an explicit target too. Especially since we expect users to interact with it.

[thejcannon]

In which case, we don't have to worry about the address, and users can put these where they so please.

[benjyw]

Re the export part of this, I think having export take specs was (my) mistake. What we really wanted was, e.g.,

./pants export --resolve=pytest --resolve=mypy

specs don't really make sense for export in general (they may for some specific backend's export logic, but then they should be provided via option to that backend).

Plugin options for subsytems makes this possible at last!

[thejcannon]

Yeah. Makes sense.

[Eric-Arellano]

Strong +1 to this. I think it's partially because export happened before resolves were stable. Regardless of this discussion, it would be great to fix this.

[benjyw]

So anyway, exposing tools as targets may have benefits, but this is not how I want to solve the "export is currently dumb" problem.

However, maybe instead of exposing tools as (synthetic) targets, we should allow (real) targets to be tools:

Today a tool is a set of requirements and the lockfile generated from them, and those requirements are in pants.toml. Yet we also have mechanisms for generating a (user) lockfile from targets, so why not lean into that? It seems arbitrary that some requirements go in pants.toml and others go in BUILD files.

If, instead, the way you customized tools was to point the logical name, e.g., pytest to a target, then:
A) You no longer have the version sync problem, where you also have to have pytest in your requirements.txt so you can import it
B) We can lose a lot of the special-casing for tools vs user lockfiles.
C) We present a more consistent interface to the user
D) The targets in question need not by python_requirements. They can be anything that builds a dist or a pex. This opens the door to building tools from in-repo code and then using them, all in a uniform way.

So basically, this: #12449

This is how it worked in v1 by the way.