Treat Pants like a Pants repo

[thejcannon]

What

I'd like to propose we treat the Pants repo itself like any other Pants repo.

What this boils down to is that we don't delegate to a ./pants bootstrap script here in the repo for running Pants itself, but rather use the latest git tag (e.g. release_2.18.0a0) to determine which off-the-shelf Pants to run. This makes the Pants repo not unlike a compiler, where an older version of the code is used to produce the next version.

Additionally, the engine compilation is bypassed and scie-pants will copy it's own already-compiled engine into the source tree to be used by the Pants-code-as-user-code Pants (at least until Pants can compile Rust binaries 😉)

(As a follow-up I'll likely propose we use a Python Build Standlone Python provider for this repo as well.)

Why

• Faster development in the common case: Not having to recompile the engine when upgrading main or switching branches would be a humongous dev lift.
• Faster release process: Our post-release steps end up re-compiling the engine which we literally just built. Once those are built and put on a shelf, we should be able to use them immediately.

(those two reasons are also abundantly clear when you want to do something extremely simple like pants run build-support/bin/foo.py)

• Yummy yummy dogfood: We should be able to run Pants on the Pants repo. We should walk a mile in our users shoes (which we already do to some degree).
• Gets us over the last hurdle of "one-button-releasing". Right now Pants in-repo requires loading the VERSION, which would ideally be replaced with a vcs_version target.

How

What this means in practice is:

• We separate our plugins into a different resolve, and those will use the pants_requirements macro to find references to pants modules
  • This means any changes required to both the plugin and the core Pants code must be done piecemeal. Change Pants -> release -> update plugins. This is how our out-of-repo plugins make changes, so we should walk a mile in their shoes.
• Changes to the engine are more nuanced. Since Pants is running the repo, we should be able to hide the engine compilation behind Pants target/targets (e.g. adhoc_ or shell_ tooling). This doesn't mean we avoid compilation altogether, but does mean we gain caching of the compilation such that rewinding/fast-forwarding doesn't cost another compilation. Additionally, CI benefits from a remote cache.
  • We could also do all sorts of fun tricks to bypass compilation at the cost of complexity by grabbing a pre-built engine, if compatible. This could be done by extracting the engine from a GitHub Release Asset wheel, bullying scie-pants into doing this for us, or something else.

[stuhood]

Thanks for the proposal: I'm generally interested, as I do think that it will accelerate a common case. But...

The same is done for engine changes. If you need to change the engine and consume it on the Python side, it must be done across a release boundary

• There'll be a way to forcibly compile the binary to engine changes locally

This feels like a non-starter. We only want to be doing more in the native code moving forward, and this change would put a massive tax on that development.

---

One way to potentially pull this off (which I've utilized in the past) would be to have a "mode" toggle between "binary engine mode" and "source engine mode", where the latter triggered a source build of the engine as we do today. That would allow for periods on main where new Rust code was being used before it had been released.

Another way would be to decouple the engine release process from the release process for the rest of Pants, so if you wanted to consume a new Rust change from Python, you'd have to run a step to "release it" somewhere (possibly locally or to a staging destination, rather than necessarily publicly). I've seen this strategy used on the JVM side and it is ... ok. Definitely increases your cycle time.

[thejcannon]

OK, had a little think on this one. I edited it to say that the engine compilation can probably be accomplished through the "codegen-y" support targets we now offer (or any number of in-repo plugins as well) since we already have a full Pants instance which can execute the rules for those targets (whereas we didn't before).

This is actually even better than I thought because it means we leverage the local/remote caches.

[huonw]

The "probably" is even stronger: @chrisjrn has a prototype of actually doing it via #18688!

[thejcannon]

What a find! Thanks @chrisjrn I'll see if this can be used for my own nefarious purposes

[chrisjrn]

And it mostly worked, too!

…

On Wed, 6 Sept 2023, 17:22 Huon Wilson, ***@***.***> wrote: The "probably" is even stronger: @chrisjrn <https://github.com/chrisjrn> has a prototype of actually doing it via #18688 <#18688>! — Reply to this email directly, view it on GitHub <#19774 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEX5XDQ447XI76LAWC5KXTXZDY77ANCNFSM6AAAAAA4NXLA2A> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[huonw]

Woohoo, bootstrapping! I think it'll be great to separate "artefact that is being produced by the repo" from "tool that is used to orchestrate the builds" (which both just happen to be called "pants"). It'll be great for working in the Pants repo: so many spurious pantsd restarts at the moment!

Relevant previous discussion: https://pantsbuild.slack.com/archives/C0D7TNJHL/p1666375916738029, #18688.

Bootstrapping comes with a circular dependency that will need to be considered: how do we build pants for a new platform, e.g. if we got windows working (#11170), naively, we'd need to be able to run the orchestrating-Pants on Windows in order to build the windows artefact-Pants, and that doesn't work for the very first one (there's no orchestrating-Pants that works on Windows) 🤯

There's a few ways we could solve this for getting the first artefact:

1. "Manually": ensure that we can build native code outside of pants and then some basic way to turn that into a working PEX, potentially ./cargo build followed by a PEX CLI invocation.
   • this means ./cargo build to get the engine etc. still works, and I think this is desirable anyway, at least in the short term, for a better dev-experience when working on the Rust code directly
   • the PEX doesn't have to be "perfect": it is fine to contain the whole src/python/pants directory (all the tests and BUILD files unnecessarily etc), it just needs to be functional
2. "Reimplementation": have cross-platform cut-down version of enough of Pants to build Pants itself. This can cut a lot of corners:
   • minimal error checking: we can assume the Pants repo is valid because "real" Pants checks it on other platforms
   • slow: no caching, no concept of rules etc.
   • minimal targets and goals:
     • support only pants package, and only for the actual target used by the Pants repo for the PEX, e.g. potentially only python_sources, pex_binary and adhoc_tool?
     • no linting, formatting etc.
   • minimal dependency inference: potentially even just static output of pants peek :: run with real pants on a different platform

I'd suggest 1 be our approach for now, as 2 sounds like it'd be hard to maintain reliably.

What this boils down to is that we don't delegate to a ./pants bootstrap script here in the repo for running Pants itself, but rather use the latest git tag (e.g. release_2.18.0a0) to determine which off-the-shelf Pants to run. This makes the Pants repo not unlike a compiler, where an older version of the code is used to produce the next version.

I'd suggest a different policy: keep the pants repo as simple as possible and just use a normal pants.toml with a normal pants_version = "2.17.0" (or whatever), that is explicitly upgraded. That is, scie-pants wouldn't necessarily need special handling for the pants repo.

In terms of actual policy, "the pants repo uses the latest stable release" seems like a sensible one to me. That'll make the dogfooding better, and encourage us to keep up a regular release cycle, and a nice deprecation/upgrade experience.

(Another option would be to do "proper" bootstrapping, like a compiler where there's a binary "stage0" compiler that's used to build stage1 from the compiler sources, and then that stage1 builds stage2 from the same sources, i.e. ensure that the compiler can compile itself, https://rustc-dev-guide.rust-lang.org/building/bootstrapping.html#stages-of-bootstrapping. I'd suggest that this isn't worth it for Pants.)

Yummy yummy dogfood: We should be able to run Pants on the Pants repo. We should walk a mile in our users shoes (which we already do to some degree).

Yeah, I think this is a great driver: if something is annoying in the Pants repo, it's likely to be annoying for users too, and things like "set up a developer environment" (e.g. your PBS idea) are good examples of this. (I view PRs like #19586 as good interim steps, but also revealing features pants should have.)

[thejcannon]

Agreed on 1. Some amount of manual bootstrapping for newer platforms for sure. Depending on Pants own complexity at that point, we'd want to write down what those steps are. Bit of a chicken-and-egg there, but I'm sure we'll figure it out 😉

[thejcannon]

And part of why I like the "latest tag" for the version is because I envision that maybe for most commits, the compatibility of the latest tag's engine and the code's is that copying the engine into the source tree would mean bypassing compilation altogether.

Our wheels have the engine and their recompilation metadata embedded, so it should be safe and easy to check if the copy would be beneficial or not.

---

Using latest tag also has the benefit of dogfooding a little better. E.g. today we're testing bleeding edge Pants on every run. That's a lot of dogfood. Having all of our development gap by several versions means those versions see MUCH less testing.

I'm happy to be outvoted on this one though. Any solution along this path is an improvement.

[huonw]

Using latest tag also has the benefit of dogfooding a little better. E.g. today we're testing bleeding edge Pants on every run. That's a lot of dogfood. Having all of our development gap by several versions means those versions see MUCH less testing.

Yeah, I'm convinced. I'd still suggest we don't do it implicitly, but have it explicit in pants.toml: if it's implicit, we could get jammed, if there's a bug in a release that blocks pants development, we'd need to build a fix to allow us to build the fix. (For example, as part of the release process, if it's on main, have @WorkerPants submit a PR that bumps pants_version and let that go through CI at least.)

That raises another detail to consider: what version do the 2.17.x (etc) release branches use?

And part of why I like the "latest tag" for the version is because I envision that maybe for most commits, the compatibility of the latest tag's engine and the code's is that copying the engine into the source tree would mean bypassing compilation altogether.

(I'm not sure if you're thinking this way, but I would think that the total changes since the orchestrating-Pants matter, not just the current commit. So if there's been any change to the artefact-Pant's engine code since the orchestrating-Pants, it's likely that the engine won't be compatible.)

[thejcannon]

I'm really hoping to avoid any PRs required before/during/after a release, so us bumping the version in pants.toml goes against that.

It also opens the door for the possibility of drift, if the PR doesn't get merged in a timely manner (or doesn't get created in the first place).

---

I understand that any amount of rust changes would make the engine-copy shortcut moot, but all the commits between the tagged release and the rust change would benefit from the shortcut.

That means even if every time we release, even if the next change is a rust change, the shortcut wouldn't help. And that's what we do today. So at best, we have N commits that get a multi-minute boost. At worst, we do what we're doing today with some added complexity.

[huonw]

I'm really hoping to avoid any PRs required before/during/after a release, so us bumping the version in pants.toml goes against that.

It also opens the door for the possibility of drift, if the PR doesn't get merged in a timely manner (or doesn't get created in the first place).

Sure, it's annoying, but it can definitely be created automatically. The consequence of drifting doesn't seem so bad other than slightly reducing how much dogfooding that particular release gets.

What do you think about the failure modes of doing it implicitly?