A generic "generate" goal for version-controlled generated files

[cognifloyd]

For a variety of reasons (many of which could probably be summarized as "legacy"), some generated files need to be committed to version control.

Since these files need to be in version control, it would be great if pants could make it easier to manage them. Here are the goals I have for whatever solution we use:

• Generated files should be up-to-date.
• If the files are out-of-date (eg one of the inputs used to generate it has changed), then the user needs an error or warning with an explanation of how to re-generate the files.

A survey of current pants goals

goal: export-codegen

At first blush, codegen seems ideal for making generated files. But codegen in pants is not designed for version controlled files.

When pants generates code, it is considered an internal by-product as described in the PR that introduced the export-codegen goal:

It's important that we write to dist/, rather than saving to the build root, as these are generated files that should be kept out of version control. Also, we'd have major issues if Pants starting thinking those generated files were actual production files, e.g. a [target]'s default sources field claiming ownership of those files.

The export-codegen goal makes these generated files available in dist/ for those who want to:

use the results of codegen outside of Pants, including:

1. Inspect the files for debugging purposes.
2. Save the files for IDEs to use.

So, we need a different form of codegen in pants for files that should be version controlled.

goals: lint, fmt, fix

Version controlled files are the bread and butter of lint, fmt, and fix which is opposite of export-codegen which is for files that are not version controlled.

Fixers and formatters also run during the lint goal so that lint fails if they need to fix or format a file.

Code generation of files that are version controlled is remarkably similar to fmt and fix. Files that need to be changed are advertised via the lint goal. Logically, fmt and fix are very different than codegen - but the process, the UX, and the underlying rules seem quite similar to me.

In the StackStorm project, I've shoved my (version controlled) file generation under the fmt umbrella even though it's not really a formatter. Logically this generation is also not a fixer. But, one of the benefits of doing this is that: lint will fail if the generator needs to regenerate the file(s) and it will provide an error message that explains how to run ./pants fmt ... to resolve the lint error.

I've heard of others that create a "test" that fails if the generated file(s) were not properly regenerated. I really like using lint for this to take advantage of an the fine grained caching and reusing the cache for both lint and fmt/fix.

goal: generate-lockfiles

Another goal that is very similar to the file generation feature I'm looking for is the generate-lockfiles goal. The warnings/errors saying to regenerate can be triggered in many places where rules need to use the lockfiles. So, there is no need to integrate that warning in a lint backend/rule. The goal also provides a fairly ergonomic interface for regenerating those.

generate-lockfiles looks up the resolves, translates them into lockfile paths, and then generates the lockfile at that path.

Proposal for a new "generate" goal

If there were a generic "generate" goal, it could go in the opposite direction of generate-lockfiles: given the path of a lockfile, look up the resolve associated with that lockfile, and then generate the lockfile at that path.

Similarly, that "generate" goal could take a path to another generated file, look up owning target for it, and if that target is explicitly for generated files, run the plugin/subsystem/tool that encompasses the codegen logic for that target.

This goal could be only for non-lockfile generated files. If we did include lockfile generation in this goal, then the owning target would probably be a synthetic target synthesized from the config in pants.toml -- so it is explicitly defined (even though not defined in a BUILD file).

This goal should be modeled on fix/fmt (not modeled on export-codegen). This would extend the lint request/result classes (like fix/fmt do) so that file generation happens in a sandbox during lint, failing lint if something should be regenerated. When lint fails here, the error message would prompt the user to run ./pants generate path/to/generated/file/or/dir.

For a plugin to provide this generate rule, it would either have to add a custom target, or add a field to an existing target to allow flagging it as owning generated files. This would be a codified assumption built into the generate goal: all generated files must be owned by exactly 1 file generating target: If unowned, they will not be materialized in the workspace; If owned by more than one file generating targets, raise an error. Generating a file that matches a file glob for that target counts for this.

Then, if any other linters can work with a generated files the standard skip_tool fields provide an escape hatch if the generated code does not need to meet the same standards as other files. Or, for StackStorm, I can add an extra linter specific to the generated file that checks more than just whether or not it needs to be regenerated.

Goal: run

A lot of work has gone into runnable targets like adhoc_tool and shell_command. If combined with the generate goal, those could allow repos to use light-weight run targets and file generator targets instead of writing a medium- or heavy-weight pants plugins.

So, a runnable target could be tied to the file generating target such that the adhoc_tool or pex_binary is used to actually do the generation when a user does ./pants generate path/to/generated/file.

For the run goal, the UX focuses on the generator script/target. For this "generate" goal, the UX focuses on the files to generate. So they are complementary.

goal: go-generate

#16909

This seems very similar and could perhaps be rolled into the new "generate" goal. go-generate references the go modules that define the generation. I'm looking at something that targets the generated files. But, often go:generate commands work within the same module where they are defined, so maybe it is effectively the same thing.

But, this is something that would not run via Lint. So, that would probably be a go backend option to skip running for lint is the goals were combined.

discussion

Actually naming the new "generate" goal is a different discussion. When we discuss that, we'll need to make sure it can be distinguished from the codegen that already exists in pants.

So, would a generic generate goal be helpful for anyone else? Does a generate goal that follows a process similar to fix/fmt make sense for other projects?

[huonw]

Native ability to define simple targets that write back to the non-sandboxed filesystem (with built-in up-to-date-ness checking) would be great!

Some use-cases I've encountered:

• files that are read by external systems straight from the repo, without the ability to execute code. Examples: GitHub Actions workflows (e.g. Pants' build-support/bin/generate_github_workflows.py), dependabot configuration.
• "Snapshot" testing of critical derived data. For instance, the API schema derived from an API server checked in means any changes in the schema will be directly visible during code review, increasing the chance of catching issues like backwards incompatible changes.
• "Legacy"-ish of integrating with non-pants parts of a repo. For instance: exporting a schema from an (Python) API and doing code generations for (TypeScript) clients.
• Relatedly, even if the clients were fully integrated into pants, I'd be pretty tempted to keep checking in the generated clients, because my feeling is it generally integrates better with editors/IDEs etc. to have the source files just in the right place with all the other files, rather than needing to manually keep exporting and configuring the editors (but maybe we just haven't worked out our editors well enough).

---

Some small/bikeshed-y thoughts:

1. Subsuming generate-lockfiles seems like it'd be tricky, as I suspect you're aware:
   • The current behaviour of generate-lock files is to always update to the latest versions that satisfy constraints (e.g. Add ability to extend a PEX lockfile without modifying existing entries #15704), and it'd be very annoying if any run of pants generate :: did that too (I think that'd effectively mean pants generate :: would be unusable as is, and would always require filtering/exclusions/more specific targets)
   • Expanding on that, there's two meanings of "up to date" for lockfiles: (1) consistent with the declared requirements (a "local"/static property), and (2) using the latest versions satisfying the requirements (a "global"/mutable property: new versions of dependencies published to PyPI will break this check).
   • there's a few potentially related issues: New goal for managing 3rd party dependencies #12880, Rename generate-lockfiles to lock #15568, Add ability to extend a PEX lockfile without modifying existing entries #15704, Add ability to check if lockfiles are up-to-date with all declared requirements #15723 (maybe more...?)
2. It seems a bit weird to have so many goals that mutate files (fmt, fix, generate, and even run in some cases)?
3. Some generation processes might both be fairly heavyweight and have to trigger often. This means I'm not 100% sure running on lint would be a great user experience. It might be better to run on check or test (related to 2, there's a lot of goals that check files...).
   • For example, generating a schema from an API server may require starting that API, and pants will, correctly, rerun for any code change (even editing a comment in some tiny utility file that's a deep transitive dependency), even though most of them won't affect the schema.
   • allow users to assign tools to fmt and fix goals #18239 is related, since different generators might want to run at different "levels".
4. Allow using a shell command/adhoc tool as part of any goal (check, lint, fmt, fix, deploy, ...) #17729 is related, in that it's other thinking of how shell_command and/or adhoc_tool might mutate files directly.

[cognifloyd]

1. Subsuming generate-lockfiles seems like it'd be tricky, as I suspect you're aware:

1. Yes. I agree that subsuming generate-lockfiles would be very problematic. But, I found it helpful to think through the generation workflow to find similarities. There are enough differences that it should probably remain separate.

2. It seems a bit weird to have so many goals that mutate files (fmt, fix, generate, and even run in some cases)?

2. Yeah. That's frustrating.

   • I started out just reusing fmt for this, but the changes to support fmt/fix batching removed the ability for fmt plugins to add new files. It's also conceptually different. I think the saving grace here is that running lint tells you which goal to run if something can be fixed automatically.

   • When can run mutate files? adhoc_tools runs in a sandbox, and makes its output available via standard codegen (ephemeral non-version-controlled). I believe shell_command is similar.

   • If run can have un-sandboxed side effects, then perhaps we could use that instead, but people will have to copy/paste all the args when regenerating something which feels error prone. run feels too ad-hoc for that.

3. Some generation processes might both be fairly heavyweight and have to trigger often. This means I'm not 100% sure running on lint would be a great user experience. It might be better to run on check or test (related to 2, there's a lot of goals that check files...).

3. Oh. I forgot about check. I have only used it in the pants repo, so I don't think of it much. It's sounding more and more like the generated target will need some kind of knob to say whether the output should be subject to lint or check or nothing (kind of like how go-generate is not currently tied into lint right now).

4. Allow using a shell command/adhoc tool as part of any goal (check, lint, fmt, fix, deploy, ...) #17729 is related, in that it's other thinking of how shell_command and/or adhoc_tool might mutate files directly.

4. Yes these are very interrelated, but the current goals don't logically match with something that generates files. So, we need something else as well.

[huonw]

Yes. I agree that subsuming generate-lockfiles would be very problematic. But, I found it helpful to think through the generation workflow to find similarities. There are enough differences that it should probably remain separate.

👍

When can run mutate files? adhoc_tools runs in a sandbox, and makes its output available via standard codegen (ephemeral non-version-controlled). I believe shell_command is similar.

I believe something supporting run is executed with its working directory as the true repo root (although the sources/deps etc. are all in a sandbox) and thus has full access to mutate things. E.g. pants run whatever.py with the file below creates (or updates) last_run.txt.

# whatever.py
from datetime import datetime
import os
import sys

print(os.getcwd())
print(sys.path)

with open("last_run.txt", "w") as f:
    f.write(str(datetime.now()))

This seems to be how a generator like https://github.com/pantsbuild/pants/blob/99d56537fa8b9020c107ab347e9e6b3416fc52a8/build-support/bin/generate_github_workflows.py is recommended to be used.

(I think adhoc_tool and shell_command don't directly support the run goal.)

If run can have un-sandboxed side effects, then perhaps we could use that instead, but people will have to copy/paste all the args when regenerating something which feels error prone. run feels too ad-hoc for that.

Yeah, fully agreed that run isn't right. For instance, we have an alias like the following so one can just run pants quick-checks and get all the obvious stuff fixed/checked, and it'd be great if codegen tasks could happen as part of it too:

[cli.alias]
quick-checks = "tailor update-build-files fmt lint check ::"

A new generate goal as you've designed (or reusing the fix goal, or whatever) presumably can just be added, but run path/to/generator.py cannot (with pant's current behaviour: the target selection doesn't work out).

[cognifloyd]

Hmm. I guess my mental model around run and adhoc_tools wasn't quite accurate. Thank you for also confirming (err agreeing with me) that it isn't a good fit for code generation.

[huonw]

Just recording the workaround we use for this for now:

1. have the codegen as a normal shell_command or adhoc_tool or whatever target that generates the sources
2. a separate run_shell_command target that copies the generated files from the {chroot} sandbox to the right place on disk, which can be run like pants run path/to:target
3. a files (or resources) target for the files on diskl
4. a experimental_test_shell_command that diffs the generated files and the ones on disk, to confirm that the codegen matches what's checked in

E.g. for a file foo.txt something vaguely like: (can work for a directory too, with some variations like using rsync instead of cp and diff -r)

# path/to/BUILD

shell_command(
    name="generate-foo",
    command="echo abc > foo-generated.txt",
    tools=["echo"],
    output_files=["foo-generated.txt"],
)

files(name="foo", sources=["foo.txt"])
run_shell_command(
    name="write-foo",
    # different file name so that they can be compared in the test
    command="cp {chroot}/path/to/foo-generated.txt foo.txt",
    execution_dependencies=[":generate-foo"]
)

experimental_test_shell_command(
    # if this fails, run `pants run path/to:write-foo`
    name="check-foo-is-up-to-date",
    command="diff foo.txt foo-generated.txt",
    execution_dependencies=[":generate-foo", ":foo"],
)