Remote caching issues beyond pants 2.19

[basvank]

Hi all,

First and foremost, thanks for all the hard work on maintaining pants! Me and my team are happy users of pants, and it has been making our lives a lot easier for well over a year now!

That being said, I do currently have a little issue when upgrading pants. I’m not sure this is an issue in pants itself, or I’m overlooking something maybe, so for now I’m starting this as a discussion.

The problem is with remote caching, we use a pretty straightforward setup for it:

• pants with remote_cache_read and remote_cache_write set to true
• A locally running process of bazel-remote (version 2.4.4).
• bazel-remote points to an S3 bucket on AWS

This setup has been running for well over a year without any major issues whatsoever.

However, now I’m trying to update pants, but then issues arise with the caching. It looks like the cache isn’t used properly, as when I run the same CICD job multiple times the runtime doesn’t go down. This is something that did use to happen before, and of course would be expected as pants should be able to skip tasks that were already run on previous runs.

The issue arises whenever I update pants beyond 2.19.1 (I’ve tried up until 2.22.0). I do see notes regarding the caching system in the release notes for 2.20. It doesn’t look like it says anything about the reapi provider, which AFAIK this setup is using.

So, my first question is if anything changed at this point that should be adjusted or configured in my setup to keep this working?

If that is not the case, I can provide pants stats output and other details to show the difference between 2.19 and other versions. If there’s other configs or logs that could help trace this issue, please let me know as well!

[basvank]

It seems like I have finally found the cause for this issue. I'll leave it here as a reference for potential future readers.

So the issue had to do with an adhoc_tool target that was an upstream dependency of the targets I was trying to run (or test, in this case). Basically the adhoc_tool was just a custom script that did some checking on the constructed build artifact, before using it in the test. Very generically, it looks like python_source <- adhoc_tool <- python_test, and then I tried to pants test the python_test target.

Now, the problem is that this adhoc_tool does not generate new output that was added to the build artifact. Therefore, the naive approach that I used looked as follows:

adhoc_tool(
  name="placeholder",
  runnable="reference/to:runnable",
  args=[],
  execution_dependencies=[":some_python_source"],
  output_dependencies=[":some_python_source"],
)

This apparently caused the caching key to change on every invocation from pants 2.20 going forward (it worked like a charm on 2.19 and earlier). That meant the cache entries could not be used on subsequent invocations.

This simple change fixed the issue:

adhoc_tool(
  name="placeholder",
  runnable="reference/to:runnable",
  args=[],
  execution_dependencies=[":some_python_source"],
  output_dependencies=[":some_python_source"],
  output_files=["empty"],
)

And then touching that empty file in the adhoc runnable script.

The only, very small, downside of this approach is that there is now an unused empty file empty in the build artifact. I can live with that, but if people have other suggestions for this, let me know.

---

I'm not sure if this is intended behaviour for pants, and/or if there should maybe be a warning or error when no output files and/or directories are specified in an adhoc_tool. That's probably something for the pants maintainers to decide if they happen to read this.