Javascript/Typescipt support

[sureshjoshi]

High Level

prettier is a fmt plugin, with eslint support on the way as a fix and/or lint plugin.

This leaves us with the question(s) of transpiling/bundling and how they should work. The JS ecosystem is the wild west, with a new build system coming out ... monthly? So, the prospect of adding yet another tool to that mix doesn't give me the warm and fuzzies.

What would Pants's "purpose" be in TS/JS support? Would the idea be to be:

A) a replacement for stuff like webpack, vite, esbuild (yay Go), swc (yay Rust), turbopack (yay Rust again?), etc (to which I'm a giant 👎 ) - or,

B) would it be to use those as plugins and facilitate their usage in a consistent matter, and maybe throw some dep inference + caching in for good measure, or

C) Alternative/middleground

The prospect of using Pants and migrating config from vite to yet another transpiler/builder (since the JS ecosystem is the wild west), is a non-starter for me.

Notable Notes and Quotable Quotes from the Slack thread

I'd like to be able to use prettier, other tools the same way I do for my python tools (pants fmt).
I'd like to have something to use with npm/yarn/webpack to do dependency management and caching.
I definitely don't think its A. There's enough choice in the JS ecosystem without rebuilding the wheel here

• Thomas

...there's lots of language specific functionality built into the bundlers (often as plugins) ...
...I suspect it'll be a hard sell for front end engineering to switch away from that normal JS/TS tooling, at least until pants has proven itself...
I suspect pants only prove itself convincingly via B (ie running outside pants gives the same behaviour as running inside) or a relatively extensive A.

• Huon

I think the idea that Pants should not get in the way is a good one. As much as possible it should just run your tools for you, not prevent you from using your tools. Basically, you should feel ~comfortable in a --keep-sandboxes chroot even if you don't know Pants.

• John

The purpose should be: 1) provide a uniform interface, 2) ability to build a JS bundle and package it as an asset into a pex, 3) perf improvements where feasible

• Benjy

[benjyw]

What about test running, BTW?

[sureshjoshi]

Good question. These can be run as independent tools that run through the codebase, or they will rely on the top-level build tool and run as plugins of that.

However, I think most of the tools install their own deps. This might be another case where the test tool is a backend plugin, because even though a lot of these tools exist, they also mix and match pieces of the others (e.g. assertion libs and such).

I'm using vitest, but can't quite remember how I've set up test... If memory serves, I run an npx vitest and ensure that all my configs are brought in as well.

[benjyw]

I think starting out as a very thin passthrough to npm/yarn/webpack makes sense? Or whatever the most popular thing is in the JS world this month?

[sureshjoshi]

I thought about that, via run or something - which might have to be an escape hatch either way. Use ./pants run ... to emulate npm run ... from the package.json

[kaos]

I guess there are two different users (or projects) one may consider here.

a) a novice (or new) project without any particular ties towards what's hip for the month. Just want it up and running smoothly, with as little bother as possible.

b) an experienced user (or existing project) that want to leverage what Pants brings to the table. Would want to be able to re-use existing setup/configuration as much as possible.

I've got a feeling that one may want to peel away as many layers as possible to avoid too much bloat floating around, which suggests that the setup supported for (b) ought to be a temporary position while migrating towards (a), while still using relevant tooling from the js ecosystem to get the job done. a balancing act for sure ;)

I guess my question becomes, what technologies to support/accommodate, which to delegate to and how. Feels like it would be very easy to slip and drop down to becoming yet-another-js-build-tool. (well, which in a sense it is, but hopefully at a "higher level") perhaps closer to lerna in that sense.

[tobni]

I've done abit of hacking as a followup to my attempt to get the ball rolling in this area in #18138.

I have a suggested design, which I'm under way with prototyping:

package_json is a multi-target generator. It generates targets for

1. A node package. This target is for crafting node packages & libraries, and to describe in-repo package dependencies. So e.g ./pants package package.json creates a .tgz according to npm pack.
2. Resource targets used as bundles to embed in other apps (.css and .js files) to be served from e.g a flask pex app or express.js app.
3. Direct third party dependencies. These can be used for catching un-owned dependencies in dependency inference.

Example BUILD file for this suggestion:

package_json(
    package_manager="npm", # or yarn or pnmp, essentially package.json consumers, default configured in sub-system
    scripts=[
        node_build_script(entry_point="build", outputs=["dist"]), # A script specified in package.json that will tell pants it produces `resource()` targets
    ],
)

The real kicker here I guess is node_build_script, which I envision is another generated target, which in turn will be a target similar to other GenerateSourcesRequest implementors like experimental_shell_command, but letting the package manager + package.json do the heavy lifting.

A challenge here is how to facillitate caching to not make ./pants a lot slower than "bare" alternatives. I know at least webpack and parcel provide caches.

{
    "name": "spam",
    "private": true,
    "version": "1.0.0",
    "main": "./src/index.mjs",
    "scripts": {
        "build": "webpack"
    },
    "devDependencies": {"webpack": "4.0.1"}
}

File-tree for the example:

├── BUILD
├── src/ 
│   ├── BUILD
│   └── index.mjs
└── package.json

I have working dependency inference using babel to parse *.(c|m)js source files*. I also believe that a well placed files or resources glob could be very adequate in providing sources to the package_json:s dependency field for things like .css, .sass, .svelte or w/e ppl are using this month.

I have the poc on a branch of my fork here with support for npm: https://github.com/tobni/pants/tree/add/node-package-goal.

* seemed most prudent since @babel is an industry standard at this point. There will be work here to match the module-resolution hacks widely in use to avoid having to use relative imports, but it is not that different from user provided module mappings in python.

[sureshjoshi]

package_manager="npm" hey that's an interesting approach. I think I like it better than registering subsystems galore, since the ecosystem changes so frequently.

[tobni]

I envision there'll be some form of node.package-manager subsystem, to at the very least specify default tool & version for the binaries. I think using corepack for this is a good option, as it is shipped with node.js since current LTS, and it looks like it is going to position itself to be the de-facto node package manager manager (lol).

The package managers are thankfully similar enough that there need not be anything different for pants to do when inspecting package.json files or generally when invoking common scripts like pack, publish, audit etc. But they also have differences that need to be part of a solution for each, mainly when constructing chroots for the Process invocations.

Most notably are

1. Their invocations for generating and installing from "lockfiles". They all use different formats, and yarn@v2 has plug-n-play which means a second .pnp.cjs file is part of the lock. yarn@v2 also has zero-install mode which means .yarn/cache is part of a "lock", since it is checked into source.
2. yarn and pnpm use install --frozen-lockfile while npm has clean-install to install a package dependencies without mutation. Once again yarn@v2 has PnP Zero-Install which means the installation is checked into source, so pants would need to also accomodate the potentially checked in .yarn/cache and treat this as a "lockfile" for this purpose, instead of a named cache.
3. Workspaces (~packages in monorepo) are allocated and invoked a bit differently, npm install -w <package> vs pnpm --filter <package> install. pnpm also use pnpm-workspaces.yaml for this while yarn and npm use a package.json#workspaces. They are all just glob + cwd specifications though, so generalizing this ought to be easy.

All in all I think a way forward should be to start with npm support (because it is the most likely to be what users use). Followup to generalize over yarn@v1 and pnpm support later on.

I think yarn@v2 is not much different, but requires more pants code to "just work", because of the PnP and zero-install which seems to be the odd ones out. But that is honestly the case outside of pants as well, many workflows & packages do not work with yarn PnP at all.

[sureshjoshi]

I've been messing around with Corepack recently. So far so good. I kinda wish we'd be in a position to say something like "we support Corepack provided package managers" or something, just to keep this concept to a reasonable scope?

Aside: I'm still very hyped about this - but kinda stuck on some urgent projects until end of month, when I can put my Pants back on a bit and help out.

[tobni]

Agree on that position, I think a lot of users would turn away if pants dont support "their" pkg manager.

[tobni]

I'm actively working on this idea, current branch is https://github.com/tobni/pants/tree/add/node-build-scripts-resource-gen. I have working run for the node_build_script and package goal for the package_json taget for single NPM packages. My current plan of attack is

1. Get Resource codegen to work for node_build_script, I think it is mostly a refactor-of-and-jacking-into experimental_run_in_sandbox. I'm also interested to explore the possibilites of letting users provide extra caching directories as named caches so that this feature of bundlers isn't nuked by pants being hermetic.
2. Generalize workspace handling to not assume the workspace root == a package.json file, since that is only true for half the package managers I mentioned earlier.
3. Start exploring required subsystems and how support for locally installed nodejs, npm & corepack could work. I think this is a must to accomodate monorepos that have server side JS, as they tend to be stingy about nodejs versions, and add support for engineVersion (similar to python interpeter constraints, but specified in package.json). I think this would be a good point to add lockfile support to NpxToolBase as well.

[benjyw]

Agree re delegating test (and as much as possible in general) to the existing tooling, which is already rich.

The goal here should be to bring the existing tooling under the Pants umbrella with as little overhead as possible, and it sounds like that's exactly what you're doing!

[sureshjoshi]

Yep, agree with delegating as well. I had the idea and prototype of having each build system as an opt-in subsystem (starting with Vite, which is what I use), but it ends up being a maintenance mess.

I think what you're doing makes more sense to allow Pants to get out of the way, as much as possible - when there is already a solution (or, a huge set of solutions in JS that change every 2 weeks).

I think dep inference and maybe caching are where Pants can be more useful here. Not as build or test runners.

[sureshjoshi]

I guess delegating to the point of not just being a functional wrapper around experimental_shell_command - we'd want to add some value, otherwise we may as well just have a sample shell use case. 🤷🏽

[tobni]

The finer grained caching/invalidation of caches that is achievable by analyzing workspace dependencies is imo reason enough to not discard "proper" nodejs support in favor of experimental_shell_command.
Configuring that would be annoying repetition, and there's also the nicety of letting pants keep track of digests for clean installations instead of worrying coworkers are using outdated node_modules/, EDIT: or ran npm install and accidentally bumped some dependency because nodejs is a mess 😅

[sureshjoshi]

Right exactly, I think we're saying the same thing here :)

There should be enough value in a "proper" nodejs support, to not just be a superficial wrapper. And it sounds like we'll get some of that along the way.

[tobni]

I have about 3.5k loc and working parcel integration tests now. I am unsure how to proceed in terms of contributing this to pants. But I think it would be benificial for me if this could be reviewed and mainlined/rejected, as the larger this change grows the more "in the wrong direction" I might be running.

[sureshjoshi]

I'm basically unavail for reviews for another 2 weeks I think. I'll be free-er after that for reviews.

[stuhood]

Please absolutely consider contributing this to Pants, as that will be the best way to stay abreast of plugin API changes until it has stabilized. Thank you!

[sureshjoshi]

The 3.5k LOC is wild though!

[tobni]

Opened #18326, review when it is convenient :) I'd be happy to split the PR into parts if it eases review burden.

Edit: Also note the 7k loc is about 4k:s worth of package-lock.json... Not sure how to write integration tests without them.

[tobni]

Could someone please tag these with backend: javascript:

1. javascript_test target, and support running npm test <file> via pants test #18524
2. Add support for corepack the node package manager manager #18525
3. Add support for yarn@v1 for the javascript backend  #18526
4. Add support for pnpm for the javascript backend #18527

[huonw]

Done!

[tobni]

Status update:

Testing

#18742 - Adds batch testing support to javascript tests. I did this because I think a lot of test-runner optimizations were dodged by pants "1 file 1 process" default behavior. Batching should remedy that.
There is still no support for generating test reports, only coverage reports. And there's probably test workflows that are not (properly) supported such as snapshot testing (common in frontend work).

Package managers

#18748 addresses #18525 and the issue of "how to support versions" for package managers. The PR does not introduce actual support for any other package managers
I think I will work on #18527 next, as pnpm has more differences than yarn@v1 compared to npm. My reasoning is the chances of implementing the correct level of abstraction is higher.

Transpiled languages (mainly Typescript)

There's no news here w.r.t to implementation. The current idea of implementing a plugin interface for
transpiled languages that will be treated generically in goals (test, package, export-codegen etc) is still top of mind, though.

I think the current javascript dependency inference needs to support the full import subpath syntax outlined in https://nodejs.org/docs/latest-v19.x/api/packages.html#subpath-imports. It is similar to typescripts https://www.typescriptlang.org/tsconfig#paths, which is why I mention it.

[benjyw]

Thanks for the update! Which of the PRs above are ready for review?

[tobni]

I think both? I'm not 100% that all errors are CI flakiness, but honestly at this point it is hard to tell. I can confidently say tests pass on my ubuntu machine 😅 .

[tobni]

pnpm support: #18864

I think yarn support is a natural and cheap next step.

[tobni]

State of things:

• yarn support: javascript: Add yarn@v1 support #18928
• subpath-import inference: javascript: Support Node.js subpath imports #18934

@theoribeiro also did some exploratory testing (thank you!) which spawned:

• docker compat: javascript: Support package goal in node_build_script target #18926 + javascript: Add an explicit npm_distribution target instead of coupling to node_package #18925
• some things that broke/became obviously broken: javascript: Do not add literal {chroot} to pnpm cache path #18921 + javascript: Handle package.json placed at the root of the repository #18919

I'm looking at starting to add a typescript backend, preparatory to that is:

• Toggle:able and less annoying inference javascript: Implement dependency inference opt-out and improve resillience #18931

[tobni]

I'd like to air some details around the design space for a typescript backend, as I'd like to treat typescript both as a transpiled source, but also enable pants check to typecheck typescript files.

I think the path of least resistance for a check goal is NpxToolBase, but I'm not very enthusiastic about users having to maintain two typescript compiler versions, which would happen if user code uses a node_build_script to compile and bundle transpiled typescript.

This leads me to think maybe current NpxToolBase is not fit for the task of implementing the check goal. Maybe a solution to #17615 is to let a "NpxToolBase" (or something similar) point to a user node resolve, essentially letting a package.json#devDependencies version of typescript dictate what typescript compiler version is used by utilizing the hermetic, package manager aware sandboxing I've already implemented, instead of running with the npx binary bundled with the downloaded node distribution.

The other thought is how to provide typescript sources to the test, lint and package goals already in place. These goals are in theory source agnostic, but currently only accepts JavascriptSourceField. I think a pluggable approach is best. I'd like for the implementation to not make it difficult to use the dependency inference already in place. I think for jsx, tsx and typescript sources the current import parser will work fine, as babel has intrinsic plugins for those languages.

Outstanding question is how to deal with .svelte, .vue, or w/e else is the next framework of the week. Maybe JSImportString should be a @union, and just leave it up to plugins to implement ast parsers for the other frameworks to get dependency inference.

Typescript sources have special rules w.r.t to import strings that let them omit file endings and remap paths, this needs to be handled with separate logic in a typescript backend.

Another thought that has occurred to me is the IDE support. I think pants export needs to support exporting the node_modules folders of a node resolve, so that IDE:s have something to look at. There's probably details around workspaces that makes this... Non-obvious how it should work.

Other than that, I believe a typescript backend is fairly straight forward.

[tdyas]

The other thought is how to provide typescript sources to the test, lint and package goals already in place. These goals are in theory source agnostic, but currently only accepts JavascriptSourceField.

I have a question for my education on how the Javascript / Typescript is modeled. Do the rules which actually invoke node deal with targets/fields or with a javascript-backend specific abstraction?

I ask because the Go and JVM backends have two ways of modeling a project: the "higher level" abstraction is targets/fields but before actually invoking a tool they translate to a backend-specific abstraction:

• For Go, this is a tree of BuildGoPackageRequest instances. Each BuildGoPackageRequest has only a Digest with the sources to operate on and the direct dependencies (and other metadata).
• The JVM backends all use the ClasspathEntryRequest union to install their own language-specific compilation or processing logic. Particular implementations of ClasspathEntryRequest use the field_sets attribute to identify which multiple FieldSet instances they can support generating jar files for.

Maybe one or both of those approaches can inspire a solution for the issue at hand with Typescript?

[tobni]

I think it is more akin to BuildGoPackageRequest in some ways.
For any target that has applicable field sets for the backend, an InstalledNodePackage is constructed by sniffing out the closest package_json, resolving its deps and doing npm clean-install. After that, typically the goal is ran by running some script configured in the same package_json, also via npm.

node isn't really invoked directly anywhere in rule code. The layer that is currently being targeted is the package managers.
I think the JVM backend approach is the right one to achieve extensibility. However, I'm not planning for the transpilation process to be a modeled step in pants, rather something for the user to configure themselves in their nodejs project configuration. If the transpiled sources need to be re-introduced into the graph, you do so via node_build_script. I think this approach lets you re-use established project configs with ~zero modification.

I guess a "cost" in this approach is you cannot cache the transpiled outputs per file + deps, which otherwise would be awesome for big TS projects.

[tdyas]

I guess a "cost" in this approach is you cannot cache the transpiled outputs per file + deps, which otherwise would be awesome for big TS projects.

Do any of the transpilers output source<->destination file mappings?

In the longer term, could Pants supply plugins to (popular) transpilers to capture and output such information?

[tobni]

Yep, source maps provide that same information to debuggers and the like.

[tobni]

I'd just like to say that I am still keen on continuing to implement JS support, even if my free time was ~gutted this summer. I'm hoping to be able to start tackling #19819 next month-ish, and then continue with the bigger issues like re-visiting how workspace packages are built to handle integrity files and inter-workspace references.

[tobni]

#21176

This PR introduces the building blocks to support any transpiled js dependency inference. I think it is the first missing piece for typescript, jsx, svelte and vue.

My hope is to followup this PR with support for typescript, jsx and tsx support in build scripts and tests, as those are what my org uses.

[tobni]

And here's JSX support #21206.