Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dodge node's require cache #13

Merged
merged 2 commits into from
Sep 26, 2023
Merged

Conversation

airhorns
Copy link
Contributor

@airhorns airhorns commented Sep 19, 2023

requirefire intentionally re-creates the entire require stack such that when used, its as if the returned require function is from a totally different process. Two requirefire instances should be totally independent, with different require caches. Crucially, if the files on disk change between the creation of one instance and the next, the second instance should reflect those changes.

Before this change, that wasn't the case in a particular circumstance that burned us in Gadget. Specifically, if a module was required by require fire, and that module required another module from it's node_modules directory, subsequent requirefire instances requiring the entrypoint module will always get the same copy of the node_modules module, regardless of if it changes.

An example:

  • const a = requirefire();
  • a("foo") requires "@gadgetinc/api-client-core" from foo's node_modules, gets one copy of api-client-core
  • we change api-client-core in the node_modules on disk
  • const b = requirefire();
  • b("foo") requires "@gadgetinc/api-client-core" again, should return the new copy, but didn't.

The reason the old copy was returned by new instances of requirefire is that there was another cache inside the node internals that all requirefire instances were sharing by accident: Module._pathCache.

The path cache is used by node to cache resolutions from entrypoints to files on disk. Usually in node the path cache is only ever filled and never emptied, and thus node will never pick up new or changed resolutions after the first time they are run. We explicitly want the opposite with requirefire, so we need to change to using one path cache per requirefire instance instead of one shared one.

Super regrettably, this shared global cache is baked in deep to node. I couldn't figure out any way to re-use node's internal logic while still changing the semantics of which path cache is used, so I had to skip node's resolve stuff that uses that cache entirely. Luckly, there's a great userland re-implementation of resolving in webpack that we can use, that has great support for ... everything. As webpack would need!

@airhorns airhorns force-pushed the file-alias-changing-resolve-package-2 branch 5 times, most recently from 6262d78 to e9bdd5a Compare September 20, 2023 17:28
…een requirefire instances

requirefire intentionally re-creates the entire require stack such that when used, its as if the returned `require` function is from a totally different process. Two requirefire instances should be totally independent, with different require caches. Crucially, if the files on disk change between the creation of one instance and the next, the second instance should reflect those changes.

Before this change, that wasn't the case in a particular circumstance that burned us in Gadget. Specifically, if a module was required by require fire, and that module required another module from it's node_modules directory, subsequent requirefire instances requiring the entrypoint module will always get the same copy of the node_modules module, regardless of if it changes.

An example:

 - const a = requirefire();
 - a("foo") requires "@gadgetinc/api-client-core" from foo's node_modules, gets one copy of api-client-core
 - we change api-client-core in the node_modules on disk
 - const b = requirefire();
 - b("foo") requires "@gadgetinc/api-client-core" again, should return the new copy, but didn't.

The reason the old copy was returned by new instances of requirefire is that there was another cache inside the node internals that all requirefire instances were sharing by accident: `Module._pathCache`.

The path cache is used by node to cache resolutions from entrypoints to files on disk. Usually in node the path cache is only ever filled and never emptied, and thus node will never pick up new or changed resolutions after the first time they are run. We explicitly want the opposite with requirefire, so we need to change to using one path cache per requirefire instance instead of one shared one.

Super regrettably, this shared global cache is baked in deep to node. I couldn't figure out any way to re-use node's internal logic while still changing the semantics of which path cache is used, so I had to skip node's resolve stuff that uses that cache entirely. Luckly, there's a great userland re-implementation of resolving in webpack that we can use, that has great support for ... everything. As webpack would need!
@airhorns airhorns force-pushed the file-alias-changing-resolve-package-2 branch from e9bdd5a to 2b47171 Compare September 20, 2023 17:29
@airhorns airhorns changed the title File alias changing resolve package 2 Dodge node's require cache Sep 20, 2023
@airhorns airhorns merged commit 638a406 into main Sep 26, 2023
5 checks passed
@airhorns airhorns deleted the file-alias-changing-resolve-package-2 branch September 26, 2023 03:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant