Add parser for pyodide.js and pyodide.mjs to avoid webpack traversing dependencies #22
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Reduces the amount of other configuration that may need to go in like stubbing node core, ignoring certain node modules, or writing your own loader. Use acorn to find and replace global pyodide and add in named exports so we have a consistent interface between common and esm. Regular expression isn't as resiliant to change as AST. CLean up configuration so it's easier to understand the different build outputs for debugging. Add examples for users.
Main change adds the after build into a new script to be reused in the other configs. After build is used to copy the build into the example folders which are useful for testing. You cannot npm link the build because the links will resolve pyodide differently and a check for specific types will fail when it loads the same file twice from different location (they receive different memory addresses).
Using esbuild to transpile the pyodide module (.mjs) to common js format. There may be a better way to do this but the issue is: When you build a module project (ie your project using pyodide is a module) with webpack then it loads the pyodide.mjs. We do not want webpack to process the pyodide.m?js file at all because they have been output by emscripten and are already compatible with the browser and node and contain pollyfills. The easiest way to handle skipping a pollyfill is to load the file as a string and eval it into existence. Webpack will skip processing this and treat it as a string. You cannot eval a module. eval doesn't recognize import/export. I convert the build to commonjs, eval it, then export the loadPyodide function back for esm compatability.
This was a copy/pasta from a fit of debugging
This file needs to be commonjs. Webpack loaders do not currently support esm.
Done testing. Loading the esm module like a typical import should work without having to specify the full file path
One code path to rule them all. All code is commonjs all the time now. Is this backwards? I don't think so. Mentioned in previous commits you cannot eval an esm piece of code and webpack will try to parse any js you throw at it. There's got to be a better way and I just don't know it.
More exhaustive tests that verify each build passes without errors.
Required now for testing
We need to know where to copy files from so resolving the pyodide package location is necessary. The old way assumed there was a node_modules folder in your cwd and then listed all folders in there looking for 1 named pyodide. Now using require.resolve to let node find the package for us then walk up to find the root of the package. Require.resolve returns the main / entrypoint of the package which can be nested. We need the folder that contains package.json. There is a catch to stop traversing any lower than the node_modules folder so it doesn't try to walk into protected directories.
Share dependencies during development with examples making it easier to develop and run tests.
Required minor changes to test and build phase so that we only count "examples" as the directories in the examples folder and not everything in that folder (files, links, etc...).
No need to support anything less than 18. 14 and better should work but 18 is already in maintainance mode. 16 is EOL. Recommend node 20 for new projects.
esm builds are working now without issue. Full support for both commonjs and esm builds is added. Addresses #21. We no longer allow webpack to try and parse the emscripten build. We do that ourselves.
dlech
reviewed
Nov 18, 2023
| @@ -0,0 +1,15 @@ | |||
| const { loadPyodide } = require("pyodide"); | |||
There was a problem hiding this comment.
Since the examples are used for tests, this should import everything. I.e. by also importing version, hopefully it would reveal the same error as in #25
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #20 and #21
Pyodide is an emscripten project. There are various ways to produce those builds with emscripten and only certain configuration are appropriate for webpack. Not all consumers of pyodide know or care about webpack - we cannot enforce certain build patterns on the core team for a subset of users.
This large change adds a loader with a parser to the plugin. Webpack will no longer look at the pyodide.m?.js file and try to make sense of it. Instead we parse it ourselves to make changes appropriate for webpack and then inject it as a chunk.