Separate the (Python) server from the frontend in the source tree

[justvanrossum]

This is with @simoncozens: to clean up the Fontra source tree, and clearly separate the Python server source code from the front end code and assets, with the purpose of opening up more deployment options for Fontra, such as a serverless version.

Possible breakage that we must keep in mind

• Fontra Pak
• Fontra RCJK deployment

Steps:

• Rework URL scheme so the "project identifier" becomes a query key/value #1960
• Move script imports and inline JS in HTML files into view-specific start.js files (Move JS code from HTML to start.js #1984)
• Add scaffolding which does not interact with existing development (new webpack.config.js, package subdirectory structure, etc.) (JS package scaffolding #1985)
• Rename bundle-rollup to build in package.json, make build hook similarly generically named (Make bundler actions generically named #1983)
• Update Node requirement to >=20, in mention in README.md, set GH Actions workflow Node to 20.x (or latest?)
  • Make test-js/test-classes.js use "with", now that we require Node >=20
• Update README.md:
  • Build instructions for deployment
  • Build/watch instructions for development cycle
• Update .gitignore to ignore src/fontra/client/
• A check that we have Node >=20, rather than confusing syntax error with npm test
• Implement solution for fontra-rcjk's project manager (does it need its own little big-bang?)
• Commit correct package-lock.json
• Schedule a big-bang migration of JS files into packages plus server modifications

(To be edited...)

[simoncozens]

OK, I think I am getting close to this. Everything is working nicely locally; you can test my js-split branch.

To answer the remaining question about JS tracebacks, I've added two build modes - when Fontra is installed with pip install ., it runs npm run bundle which runs webpack in production mode. This gives you minified JS and tracebacks like start.1a2b3c.js:17839. When you run in development mode (npm run bundle-watch), then you get sourcemaps, sensible backtraces, and original JS code in the developer tools console. There are apparently many options between those two extremes if we want to tweak things.

FontraPak works with a small change to the fontoverview URL.

I think I need to check that other projectmanager implementations also work.

[simoncozens]

Oh... and the renaming of JavaScript imports is now scripted, so we can quickly recreate the "big-bang" commit whenever we decide to do it.

[justvanrossum]

the renaming of JavaScript imports is now scripted

Do you mean the "making all JS imports consistent as well as working in the new environment"?

[simoncozens]

Right. Once the JS scripts are moved into their new src-js/ locations, this script can be used to make the imports consistent and to use absolute package names (@fontra/core/... etc.).

[justvanrossum]

Lovely.

[justvanrossum]

If I start the server in the js-split branch:

python -Wall -m fontra filesystem path/to/folder/with/fonts/

...and go to http://localhost:8000/, the server errors:

FileNotFoundError: [Errno 2] No such file or directory: '/Users/just/code/git/BlackFoundry/fontra/src/fontra/client/landing.html

I did pip install -e . before I launched the server. Is there something I missed to do on my end?

[simoncozens]

I'll try it from a clean checkout but it feels like the npm run bundle step didn't put anything into src/fontra/client.

[justvanrossum]

src/fontra/client is populated, and Fontra Pak (with fontoverview/ -> fontoverview.html change) actually works (awesome!), so it seems to be isolated to the landing stuff?

[justvanrossum]

npm test does not work, though:

$ npm test

> test
> mocha tests --reporter spec

Error: No test files found: "tests"

[simoncozens]

Nope, I haven't done that bit yet. (I did it in the first PoC, I'll bring that change over.)

[simoncozens]

src/fontra/client is populated, and Fontra Pak (with fontoverview/ -> fontoverview.html change) actually works (awesome!), so it seems to be isolated to the landing stuff?

Yeah, I'd forgotten a package.json file for the filesystem projectmanager. Added now.

[justvanrossum]

That works, great!

[justvanrossum]

The RCJK project manager isn't working, but it sounds like it isn't expected to be yet. I do see some login fields, but the CSS is missing so the layout is all garbled. I'll send you an email with a private server location so you can test.

[simoncozens]

Thank you. I've just done a new (force) push and tests are working.

[simoncozens]

I think I need to think a bit more deeply about how to make pluggable project managers like fontra-rcjk work. They need to ship their own HTML/JS assets, but we can't expect them to be bundled into the fontra.client package; so they probably need to do their own bundling. But at the same time they have to be able to access JS API stuff in @fontra/core.

[justvanrossum]

(Also struggling with this.)

We can let go of the fontra.client idea if it helps, and find another way how the server finds the front end assets, and another way how Fontra Pak can bundle everything into the app.

I do like the idea of having a Python package that we can use to target the bundler output, but it doesn't have to be fontra.client. Would it help if we separate that from the main fontra Python package?

[simoncozens]

Well, the issue is that we build a bunch of HTML/JS assets at Fontra install time, and then later a user does pip3 install fontra-rcjk or whatever and expects the plugin to be available. At that point, we can't rewrite the package which is holding all bundler output. However, I think it would work for the plugin architecture to be basically the same as it is now (the server asks the plugin to serve assets related to itself from out of the plugin's own Python package), if the plugin could also run a bundling step on install similar to what we're doing with Fontra core. I'm playing with this ATM.

[simoncozens]

OK, I think I have something but I'm not entirely happy with it, so I'm just whiteboarding here. We have multiple classes of thing:

Current model

Views

What is it? A HTML file and one or more JS files and other assets.
Where does it live? In a Python package
How does Fontra it exists? It registers itself with the fontra.views package entry point.
How does Fontra serve it? viewPathHandler verifies the project and the authentication token, then serves the viewname HTML file from the Python package. Other assets such as the JS file get served via the fontra.webcontent package entry point, which the server knows and routes to the right Python package.
How are custom, out-of-tree instances handled? Python modules get installed with pip, and register their entry points so the server knows what to do with them.

Projectmanagers

Basically the same as views, but:

1. Their HTML file gets delivered as the root file. Other assets are served through webcontent entry points as above. (This has subtle implications for the paths of bundled files - HTML files which expect to be served as / need to put their JS/CSS assets under /filesystem/ or similar. The webpack output.publicPath option can help with this.)
2. They have some Python code attached to them which may alter their behaviour.

New model

Views

What is it? A HTML file and one or more JS files and other assets.
Where does it live? In a Javascript package
How does Fontra it exists? The package is added to Fontra's package.json
How does Fontra serve it? All views are bundled into a fontra.client Python package - when the Python fontra module is installed - and anything in fontra.client is served as web content.
How are custom, out-of-tree instances handled? So this is where I was hoping to get clever and say "Javascript modules get installed with npm, fontra knows about them because it can read the package.json dependencies, and handwave." But if views need bundling so that they can use @fontra/core JS API calls, and if custom things can be installed after base Fontra is installed, then handwaving is not enough here.

And I can't punt on the "custom views" problem, because it's broadly equivalent to the "custom projectmanagers" problem and we need that to work for RCJK. So.

Maybe we're in a world where

1. Custom views and project managers continue to register themselves with Fontra via entrypoints.
2. Built-in views are known and get bundled into the fontra.client Python package, but nothing else does. So Fontra's webpack.config.cjs can be a lot simpler and I don't need to grovel around in package.json to find all the views.
3. Custom views aren't just JS modules; they're Python modules (so that they can register themselves with Python as endpoints) which contain JS/HTML bundles - bundled on Python package install just like as in Fontra itself - which go into a mycustomplugin.client Python package.
4. The server serves those broadly the same way it does now.

Yeah. I think that's basically where I had got to in my head anyway, but now I'm convinced.

[simoncozens]

OK, I now think I have this working, including custom out-of-tree views (demonstration package) and RoboCJK projectmanager.

[justvanrossum]

I can't get this to work. I started with fresh checkouts of your fontra and fontra-rcjk forks, in their js-split branches.

In fontra, with Node 23:

• pip install -e . works
• npm test does not
• (there's a failing Python test about the removed version token stuff)
• (I initially tried with Node 21, as that was my active version, it doesn't work with that version. That's fine, but we'll have to document.)

In fontra-rcjk:

• pip install -e . fails with "OSError: Build script does not exist: scripts/bundler_build_hook.py"

[justvanrossum]

fontra-view-features seems to work, though. That's very cool.

[simoncozens]

Should have fixed those issues now.

[justvanrossum]

pip install in fontra-rcjk still fails. npm install there fails like this (which I think is what pip install is failing on):

(venv) JvR-M3:fontra-rcjk just$ npm install
npm error code ENOENT
npm error syscall open
npm error path /Users/just/code/git/BlackFoundry/simon-fontra/fontra-rcjk/package.json
npm error errno -2
npm error enoent Could not read package.json: Error: ENOENT: no such file or directory, open '/Users/just/code/git/BlackFoundry/simon-fontra/fontra-rcjk/package.json'
npm error enoent This is related to npm not being able to find a file.
npm error enoent
npm error A complete log of this run can be found in: /Users/just/.npm/_logs/2025-01-31T09_58_43_074Z-debug-0.log

[simoncozens]

Argh, failed to git add the relevant files. Try again!

[justvanrossum]

That works! Thanks.

If I look into fontra-rcjk's package-lock.json files, I see entries like this:

    "node_modules/@fontra/core": {
      "version": "0.0.1",
      "resolved": "https://gitpkg.vercel.app/simoncozens/fontra/src-js/fontra-core?js-split",
      "integrity": "sha512-z6MSlkveHndgmlxceVm8e5cDoc52aXyNA6PN1jBcOAURUMetgPp6W/cGXhbXFHlZ6rHn9r5cR3yb8cjMrKPf7A==",
      "dependencies": {
        "@fontra/core": "file:",
        "bezier-js": "^6.1.4",
        "fflate": "^0.8.2",
        "lib-font": "^2.4.3",
        "tsc": "^2.0.4"
      }
    },

Can you elaborate a bit on how this works and why it needs to refer to an external fontra-core?

[simoncozens]

Currently fontra-rcjk does this:

import { parseCookies } from "/core/utils.js";
import { startupLandingPage } from "/filesystem/landing.js";

assuming that it will be able to find those individual JS files in the server's deployment directory. This is no longer true in the new model, because (a) core/utils.js may not be at the server root, and (b) bundling.

Post-change, fontra-cjk will do this:

import { parseCookies } from "@fontra/core/utils.js";
import { startupLandingPage } from "@fontra/projectmanager-filesystem/landing.js";

That is, it will import these two functions from JS dependencies, and the code for these two functions will be bundled into the single JS file which makes up @fontra/projectmanager-rcjk. So the idea is that @fontra/core and @fontra/projectmanager-filesystem (etc.) become "ordinary" Javascript packages, which means they get added as dependencies to the @fontra/projectmanager-rcjk package which can then use their functionality.

For that to happen, @fontra/core and friends need to be hosted somewhere that npm install can get them. Eventually they may become plain npmjs.com packages; or they may get their own independent repositories - or they may stay where they are and get served from GitHub subdirectories. For the proof-of-concept, I'm serving them as npm modules inside a GitHub subdirectory, and gitpkg.vercel.com is currently the best way to do that.

[justvanrossum]

Understood, so these would ultimately point to our own repos via the gitpkg.vercel.com tool.

Does this also imply that in the fontra-views-features demo, all needed bits of fontra/core will be bundled for that view, and not be shared with the builtin views? (Assuming fontra/core is at least shared among builtin views)

[simoncozens]

It does, yes. Once I have a useful demo plugin we can see how much of @fontra/core is actually used and how much that adds to the JS, but in a way it's irrelevant; if we're going with this approach, that's the price of entry.

[justvanrossum]

Is it the same for builtin views, as in, are they each bundled individually?

[simoncozens]

Yes and no; webpack can be configured to split out common code. I haven't played with the different ways to optimise this yet.

[simoncozens]

I have now played with this, and we really can save whatever chunks we like - all the node_modules code in one file, or separate files per module, or all the fontra code in one file, or separate, or whatever. An absolutely minimal strategy would bundle everything into a single shared fontra.js and have start.js files per view:

936K	src/fontra/client/fontra.42214453495ccb6be1cb.js
4.0K	src/fontra/client/landing.d241fa29fae71d17c6b5.js
 12K	src/fontra/client/start.1.2990ddbb954cd4f579b9.js
 12K	src/fontra/client/start.2.0563f2d1b27f7c3ccf10.js
 12K	src/fontra/client/start.3.ddc68521d2c65abcb40a.js
4.0K	src/fontra/client/start.b6cadb84a938fb042a43.js

Or we can configure it to do essentially what we have right now:

408K	src/fontra/client/fontra-core.c0048cffdc78fdd0ff85.js
 80K	src/fontra/client/fontra-webcomponents.a3aa8cc120bdcc2093a5.js
4.0K	src/fontra/client/landing.73d0884b25a7361863be.js
 24K	src/fontra/client/npm.bezier-js.73670c411a8fe613569a.js
 12K	src/fontra/client/npm.fflate.cceee27831537519ed53.js
 68K	src/fontra/client/npm.lib-font.e5b84eee0ebdbd281895.js
 12K	src/fontra/client/start.1.057ee13d4324786d95e5.js
 12K	src/fontra/client/start.2.adcc93048ec82893a629.js
 12K	src/fontra/client/start.3.edf99d18e2131f545b5e.js
4.0K	src/fontra/client/start.f5838803d9fc8821c52e.js
 16K	src/fontra/client/views-applicationsettings.772c9f4c138f93e6c2eb.js
268K	src/fontra/client/views-editor.eb5ec121d0e084c9d8b3.js
 48K	src/fontra/client/views-fontinfo.2d2d2f1d4d5a58a960f8.js
 24K	src/fontra/client/views-fontoverview.bada1fdd6c8943263707.js

It's all tweakable.