Architecture revision

[noahlange]

I couldn't find any threads in this vein in discussion, so I figured it might be worth bringing up.

The codebase is messy and requires re-design, but I will appreciate if you start with minor changes.

I think there are options for incremental rejiggering that might be worth considering. esbuild is a very fast and robust (quasi) spiritual successor to browserify. The scope is much lower than something like Webpack or Vite—it's not really designed to handle asset bundling, &c. Command-line usage is straightforward so there aren't any massive config files, which is nice.

npx esbuild index.js --bundle --outfile=bundle.js

Situation

I was digging around in this repo and an approach stood out that may work. Right now—maintainers know this, obviously, but for the sake of clarity—the code is all laid out like this:

├── modules
├── libs
├── utils
│
└── index.html

And index.html contains a list of every JS source file in the appropriate order for loading. And everything's in the global scope and that all works, and that's great. But as we all know, it's not super-scalable in the long term and makes some fixes/improvements basically impossible (e.g., moving the generation code from the UI thread to web workers to eliminate UI lag issues).

So the question becomes 'how do we move things out of the global scope in a way that's both maintainable in the long term and can be migrated piecemeal without needing to do everything all at once'.

Suggestion

So, thought: without modifying the existing project structure, add a src directory that contains code that won't be loaded at runtime but instead at bundle/build time. A new file, ./src/index.js is added. This file does one thing: imports stuff from JS modules in src/**/*.js and makes it globally available by explicitly assigning it as a property on window instead of implicitly declaring it in the global scope.

We add an esbuild script to bundle everything from src/index.js into bundle.js and pull that into index.html via a standard script tag.

Now code can be moved into src and modified to use the module system, while existing code that uses global references can keep using those global references. And gradually, more and more code can be pulled into src until a given property/function on window is no longer being used globally, at which point it can be removed and we're one property/function closer to getting everything out of the global scope.

Issues with this approach

• some existing "idioms" probably won't work in both the old and new environments
  • generation code may need to change slightly / some functions would probably acquire "data" parameters
    • this could be a good time to reassess those kinds of revisions, though
• adds a mandatory compilation step. this is obviously a bummer, but it's nearly inescapable at this point
• increases repo complexity, adds cognitive overhead with a will-throw-an-error-if-you-try-to-import-it directory

Case study: probabilityUtils.js

1. move ./utils/probabilityUtils.js to ./src/utils/probabilityUtils.js
2. prefix each function declaration with the export keyword
3. globally assign/export function in ./src/index.js

// src/index.js
import * as probability from './utils/probabilityUtils';

// would probably create some sort of shim to do this automatically
window.P = probability.P;

4. New code imports from src/utils/probabilityUtils.js:

// ./src/modules/whatever.js
import { P } from '../utils/probabilityUtils'
// could do something shorter with TS paths + re-exports
import { P } from '@/utils';

P(1);

5. old code doesn't change

// ./modules/whatever.js
P(1);

Anyway—interested to hear what y'all think. It's obviously a complex problem, but this could be a way (to mix metaphors) to get the ball rolling without rocking the boat.

[StempunkDev]

There is the ongoing vite-migration migration effort as a first goal, that has a simmilar approach to your suggestion(for more detail look at the vite PR and Braches)
A even more abstracted version would be the Package-Architecture that has a proposed model over at trello.
The Package-Architecture sounds promissing, but I can only speak from a high level(not much experience with js/ts)

[StempunkDev]

Note: Package-Architecture will probably lead to subrepos. Probably better Testing of submodules and modability of modules depending on the type that is wanted, like realistic, fantasy, planets

[Azgaar]

Hello, please see here: #842

I don't see issues in using Vite, it works fine and it useful for some of the things we also want.

moving the generation code from the UI thread to web workers to eliminate UI lag issues
Generation code is fast, you won't get anything if you move it. What is slow is rendering, and it's handelled by browser. SVG engine is slow and getting slower each year. No matter how many GPU it takes, it still very slow. As far as I know, no one is working on it in Google/MS/Firefox.

[noahlange]

Gotcha. Yeah, Vite is great — I definitely wouldn't recommend going back to ESBuild from that.

I didn't see any recent activity on that branch, so I assumed that effort had sputtered out a bit. (Which would be totally understandable, given the immense scope of an all-at-once sort of migration.)

[Azgaar]

Yes, I switched to other tasks, but it's still planned to be continued. It a big task, what is already done took me about 100-200 hours of pure work.