Move all stuff to monorepo?

[dafanasiev]

Is your feature request related to a problem? Please describe.
Feature request

Describe the solution you'd like
haraka/Haraka depends on many other haraka/* packages (plugins, core...). Its not easy to maintain all of them (and packages deps).
I thinks that monorepo can help with that. Using this solution plugins will be easy to debug and develop.

Describe alternatives you've considered

• npm workspaces
• yarn workspaces
• lerna

Additional context
For example: I try to make PoC (I use master for all haraka/* packages):

$> npm --version
9.6.4
$> node --version
v20.0.0
$> npm i

added 86 packages, and audited 925 packages in 6s

74 packages are looking for funding
  run `npm fund` for details

13 vulnerabilities (6 moderate, 5 high, 2 critical)

$> npm outdated
Package                 Current  Wanted  Latest  Location                                                    Depended by
@elastic/elasticsearch   7.17.0  7.17.0   8.7.0  node_modules/@elastic/elasticsearch                         [email protected]
ldapjs                    1.0.2   1.0.2   3.0.2  node_modules/ldapjs                                         [email protected]
ldapjs                    1.0.2   1.0.2   3.0.2  node_modules/ldapjs                                         [email protected]
mocha                     9.2.2   9.2.2  10.2.0  packages/haraka-net-utils/node_modules/mocha                [email protected]
mocha                     9.2.2   9.2.2  10.2.0  packages/haraka-plugin-aliases/node_modules/mocha           [email protected]
mocha                     9.2.2   9.2.2  10.2.0  packages/haraka-plugin-karma/node_modules/mocha             [email protected]
mocha                     9.2.2   9.2.2  10.2.0  packages/haraka-plugin-limit/node_modules/mocha             [email protected]
mocha                     9.2.2   9.2.2  10.2.0  packages/haraka-plugin-recipient-routes/node_modules/mocha  [email protected]
mocha                     9.2.2   9.2.2  10.2.0  packages/haraka-plugin-redis/node_modules/mocha             [email protected]
mocha                     9.2.2   9.2.2  10.2.0  packages/haraka-plugin-spf/node_modules/mocha               [email protected]
mocha                     9.2.2   9.2.2  10.2.0  packages/haraka-plugin-watch/node_modules/mocha             [email protected]
mocha                     9.2.2   9.2.2  10.2.0  packages/haraka-plugin-uribl/node_modules/mocha             [email protected]
mocha                     9.2.2   9.2.2  10.2.0  packages/haraka-tld/node_modules/mocha                      [email protected]
mocha                     9.2.2   9.2.2  10.2.0  packages/haraka-utils/node_modules/mocha                    [email protected]
nodeunit-x               0.15.0  0.15.0  0.16.0  node_modules/nodeunit-x                                     haraka@npm:[email protected]
nopt                      7.0.0   7.0.0   7.1.0  node_modules/nopt                                           haraka@npm:[email protected]
nopt                      7.0.0   7.1.0   7.1.0  node_modules/nopt                                           [email protected]
redis                     4.5.1   4.5.1   4.6.5  node_modules/redis                                          haraka@npm:[email protected]
redis                     4.1.1   4.1.1   4.6.5  packages/haraka-plugin-karma/node_modules/redis             [email protected]
redis                     4.5.1   4.6.5   4.6.5  node_modules/redis                                          [email protected]
redis                     4.5.1   4.6.5   4.6.5  node_modules/redis                                          [email protected]
redis                     4.5.1   4.6.5   4.6.5  node_modules/redis                                          [email protected]
redis                     4.5.1   4.6.5   4.6.5  node_modules/redis                                          [email protected]
redis                     4.5.1   4.6.5   4.6.5  node_modules/redis                                          [email protected]
semver                    7.3.8   7.3.8   7.5.0  node_modules/semver                                         haraka@npm:[email protected]

[msimerson]

Short answer: been there, done that, no thanks.

Haraka used to be a monorepo. Although some things were better in a monorepo, many things were not. Things that I find worse in a monorepo:

• CI build & test times. All dependencies must be installed for every tiny change. No, I really don't want to have to install ldapjs when all I did was a a doc fix in this far-from-ldap module.
• Same with email-message. It's now the one haraka lib that loads iconv, a slow dependency to build, so every single change in the Haraka monorepo caused minutes of extra CI build times. Days of developer time were wasted fiddling with CI caches (with also have edge cases) to work around that. The solution was simple: repackage it in its own dependency, with its own tests. That made building the main Haraka project much faster.
• with discrete packages, we can iterate in them and fire off releases much faster. Releases in Haraka were much harder before and the pressure to make sure everything was perfect (which it never is) was higher. It took weeks to make changes in the monorepo, and for some things, years. For example, we're still using the long since deprecated nodeunit in the main Haraka repo. Most of the repackaged modules are using a modern and supported test runner. It's quick, easy, and not very risky to try that in a small library. It's so much work to do that in a large monorepo that years later it's still not done.
• some types of changes (lint, refactoring, es6, etc.) were really really unwieldy in the monorepo. Way too many changes all at once, takes a long time to review, and there's always an exception to every general rule. We've got enough code that when you apply a "simple" change 300 times, it inevitably breaks several somethings. That imposes a heavy burden on code reviewers to try and catch those needles in the haystack.
• Revisions. If a change in a lower level Haraka module breaks a build or causes a problem, switch versions. No need to wait for releases or monkey patches.

[msimerson]

Its not easy to maintain all of them (and packages deps).

This is a point I want to push back against. It's actually substantially easier now to maintain Haraka packages. We have shared CI workflows that most every repo uses. (Exceptions tend to be things like plugins that require Redis and won't successfully test on Windows, and stuff like that). Separating out those edge cases into lower modules where they're affected means the main Haraka repo doesn't have 413 workarounds for every type of issue that arises somewhere in our code base.

Having a pile of haraka libraries nudged me to spend the time to automate the release process. It wouldn't have ever happened in the monorepo, because releases were much less frequent. Now releasing is something that's done often. (Yes, we did look at a dozens of already baked release workflows, but all depended on particular workflows that the history in our main repo didn't adhere to, or required that every developer adhere to things like Conventional Commits. Fine by me, but I won't make that a hurdle everyone else must jump.

[dafanasiev]

CI build & test times. All dependencies must be installed for every tiny change. No, I really don't want to have to install ldapjs when all I did was a a doc fix in this far-from-ldap module.

Cache available + npm workspaces:

# NOTE: npm ci - about 4 sec.
$> npm i  
added 86 packages, and audited 925 packages in 6s

with discrete packages, we can iterate in them and fire off releases much faster.

You can also do this with workspaces ("monorepo" != "move all js to one directory").

some types of changes (lint, refactoring, es6, etc.) were really really unwieldy in the monorepo.

You can use pnpm workspaces + nx (article: https://blog.nrwl.io/setup-a-monorepo-with-pnpm-workspaces-and-speed-it-up-with-nx-bc5d97258a7e)