Allow for package.json deep merge to support more scenarios

[gossi]

When supporting addons from ember@3 up to ember@6, there needs to happen various changes here and there.

My current problem is roughly mentioned here: embroider-build/addon-blueprint#131
Basically the resolving @ember/owner in ember <> 4.12 - that in some scenarios is missresolved by the package manager, namely pnpm for me. That's why we are advised to use dependenciesMeta.*.injected. However, that comes with a big downside for actually developing the addon and requires third-party-tools only to fix that.

A handy trick would be for CI to have that scenario modelled, in my case for older ember versions.
Here is my sample commit for that: gossi/ember-ability@9ecfd9b which adds a script and dependenciesMeta to some package.json - but that does not work, because only a couple of selected properties are overwritable, see:

ember-try/lib/dependency-manager-adapters/base.js

Lines 142 to 150 in e0d196e

	_packageJSONForDependencySet(packageJSON, dependencySet) {
	this._overridePackageJSONDependencies(packageJSON, dependencySet, 'dependencies');
	this._overridePackageJSONDependencies(packageJSON, dependencySet, 'devDependencies');
	this._overridePackageJSONDependencies(packageJSON, dependencySet, 'peerDependencies');
	this._overridePackageJSONDependencies(packageJSON, dependencySet, 'ember');
	this._overridePackageJSONDependencies(packageJSON, dependencySet, this.overridesKey);
	return packageJSON;
	}

What about a deep merge instead and everybody is free to change what they seem necessary for them?

[kategengler]

I think deep merge might be a bit too free, but I support adding hooks to do this kind of thing.

I ran into the issue with dependenciesMeta*injected this week. I found on pnpm v10 it wasn't necessary but also that removing it exposed issues with ember-try with pnpm in workspaces that causes errors around peer deps. I think we can solve this in ember-try.

[gossi]

I forgot to share my thoughts behind this, let me add them.

I was thinking a deep merge might be a bit too much in the first place, too. I had some security gotchas in mind, but I don't think they hold true as its the engineers who write these scenarios. If there really is something I'm not seeing it - please let us know.

Then I thought, ok allow more fields to be overwritten. But then it comes down to which fields and in essence, that ember-try will make a judgement what fields engineers are allowed to overwrite. I consider its the engigneers who best know what they need to change, which is why I considered this judgement a bit too limited and will result in "one issue/pr for a each new field"

I ended with the idea for a deep merge, as it is in control of package authors, neither harmful nor limiting in any form but enabling in any way possible.

How does that incorporate with your thoughts?

[kategengler]

I could be convinced of a deep merge. I have also long thought a hook that is passed the package.json and allows users to do what they will would good.

I'd like to hear @bertdeblock's thoughts since he's been doing the most maintenance of this package.

[bertdeblock]

My simplified ember-try clone (try-or-die) uses a "deep" merge/set approach. It doesn't deep-merge arrays though, but it is super flexible.

It will work well for single-project repos, but in case of pnpm workspaces, e.g. overrides need to be defined in the root package.json file. So that needs to be taken into account as well somehow.

If we would do something like this, packageJson might be better as a config key than npm as well?

Also, with the current API, changing package manager is a single line change, which would not be the case with the deep-merge strategy (different keys/locations for overrides/resolutions). Probably not a big issue, but mentioning it nonetheless.

[gossi]

I could be convinced of a deep merge. I have also long thought a hook that is passed the package.json and allows users to do what they will would good.

I like that idea. Usually other configs look like: "bring us the config as object or use a function to have full control over this". Putting this into some types with what bert said:

interface EmberTryConfig {
  packageJson: object | (packageJson: object) => object;
  rootPackageJson: object | (packageJson: object) => object;
  // ...
}

something around these lines?

[gossi]

I made a sample PR to gain some experience, see #1064

Here is what I learned:

1. ember-try carries on very much history similar to ember itself. As such it is designed for prime v1 addon use-case. Workspaces didn't exist when the design was made.
2. Supported package managers are manually integrated. Today, you'd rely more on npm packages that provide support for the needed functions and as such cover all package managers (higher maintenance)
3. Multiple package managers? I've never seen that. The only case I was aware, when on CI npm was used to install pnpm -> then two lockfiles were present (not necessarily the supported use-case to me?)
4. Since monorepos/workspace support is missing, some of the functionality isn't supportable (such pnpm overrides for worspace root).
5. It supports single-v2-addon repo. monorepos with many v2 addons are tricky in support or take more time in execution (given each of them has their own test-app).
6. I also looked into try-or-die. Looks much more modern and is less bound to ember (but efficient minimal compatibility layer in there, that I like).
7. Changing files for scenarios isn't supported in ember-try (does try-or-die support it?) - but is a becoming more and more needed?

In the long run, I think this tool is super useful for many packages/frameworks and can become framework agnostic. It plays a vital role of tools developed within the ember community (I'm having release-plan and scenario-tester in mind here, too).
To me, I saw ember-try more as a pre warn system - but than I also realized its backwards compatibility check, dunno if these are better words to go by in teaching it (outside of ember).

A compat tester for backwards compatibility and pre warn system for new releases

I think the future lies in its agnostic nature.

But to address the current needs with pnpm and v2 addons (such my initial case), some practical approach is needed (something as my PR?).

wdyt?

[bertdeblock]

I think we can tackle merging data into package.json, and overriding existing / adding new files, using the same new functionality.

Instead of a packageJson key, we could introduce a new files key, which has the following type:

files: { [filePath: string]: string | FileHandler };

type FileHandler = (content: string) => string | Promise<string>;

The key would be the path to the existing / new file, relative to the cwd.
The value would be the new content, or a sync / async function that returns the new content.
If the value is a function, it will also receive the current content as an argument.

The returned content will be used to override the existing / create a new file.

For merging data into existing JSON files, ember-try could ship a utility function that returns a FileHandler, which does the "heavy" lifting for the user.

Something like:

import { fileHandlers } from "ember-try";

{
  name: "ember-beta",
  files: {
    "package.json": fileHandlers.mergeJson({
      devDependencies: {
        "ember-source": "ember-source@beta",
      },
    }),
    "../package.json": fileHandlers.mergeJson({
      pnpm: {
        overrides: {
          "ember-source": "ember-source@beta",
        },
      },
    }),
  },
},

And this ^ would also cover pnpm overrides for example.

Probably needs some additional design work though, but sharing this just in case.

[gossi]

uuuuh. I really like that idea. Also allows the provided fileHandlers to grow overtime (maybe some codemod utils sound handy).

This sounds like an entire new codebase - ideas how to proceed here? I still have in mind this to be agnostic and be something as compat-tester or so?

[bertdeblock]

I did a quick spike yesterday in try-or-die. Made a quick WIP PR if you are interested => bertdeblock/try-or-die#1.

[kategengler]

I feel strongly that needing to modify files for try runs is a smell. I think we should only support the simplest modifications and not support codemods. If you need that functionality you should be using multiple test apps.

[kategengler]

ember-try carries on very much history similar to ember itself. As such it is designed for prime v1 addon use-case. Workspaces didn't exist when the design was made.

ember-try supports yarn workspaces and we just have not properly fixed support for pnpm.

Supported package managers are manually integrated. Today, you'd rely more on npm packages that provide support for the needed functions and as such cover all package managers (higher maintenance)

I prefer that we test any package managers we support. We're discussing this approach for ember-cli, but ember-try is more coupled to the behavior of package manages.

Multiple package managers? I've never seen that. The only case I was aware, when on CI npm was used to install pnpm -> then two lockfiles were present (not necessarily the supported use-case to me?)

This dates to when we had bower and npm. @bertdeblock has been refactoring ember-try to modernize.

It supports single-v2-addon repo. monorepos with many v2 addons are tricky in support or take more time in execution (given each of them has their own test-app).

It should work fine with many v2 addons ... I'm not sure how you can preserve the validity of tests and not increase execution time by addons x test-apps. You could absolutely have a single test app testing multiple v2 addons to improve time.

I also looked into try-or-die. Looks much more modern and is less bound to ember (but efficient minimal compatibility layer in there, that I like).

@bertdeblock has been doing the hard work to modernize ember-try but keep the niceties that ember-try has.

In the long run, I think this tool is super useful for many packages/frameworks and can become framework agnostic. It plays a vital role of tools developed within the ember community (I'm having release-plan and scenario-tester in mind here, too).
To me, I saw ember-try more as a pre warn system - but than I also realized its backwards compatibility check, dunno if these are better words to go by in teaching it (outside of ember).

It could have been a generic tool for a long time, but was coupled to ember-cli's commands. It also has had some assumptions traditionally based on Ember apps. Again, @bertdeblock is working to decouple it from ember-cli and make it a generic tool.

[gossi]

I feel strongly that needing to modify files for try runs is a smell. I think we should only support the simplest modifications and not support codemods. If you need that functionality you should be using multiple test apps.

Doesn't matter that much as people are free to write their own configs - this libraries job is to manage the run itself.

It could have been a generic tool for a long time, but was coupled to ember-cli's commands. It also has had some assumptions traditionally based on Ember apps. Again, @bertdeblock is working to decouple it from ember-cli and make it a generic tool.

Yay, @bertdeblock did an amazing job here. As much as ember itself is able to use more and more agnostic tools and libraries, so we should be encouraged to write them as such.

I did a quick spike yesterday in try-or-die. Made a quick WIP PR if you are interested => bertdeblock/try-or-die#1.

Tadaaaa - that's almost it. Fantastic job 👏