Lockfile assembly, to avoid these PR conflicts

[wbern]

Starting a discussion here based on #4322.

My organisation (13 teams, around 110+ monorepo modules and 160+ modules in another repo), are asking us to prioritize this, so I hope we can resolve this. I will direct them to this thread as well, but hope the rush community can pitch in as well.

[zkochan]

I understand that it is a big issue. I was thinking about it a lot and I think the only reliable solution would be to not commit the lockfile and to use some storage to upload/fetch it.

We can use the specs from package.json files to create a key. And with that key we can read the lockfile from a cache. But I am not sure how would we then update the lockfile? And if we update it, will it update for all environments? Seems nondeterministic.

Maybe there are other solutions to get rid of the lockfile. But I don't think it is possible to change the format of the lockfile in such a way that conflicts will forever be eliminated.

cc @pnpm/collaborators

[shellscape]

We see this occasionally in Rollup and with private repos that resemble the size the OP is speaking of. Whenever we get a conflict we just run pnpm install at the root and it's taken care of. I don't understand the use case where that's not sufficient. Sure, it can be a momentary inconvenience, but it's not that big a deal since pnpm is so fast, and it only happens on branches that fall significantly behind master.

[gluxon]

Whenever we get a conflict we just run pnpm install at the root and it's taken care of. I don't understand the use case where that's not sufficient.

I think you're right. In most cases this is sufficient.

From the developer experience perspective though, it's simply frustrating when someone gets the merge conflict prompt and has to wait an additional 30 minutes for a CI pipeline they otherwise wouldn't have to wait for.

We're using a lot of caches on CI too, so these builds end up being slower because the merge conflicts force you to rebase on the tip of develop instead of the last cached commit.

Sure, it can be a momentary inconvenience, but it's not that big a deal since pnpm is so fast, and it only happens on branches that fall significantly behind master.

pnpm is indeed fast, but we just have other parts of CI that aren't. 😞

Perhaps we're doing some silly non-optimal things, but this does happen to us on branches that fall a couple hours behind if they touch the lock file.

[gluxon]

How often does this happen due to dependencies that are expected to be at a consistent version across all packages in a monorepo?

Edit 2022-03-26: The idea below has been formalized more thoroughly at #4483

Elaborating on my team specifically

We're a little over 700 packages in the monorepo.

On my team, I'd estimate about 20% of pull requests hit pnpm-lock.yaml merge conflicts due to this. I'd like to elaborate a bit more about our setup, because I suspect it's actually quite common.

The large frontend monorepo my team works on supports a backend product split into several microservices each exposing their own TypeScript API. Whenever these backend services release a new version, the pull request updating to the new version ripples through and forces a rebase on every PR making a dependency change line-adjacent to it. We're using GitHub Enterprise, so custom merge drivers aren't available.

Here's where we're doing something that may be unique. The package.json files themselves don't see a merge conflict since we're abusing pnpm.overrides to set the version across all packages in the monorepo. We know pnpm.overrides wasn't really meant for this purpose.

Here's a shortened example of our lockfile.

overrides:
  '@backend/service-a': 1.0.0
  '@backend/service-b': 1.0.0

projects:

  packages/foo:
    specifiers:
      '@backend/service-a': *
      '@backend/service-b': *
      react: ^17.0.0
    dependencies:
      '@backend/service-a': 1.0.0
      '@backend/service-b': 1.0.0
      react: 17.0.1

Backend developers are constantly adding new functionality, and frontend developers are updating their dependency declarations for the new API definitions. You can imagine the following changes merge conflicting with at least one other change in the list:

• Update @backend/service-a to 1.0.1.
• Update @backend/service-b to 1.0.1.
• Add a new @backend/service-c to foo.
• Remove @backend/service-a from foo.
• Add a new project bar depending on @backend/service-a or @backend/service-b
• Update react to 17.0.2.

My Proposal

Perhaps consistent dependencies across a monorepo could be a first-class pnpm feature.

This would be a huge effort, but it would eliminate every merge conflict scenario listed above. I would only consider this if we think many pnpm projects would benefit from this, not just the one I work on.

Here's what it would look like:

The root package.json would specify the consistent versions.

{
  "name": "some-monorepo",
  "pnpm": {
    "workspaceConsistent": {
      "@backend/service-a": "1.0.0",
 
      "@backend/service-b": "1.0.0",
 
      "react": "17.0.1"
    }
  }
}

Projects within the monorepo would use a new protocol.

{
  "name": "foo",
  "dependencies": {
    "@backend/service-a": "workspace-consistent",
    "@backend/service-b": "workspace-consistent",
    "react": "workspace-consistent"
  }
}

The pnpm-lock.yaml file would look like:

# Spaces after each line are intentional to help git auto merge version changes.
'workspace-consistent':
  '@backend/service-a': 1.0.0

  '@backend/service-b': 1.0.0
  
  react: 17.0.1

projects:

  packages/foo:
    specifiers:
      '@backend/service-a': 'workspace-consistent'
      '@backend/service-b': 'workspace-consistent'
      react: 'workspace-consistent'
    dependencies:
      '@backend/service-a': 'workspace-consistent'
      '@backend/service-b': 'workspace-consistent'
      react: 'workspace-consistent'

The caveats are:

• The consistent versions in this proposal are exact versions. I'm not sure how necessary supporting standard version ranges would be, and imagine that to be significantly more difficult to implement.
• This would likely require a new lockfile version bump.
• Monorepos would have to upgrade and configure their consistent versions to reduce lockfile merge conflicts.
• Probably other implementation or maintenance difficulty I'm not realizing.

Here's why I think every frontend project would benefit from this, not just my team.

• I've listed react as an example here because I think most frontend monorepos should be declaring it at a consistent version. Multiple copies of React in a frontend bundle can cause hooks to break. There are edge cases where you might have an isolated runtime plugin on a different version of React, but I'd expect that to be rare.
• You generally want the same @types definitions for a package. I've seen a lot of TypeScript assignabilty issues that stemmed from packages using different versions of the same @types package.
• The pnpm.overrides feature (and yarn's resolutions) apply too broadly for this use case. Those overrides apply to the entire dependency graph. I'm not aware of a way to make it apply only the dependency declarations in the monorepo. For example, if you accidentally install a new dependency requiring react@18, you won't know if pnpm.overrides.react is set to ^17.0.0.

I'd happily volunteer my weekends to work on this if others would find it valuable. I really adore pnpm. ❤️

Questions

• Would this be helpful to anyone else?
• Are there glaring implementation difficulties?

[wbern]

It sounds like you're after the same thing as me, basically, avoid to cause conflicts if packages only got new dependencies added to them that is already used in the repo by other packages.

Am I getting it right?

[gluxon]

It sounds like you're after the same thing as me, basically, avoid to cause conflicts if packages only got new dependencies added to them that is already used in the repo by other packages.

I think so, but to make sure: What sort of change does that merge conflict with in your repo?

If someone adds a new dependency at the same version as another package the monorepo, I know for us that merge conflicts with:

• A separate PR that attempts to upgrade that dependency version in all the other packages.
• A separate PR that adds a different dependency at the same line in package.json. Although that would be a package.json conflict rather than a pnpm-lock.yaml conflict. We require our package.json entries to be sorted, so this ends up not being very common. New dependencies always getting added to the end of dependencies or devDependencies would merge conflict.

The workspace-consistent protocol would prevent the first case from merge conflicting.

[wbern]

Those cases are expected conflicts for me as well, that I don't mind.

Ok, now I understand what you mean, but don't you still want to ensure that major bumps don't break things in the app with "workspace-consistent" in it?

[gluxon]

Ok, now I understand what you mean, but don't you still want to ensure that major bumps don't break things in the app with "workspace-consistent" in it?

We do, but a lot of things merge conflict for minor version bumps too.

Those cases are expected conflicts for me as well, that I don't mind.

What are the unexpected conflicts arising from adding new dependency declarations that other packages are already using that you do mind? I'm curious what lines in pnpm-lock.yaml you may see the conflict for.

[gluxon]

Formalized this idea more in a new discussion #4483. Added examples of the merge conflicts that can happen without this idea.

[gluxon]

I think I'm also seeing some value in the "lockfile assembly" mentioned? If each entry in the packages block were a file in a directory, that would reduce merge conflicts in that section quite a bit since git won't care about ordering.

I'm sure there's a performance cost of reading that many files from disk though.

[wbern]

Yes, but also since we do appreciate lockfiles creating stability, maybe the separate lockfile can be simply named after the branch name? That means that people won't have new versions popping up in their lockfiles all the time (kind of like the npm dark ages before lock files were a thing).

Then like you say, in main we introduce a workflow that simply deletes lockfiles that aren't the main one, after we have merged them successfully.

I think this idea makes sense. The only thing we have to account for is, what if the lockfiles are incompatible? Could it happen? What would we do?

[chengcyber]

I see people migrated their repo into Rush.js and then encounter lockfile conflicts issue, and then creating their own tool chain to dedicate lockfiles per project, and then switch to NO toolchain, just run yarn for each project...

And yarn/berry users are facing the same challenge yarnpkg/berry#1223

So, i wonder whether dedicated lockfile per project is viable even though its complexity.

[zkochan]

@chengcyber there are a couple of bottlenecks with dedicated lockfiles.

Currently pnpm creates a single virtual store (node_modules/.pnpm) per workspace. I would not change this. If we will make many virtual stores, the amount of filesystem operations will grow exponentially. The virtual store is tightly coupled with the lockfile and has some limitations. For instance, in a virtual store, a given version of a package is only written once (unless it has peer dependencies). This means that if there is a [email protected] in the dependencies, it will have the same subdependencies in all places. So foo may be a dep of project a and project b. But at both places the dependency foo will be the same. For instance, it will have a [email protected] in dependencies. It is not possible to have [email protected] with [email protected] in the deps at one project, and with [email protected] in another project.

So as a consequence, even if we will create dedicated lockfiles and we update [email protected] to [email protected] in project b. This change will happen in the lockfile of project a as well. So we won't get full isolation. It is probably possible to implement full isolation but I am not sure it is a good idea. It will both add complexity to pnpm and add complexity to projects that will have a lot more variants of dependencies.

[zkochan]

also, there are entries like these for package that have peers:

pnpm/pnpm-lock.yaml

Line 8192 in fa25b10

/eslint-config-standard/16.0.3_2dae9da07848968bcc0999a5e5956768:

This hash is updated when the peer dependencies update. I think this is pretty hard to resolve even with dedicated lockfiles these conflicts will not disappear.

[zkochan]

I think this idea makes sense. The only thing we have to account for is, what if the lockfiles are incompatible? Could it happen? What would we do?

I guess we will just take the lockfile from the branch. And if we need to resolve something new, that wasn't on the branch, then during resolution we will prefer package versions that were present in the "main" lockfile. So the new merged lockfile will only contain packages that were either in the main lockfile or in the branch lockfile. No new package versions will be included.

[ylemkimon]

What if pnpm would not modify the pnpm-lock.yaml file, when on a branch. Instead, it would create a new lockfile (something like pnpm-lock-<datestamp>.yaml). On the main branch, and automatic script would update the main lockfile and remove the lockfile generated on the branch.

This is the first thing that came to mind, something like "incremental lockfile". Instead of datestamp which cannot be determined before merging the commit, I think it can use a random hash and use git to determine the merge order.

[wbern]

@zkochan I would be interested in seeing how we could make branch specific lockfiles work, it also sounds like a less complex thing to add from pnpm perspective, basically allow custom lockfile path?

Merging lockfiles is the more complex one but maybe we should start by solving that one in the user land by simply doing "rush update" on the original lockfile in CI/main? Unless you think there may be a way?

How could I assist in making this happen?

[zkochan]

Yes, I think there should be some setting for custom lockfile name.

For the merge, pnpm should read all the lockfiles from the directory, merge all the fields from packages and run installation using the merged lockfile.

It sounds not hard. If you want to help, you can try to implement.

Here's is the lockfile read: https://github.com/pnpm/pnpm/blob/main/packages/lockfile-file/src/read.ts

[wbern]

@zkochan I will give it a go this week and see how I fare. 🙂

[zkochan]

@wbern a PR is submitted here already: #4475

[wbern]

Great! I only managed to write one failing test before reality caught up..

[Jack-Works]

In our team, we found the pnpm built-in conflict resolution not always work (sometimes failed to resolve and it looks like dropping the whole lockfile to create a new one). We suggest the following steps to resolve lockfile conflicts in our team:

1. Run git checkout main -- pnpm-lock.yaml
2. Run pnpm install

This will force to use the baseline version of the lockfile and "pick" new packages from package.json therefore resolve the conflict without dropping the whole lockfile.

[chengcyber]

Hi @Jack-Works , i run into this issue occasionally. A basic implementation of lockfile assembly at #4475, inviting your ideas.

With lockfile assembly, pnpm-lock.yaml on branch is totally no changes, therefore the conflicts never happens; after MRs are merged into the main branch, another pnpm install reuses and merges some fields from branch lockfiles(`"packages" field for now) to create the correct lockfile, branch lockfiles are removed after that.