How to ensure Bevy doesn't break due to dependency updates

[mockersf]

It happened several times: a dependency breaks semver (as understood by Cargo), and this breaks Bevy main, and sometimes the released version. When this happens, we have basically two ways forward:

• Contact the dependency maintainer for a yank or a new patch fixing back semver
• Patch Bevy to fix the dependency to a known good version, and potentially release a patch if needed for the released version

To avoid getting into this situation, we could:

Fix all dependencies on main

Fix all dependencies to be exact versions. Dependabot will open PRs to update for every dependency update.

Currently, Dependabot runs weekly and can have a maximum of 5 PRs ongoing at the same time.

Pros:

• Not possible to break main
• Not possible to break released versions
• Released versions are tested during development with the exact same dependencies

Cons:

• A lot more PRs from dependabot
• No automatic updated on main for patch dependencies
• May increase the number of duplicate dependencies if not careful about updates
• Confusing cargo error message in case of conflicting versions

Fix all dependencies in released versions

Before a release, setup a branch, fix all dependencies in that branch, then release from the branch.

Pros:

• Not possible to break released versions
• No change for development branches

Cons:

• More complicated release process
• Releases may go out with an "untested" set of exact dependencies. By untested, I mean that someone has manually run some code with it
• Confusing cargo error message in case of conflicting versions

Keep as is

Keep as it is currently, handle issues as they happens.

Pros:

• Nothing changes
• Semver updates happens automatically

Cons:

• Occasional issue needing a patch or discussion with a dependency
• It is possible for a dependency to introduce a bug in an update

So... what do people think?

[DJMcNab]

What are the cases where this has occured? Obviously, we've had the notify issue most recently, but that was because we were depending on a pre-release version, so semver isn't applicable. I'm slightly surprised cargo lifted the version up for us in that one, to be honest.

I think we should always ensure that we're locked into semver compatible versions. So we should pin for notify, but for crates with other releases we should be good players in the ecosystem and allow updates.

[mockersf]

It happened three times:

• notify - we depend on a prerelease, and that has no stability guarantee... maybe cargo should handle that has breaking?
• gpu-alloc - If I remember correctly, they were not following semver at all but versions from something else, but changed since. The version was yanked
• const-random - Introduced a new feature that needed nightly, breaking build on stable. The version was yanked and a new version was published

I think I remember a fourth but can't find it

[cart]

I think there was a serde breakage at one point.

[mockersf]

fourth was rspirv

[bjorn3]

Fixing all dependencies will prevent users from deliberately using newer versions of said dependencies or adding dependencies that specify a dependency on a newer, but still semver compatible, version of those dependencies. Cargo allows multiple non-semver compatible versions of a crate, but when two versions are semver compatible, cargo will force one of those versions to be used. If one of them is pinned and the other is newer or pinned, cargo will give an error message that it couldn't resolve the versions. This error message is confusing to pretty much everyone who doesn't know about this rule and is only fixable by patching the crate that pinned the dependency version to no longer pin the dependency version.

[bjorn3]

By the way this specific case was caused by using a pre-release version of a crate as dependency. Cargo doesn't prevent a breaking upgrade for them. In the 6 years I have used rust, I think I only had a problem with dependency updates once. In this case it was not even a dependency introducing a breaking change, but a crate depending on a version before a new api it used was introduced. This worked fine with the Cargo.lock in the repo of that project, but not in my repo as I had an older version of said dependency. -Zminimal-versions can be used to detect this kind of problem. (I used it in #2460 to find two dependencies that had a too old version specified)

[mockersf]

This error message is confusing

Oh how I hate this error message. That's definitely a "cons" to add!

[cart]

Yeah I think this boils down to a "which is worse" question:

• Increasing the potential for compatibility issues between bevy crates and other crates (bevy needs X.Y.Z and the other crate only allows X.Y.(Z+1)`
• The breakage scenarios outlined here: How to ensure Bevy doesn't break due to dependency updates #2544 (comment)

[minecrawler]

I an not a fan of fixing versions inside a library or framework (and that is what Bevy is for now). Several other projects also share this opinion, for example in NPM it is impossible to publish a package-lock file for modules, which fixes the versions used.

That would leave option 3. I'd say that such breakages do not happen too often. For "notify" there was no guarantee to begin with, so mentioning it does not help at all. Bevy should build on released dependencies or encourage authors to do a stable release and follow semver - maybe providing help if necessary to complete missing pieces.

The fourth one if I have the same one in mind as you (I don't remember which one it was either) afair followed an external version scheme (of a spec or something?) and changed to use it's own versioning by now.

So there are two incidents, which can be caught by running CI regularly. I'd welcome a versioning, which usually allows for semver compatibility, but has the ability to prevent bad versions (using ranges with "multiple requirements"). That would allow Bevy to prevent known bad versions from being used, while also keeping the ability for users to depend on a newer version.

As far as that error message is concerned, maybe the Bevy CLI could include tooling to make sure that the correct versions are installed by following best practices for checking dependencies?

[cart]

Just to be clear: the latest breakage isn't an isolated incident. I don't have links off the top of my head, but "dependencies breaking semver" has broken us 5-ish times in Bevy's short lifespan. We have a relatively large dependency tree (which unlike some people, I am proud of ... I don't want to live in a world where everyone reinvents the wheel or where projects pretend they can write literally everything better than "the commons" as a whole). However this means that we are at the whims of our dependencies not breaking semver for every released version of bevy for the rest of time.

If we don't lock our dependencies, that means that some crate in Bevy 0.3's tree could break semver years down the line. That means that unless you are on a current version of Bevy, your projects might just stop working one day. To me that is completely unacceptable. If you build something in Bevy, it should continue to work in perpetuity. Therefore we either need to commit to fixing all dependency semver issues for all bevy releases for the rest of time (and have a good way of detecting this proactively), or we need to lock our dependencies to a specific version on each release.

Additionally, for teams developing a product on the current Bevy releases, this creates a situation where a "stable" bevy release could be broken by sloppiness in another project we don't control. Imagine you have a team crunching to release a game within the next day or so, and suddenly you can't build your game anymore because external_dependency_x made a mistake when pushing out a patch release. Suddenly everyone needs to drop everything to work around the problem (by patching an internal bevy version, hand-fixing lock files, or by driving the fix upstream and hoping we respond in time). Why expose ourselves and our users to that risk?

[bjorn3]

Imagine you have a team crunching to release a game within the next day or so, and suddenly you can't build your game anymore because external_dependency_x made a mistake when pushing out a patch release.

Executables should always check in Cargo.lock into the vcs used. This way dependency updates can only happen when explicitly requested using cargo update or when necessary to satisfy the version requirements of a dependency you explicitly added since the last build.

[bjorn3]

Maybe we could have an bevy_pin_deps crate that pins all dependencies of cargo. You could then manually add this crate as dependency to prime a working Cargo.lock and remove it again when you add a dependency that requires one of the pinned deps to be of a newer version or immediately when you get a working Cargo.lock.

[cart]

Hmm yeah that is a fair point. As long as the cargo.lock is checked in, "old" binary projects will continue to function. So this only really affects "new" binary projects (which will generally use new / more likely to be supported versions) and projects trying to update their dependencies.

[cart]

Maybe we could have an bevy_pin_deps crate that pins all dependencies of cargo. You could then manually add this crate as dependency to prime a working Cargo.lock and remove it again when you add a dependency that requires one of the pinned deps to be of a newer version or immediately when you get a working Cargo.lock.

Yeah thats a clever solution to the "starting a new / not locked project on an old version that has broken dependencies" problem. I'm less freaked out about the situation now. Thanks 😄

[cart]

bevy_pin_deps feels like something that we don't need today, but feels like a courtesy we should provide once we start doing "stable" bevy releases (1.0.0 and up).