Public API removal policy

[ansell]

Is there a policy regarding how many minor versions APIs are kept for before removal within the 2.x series? Ideally public APIs that existed in 2.0 would still be around in 2.8 (per the defacto standard Semantic Versioning spec) but that hasn't happened here so there needs to be a custom policy for Jackson so users can estimate how many minor versions of Jackson 2.x they are compatible with.

In my case, the mass removal of public APIs in the following commit (with an incorrect commit message) has caused some of my code to be restricted to compatibility with Jackson 2.3-2.6, as I am using APIs added in 2.3, while also using APIs removed in 2.7.

f8d8856

In that particular commit, deprecations that occurred in 2.6 were removed in 2.7, which seems very sudden given the Jackson status as a widely used library. Does Jackson treat minor versions (2.6/2.7/etc.) as public API incompatible jumps and hence any public API can be removed in a new minor version? Are public APIs preserved across patch versions, ie, 2.6.0 to 2.6.1 will never have public API removals?

[cowtowncoder]

First of all, it is unfortunate that you experienced a compatibility problem. We try to avoid breaking application, library and framework code. But exactly due to wide usage it is a challenging problem: new functionality being added often requires changes to interfaces between components, and it is difficult to gauge effect of changes.

Versioning goals and intent are briefly explained at:

https://github.com/FasterXML/jackson/wiki/Jackson-Releases

and the basic idea is that the Public API is to be kept backwards compatible for the major release; parts may (and will) be marked as deprecated, but effort is made to keep methods around where they can work. This is not always possible, but exceptions are rare.

Keyword here is "public" API: BytesToNameCanonicalizer is (was) not considered to be part of public API but rather an internal component (and implementation detail). Since a better symbol table implementation was introduced (and used) in 2.6, removal of the old, obsolete and unused implementation was done in 2.7.

So: major versions are not meant to be compatible, and will further live in different packages (i.e. can co-exist, so that 1.x and 2.x, for example do not interact or interfere).
Patch versions will be fully compatible for all interfaces, internal, public, and anything in-between.
Minor versions will try to keep even internal APIs compatible between adjacent versions (2.5<->2.6, 2.6<->2.7), and use deprecation markers. Speed of actual removal of methods is a judgment call, however, based on cost of maintaining old code and possible work-arounds.

I realize that the distinction between public and internal functionality may not be easy to distinguish; and there is also the gray area of sort-of-public, for use via sub-classing, especially for extension modules (datatype, dataformat and other modules).
I am open to improvement ideas.

[ansell]

My particular case was Lf2SpacesIndenter which seemed like it was part of the public API when I used it. It was modified in 2.5.0, fixed to be backwards compatible in 2.5.1 (see issue #178 ), and then removed again in 2.7.0 so instead of asking for it to be added in again, it is more useful to have a clarification of the overall policy at this stage as there are already multiple versions that do not link properly for that legacy code.

If there was a standard way of defining what a public API is using annotations or another method such as an animal-sniffer signatures package, which is generally only used for JDK versions so far, it would be simple to distinguish which parts of jackson-core, such as Lf2SpacesIndenter are actually part of the public API. Naively, without those explicit labels on the public API, one would think that every public class, method, and static variable that didn't look obviously like an implementation detail, will continue to exist, even if they are deprecated. Ways I have used in the past to denote Internal exposed APIs have been to prefix something with "_" or suffix it with "Internal" or "Impl", but any consistently used convention would work.

If public jackson "symbols" that are linked to by various libraries (which is the default definition of a public API if one is not explicitly specified) are removed like in that commit, it restricts the version ranges that can be supported by third parties, potentially removing the possibility of getting security updates in the worst case and causing the usual "JAR hell" in other cases.

In my particular case, support for the Sesame-2.x and 4.x major version series have stopped because the project moved to the Eclipse foundation and there are no more releases happening for it per the Eclipse policy that we have a clean split at the point of moving, leaving users in a state where they cannot ever update their Sesame reliant projects past Jackson-2.6.3. I don't expect that to be a big issue, but they may come to rely on Jackson-2.7+ features through other dependencies before they are able to migrate all of their dependencies to Eclipse RDF4J where that particular issue is fixed. In this case it is either worse or better depending on your interpretation, as the third-party linkage only results in a runtime error, not a compile time error, since the affected Jackson API wasn't exposed on the Sesame Public API.

Having said all of that, it is great to explicitly have the 2 minor versions rule on the Releases wiki page for the API in general, which is better than libraries that often break their internal and public APIs on patch versions (a certain http library being the usual source of these issues for my projects). Thanks for clarifying the situation!

[cowtowncoder]

@ansell Agreed on all parts. Situation with Lf2SpacesIndenter sounds like something were I should have been less aggressive wrt removal. While I want to keep size of jackson-core as small as possible, it should not be done at cost of compatibility.

I also definitely agree wrt it being a significant problem to be locked into specific version (or smallish version range) for transitive dependencies.

One thing I have thought about has been some sort of ability to have larger integration test suites, where it auto-build system could be used to try out new minor versions against libraries that work current latest stable version -- I would love to find out about potential breakage early, and then solve the issue (or, at worst case, at least document).
So far testing has been limited to try to at least work with commonly used frameworks like DropWizard and Spring, but even that takes quite a bit of effort.

Anyway, thank you for your comments and feedback. I hope we can improve compatibility handling going forward.

[ansell]

The issue of upgrading regularly in the future for that project is going to be a little hampered by the Eclipse Legal team who have a policy of reviewing every dependency version update before it is pushed. They don't allow us to get jackson-core/databind/annotations/etc. version "2.x" pre-approved, we had to get jackson-core-2.6.3/etc. specifically approved.

However, I have numerous other projects that piggy-back on jackson and I try to update the dependencies regularly to test that everything is still compatible. The downside there, is that it is limited by the actual runtime paths that those micro-applications actually use, and it happened that I wasn't transitively utilising the broken path in any of those applications so it went unnoticed for a few months.

[cowtowncoder]

Ouch. Did not know that about Eclipse legal team. That is especially bad if it even applies to patch revisions... ideally using the latest patch should be a light-weight and very safe option -- while there has been mixed record for minor versions, I think we have managed to keep patch updates very safe, sometimes at expense of leaving some of incremental improvements to the next minor version. This seems like a reasonable trade-offs if and when it allows timely fixing of truly critical issues.

[ansell]

I think they have a fast track option for some cases, but it still needs a formal tick of approval.