[FLINK-37661] Remove Kryo serializer reference data from version 1.x #26444
Conversation
There was a problem hiding this comment.
I see in the Flip . After this PR can the user still follow the instructions in the Flip to migrate their jobs?
Compatibility, Deprecation, and Migration Plan
Users who are using the Kryo v2 APIs in ExecutionConfig including those specifying custom Kryo v2 serializers, will have to use the Kryo v5 API analogs in addition to the Kryo v2 APIs that they are currently using. When all persisted savepoints + checkpoints have been fully migrated, then they can and should remove usage of the Kryo v2 APIs.
All Flink applications currently generate state with Kryo v2. This PR will fully support reading that state and write newer state with Kryo v5. However, some future version of Flink would eventually drop support for Kryo v2 and users will need to make sure all persisted state is upgraded before upgrading to a future version of Flink that drops support for Kryo v2 serialized state. There will be a migration pathway so that all applications can upgrade without losing state. Applications will have to run on a bridge release version that will read their state with Kryo v2 data and write it with Kryo v5 data before upgrading to a future version of Flink that completely drops support for Kryo v2.
When Flink adds support for Java 17 and newer, that would not support compatibility with persisted Kryo v2 based state. Users who want compatibility with persisted Kryo v2 state would have to use a Java 11 runtime until they have run a bridge version of Flink and migrated all persisted state over to Kryo v5 format. New applications and applications that have fully migrated persisted state would be free to upgrade to Java 17.
|
@davidradl The FLIP was abandoned for a while and it was decided later to fully drop Kryo v2 support (and hence compatibility from 1.x series) in https://lists.apache.org/thread/odhglx8tmpdt6jnorgcsvxjqjfd169x6. |
|
Thanks @X-czh , I see it was agreed to fully drop Kryo v2 support (and hence compatibility from 1.x series). For me it still makes sense to have a migration path, as this will hit many users and potentially be the biggest migration inhibitor. I am willing to implement under your guidance, if this helps - as I feel this is very important. |
|
@Zakelly Last time I heard that there was plan for a state migration tool. How's that going now? If the tool is available, it might not be a big issue to fully drop Kryo v2 support cc @davidradl |
|
@X-czh I'm not working on this and I heard there is no detailed plan for now. It seems that no one is driving that currently. |
Flink 2.0 has done a major Kryo 2.x -> 5.x upgrade and the new Flink is not compatible with older Kryo state. Reference data from version 1.x can be safely deleted as a result.