sum-up: things older java versions are blocking

[hpvd]

To get an overview which things we are missing with staying using/being compatible to older Java versions,
and determine the right point of time when its maybe worth to drop one or find a way to work around it,
I like to start this list.

needs java 21 (LTS):

• Lucene v10, https://lucene.apache.org/core/10_0_0/changes/Changes.html
  highlights
  from https://lucene.apache.org/core/corenews.html#apache-lucenetm-1000-available
  • A new IndexInput#prefetch API has been added, allowing query evaluation
    logic to let the Directory know about regions of data that are about to be
    read. This helps perform I/O concurrently under the hood. MMapDirectory
    implements this API using the madvise system call and the MADV_WILLNEED
    flag on Linux and Mac OS.
  • Lucene now supports sparse indexing on doc values via
    FieldType#setDocValuesSkipIndexType. The sparse index will record the
    minimum and maximum values per block of doc IDs. Used in conjunction with
    index sorting to cluster similar documents together, this allows for very
    space-efficient and CPU-efficient filtering.
  • Search concurrency is now decoupled from the index geometry, so that an
    index can be searched using any number of threads, regardless of its number
    of segments.
  • Kmeans clustering on vectors

issue to upgrade lucene to v10.x #14354

needs java 22:

• Use Foreign Memory API, see Use Foreign Memory API, introduced in Java 22 #12809

Please add your knowledge/findings to the comments so we can update this list.
Maybe we can also highlighting some details why supporting it may be worth it (beside always happening bug and security fixes)

[gortiz]

You can add #12809 (although it requires Java 22).

IMHO the target should not be to drop Java < 21 support but start using multi-release jars to be able to provide features that are only available in specific runtimes.

[gortiz]

Also notice that it may be acceptable to increase the minimum Java runtime version for Pinot processes (servers, brokers, controllers and minions), but increasing the minimum version of the Pinot driver is more sensible.

The Pinot driver build process is pretty coupled with Pinot processes. Due to the way the driver is being built, it depends on some core projects of Pinot processes. We in fact compile these modules twice: Once with Java 11 and one with Java 8. Given that these two Java versions are very close in terms of syntax there is usually no problem. If we end up moving the services code to Java 21 that won't be the case.

So any plan to migrate Pinot services to a newer version would need to analyze the impact in the driver. Probably we would need to split the driver code from the services code or, in a more ambitious way, try to implement a third party protocol (mysql or postgres) and use their drivers.