Arrow Flight Server bootstrap logic (#16962)

* Arrow Flight Server bootstrap logic * new plugin for StreamManager implementation * integration with server module * support for SslContext in Flight server and client * ClientManager for creating a pool of flight clients for data nodes * custom event loop group and thread pool for server and client channel Signed-off-by: Rishabh Maurya <[email protected]> * interim changes - integration with Auxiliary Transport Signed-off-by: Rishabh Maurya <[email protected]> * changes to use grpc-netty-shaded Signed-off-by: Rishabh Maurya <[email protected]> * Update javadoc Signed-off-by: Rishabh Maurya <[email protected]> * fix the shaded dependencies Signed-off-by: Rishabh Maurya <[email protected]> * Move arrow-flight-rpc from module to plugin Signed-off-by: Rishabh Maurya <[email protected]> * remove unnecessary imports Signed-off-by: Rishabh Maurya <[email protected]> * rebase fixes Signed-off-by: Rishabh Maurya <[email protected]> * Fix permissions and other runtime issues Signed-off-by: Rishabh Maurya <[email protected]> * Remove StreamManagerWrapper and Node.java changes from PR Signed-off-by: Rishabh Maurya <[email protected]> * Fix permissions for test Signed-off-by: Rishabh Maurya <[email protected]> * remove testGetFlightClientLocationExecutionError as thread interruption was causing client close failure Signed-off-by: Rishabh Maurya <[email protected]> * Fix the issue with single node ClientManager Signed-off-by: Rishabh Maurya <[email protected]> * Fix flight server integ test on unix machine Signed-off-by: Rishabh Maurya <[email protected]> * suppress JSM removal warning Signed-off-by: Rishabh Maurya <[email protected]> * Fix security policy and FlightClientManagerTests Signed-off-by: Rishabh Maurya <[email protected]> * remove StreamManagerWrapper from the PR Signed-off-by: Rishabh Maurya <[email protected]> * Set multi-release in manifest while shadowing arrow-memory-shaded Signed-off-by: Rishabh Maurya <[email protected]> * Disable jacocoReport for shaded projects Signed-off-by: Rishabh Maurya <[email protected]> * Remove multi version classes from arrow-memory-shaded Signed-off-by: Rishabh Maurya <[email protected]> * Address the PR comments Signed-off-by: Rishabh Maurya <[email protected]> * Move the arrow-memory-shaded and flight within flight-rpc plugin Signed-off-by: Rishabh Maurya <[email protected]> * Move the arrow-memory-shaded and flight within flight-rpc plugin Signed-off-by: Rishabh Maurya <[email protected]> * Detach SPI from Apache Arrow (not needed at the moment), drop all shaded libs (not needed at the moment) Signed-off-by: Andriy Redko <[email protected]> * Rebase and other minor refactoring Signed-off-by: Rishabh Maurya <[email protected]> * Address PR comments - majorly move away from grpc-netty-shaded Signed-off-by: Rishabh Maurya <[email protected]> * remove arrow flight, client from codecov as the package is non opensearch is just for overriding purpose Signed-off-by: Rishabh Maurya <[email protected]> * change compileOnly to implementation dep for arrow-spi Signed-off-by: Rishabh Maurya <[email protected]> * Rebase from main and fixes related to netty version bump Signed-off-by: Rishabh Maurya <[email protected]> * Simplify cloning and overriding logic for FlightServer and FlightClient Signed-off-by: Rishabh Maurya <[email protected]> * Only clone FlightClient::Builder class Signed-off-by: Andriy Redko <[email protected]> * Only clone FlightServer::Builder class Signed-off-by: Andriy Redko <[email protected]> * Update min supported version to 3.0.0 Co-authored-by: Andriy Redko <[email protected]> Signed-off-by: Rishabh Maurya <[email protected]> * Fix java security permission issue Signed-off-by: Rishabh Maurya <[email protected]> * Address PR comments Signed-off-by: Rishabh Maurya <[email protected]> * Fix netty system properties Signed-off-by: Rishabh Maurya <[email protected]> * Move flight service and other components of flight-rpc-plugin behind feature flag Signed-off-by: Rishabh Maurya <[email protected]> * remove system property value set numDirectArenas Signed-off-by: Rishabh Maurya <[email protected]> --------- Signed-off-by: Rishabh Maurya <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Co-authored-by: Andriy Redko <[email protected]> Co-authored-by: Andriy Redko <[email protected]>
opensearch-project · Feb 20, 2025 · abe2333 · abe2333
1 parent 636dea4
commit abe2333
Show file tree

Hide file tree

Showing 122 changed files with 7,987 additions and 386 deletions.
diff --git a/CHANGELOG-3.0.md b/CHANGELOG-3.0.md
@@ -17,6 +17,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Added pull-based Ingestion (APIs, for ingestion source, a Kafka plugin, and IngestionEngine that pulls data from the ingestion source) ([#16958](https://github.com/opensearch-project/OpenSearch/pull/16958))
 - Added ConfigurationUtils to core for the ease of configuration parsing [#17223](https://github.com/opensearch-project/OpenSearch/pull/17223)
 - Add execution_hint to cardinality aggregator request (#[17312](https://github.com/opensearch-project/OpenSearch/pull/17312))
+- Arrow Flight RPC plugin with Flight server bootstrap logic and client for internode communication ([#16962](https://github.com/opensearch-project/OpenSearch/pull/16962))
 
 ### Dependencies
 - Update Apache Lucene to 10.1.0 ([#16366](https://github.com/opensearch-project/OpenSearch/pull/16366))

diff --git a/codecov.yml b/codecov.yml
@@ -4,6 +4,7 @@ codecov:
 ignore:
   - "test"
   - "benchmarks"
+  - "plugins/arrow-flight-rpc/**/org/apache/arrow/flight/**"
 
 coverage:
   precision: 2

diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.toml
@@ -82,7 +82,7 @@ opentelemetry         = "1.46.0"
 opentelemetrysemconv  = "1.29.0-alpha"
 
 # arrow dependencies
-arrow                 = "17.0.0"
+arrow                 = "18.1.0"
 flatbuffers           = "2.0.0"
 
 [libraries]

diff --git a/libs/arrow-spi/build.gradle b/libs/arrow-spi/build.gradle
@@ -10,79 +10,11 @@
  */
 
 testingConventions.enabled = false
+
 dependencies {
   api project(':libs:opensearch-core')
-  api "org.apache.arrow:arrow-vector:${versions.arrow}"
-  api "org.apache.arrow:arrow-format:${versions.arrow}"
-  api "org.apache.arrow:arrow-memory-core:${versions.arrow}"
-  runtimeOnly "org.apache.arrow:arrow-memory-netty-buffer-patch:${versions.arrow}"
-  runtimeOnly "org.apache.arrow:arrow-memory-netty:${versions.arrow}"
-  runtimeOnly "io.netty:netty-buffer:${versions.netty}"
-  runtimeOnly "io.netty:netty-common:${versions.netty}"
-
-  runtimeOnly "com.google.flatbuffers:flatbuffers-java:${versions.flatbuffers}"
-  runtimeOnly "org.slf4j:slf4j-api:${versions.slf4j}"
-  runtimeOnly "com.fasterxml.jackson.core:jackson-databind:${versions.jackson}"
-  api "com.fasterxml.jackson.core:jackson-annotations:${versions.jackson}"
-
-  implementation "commons-codec:commons-codec:${versions.commonscodec}"
 }
 
 tasks.named('forbiddenApisMain').configure {
   replaceSignatureFiles 'jdk-signatures'
 }
-
-tasks.named('thirdPartyAudit').configure {
-  ignoreMissingClasses(
-    // Logging frameworks
-    'org.apache.commons.logging.Log',
-    'org.apache.commons.logging.LogFactory',
-    'org.apache.log4j.Level',
-    'org.apache.log4j.Logger',
-    'org.slf4j.impl.StaticLoggerBinder',
-    'org.slf4j.impl.StaticMDCBinder',
-    'org.slf4j.impl.StaticMarkerBinder',
-
-    // Reactor BlockHound
-    'reactor.blockhound.BlockHound$Builder',
-    'reactor.blockhound.integration.BlockHoundIntegration'
-  )
-
-  ignoreViolations(
-    "io.netty.util.internal.PlatformDependent0",
-    "io.netty.util.internal.PlatformDependent0\$1",
-    "io.netty.util.internal.PlatformDependent0\$2",
-    "io.netty.util.internal.PlatformDependent0\$3",
-    "io.netty.util.internal.PlatformDependent0\$4",
-    "io.netty.util.internal.PlatformDependent0\$6",
-    "io.netty.util.internal.shaded.org.jctools.queues.BaseLinkedQueueConsumerNodeRef",
-    "io.netty.util.internal.shaded.org.jctools.queues.BaseLinkedQueueProducerNodeRef",
-    "io.netty.util.internal.shaded.org.jctools.queues.BaseMpscLinkedArrayQueueColdProducerFields",
-    "io.netty.util.internal.shaded.org.jctools.queues.BaseMpscLinkedArrayQueueConsumerFields",
-    "io.netty.util.internal.shaded.org.jctools.queues.BaseMpscLinkedArrayQueueProducerFields",
-    "io.netty.util.internal.shaded.org.jctools.queues.LinkedQueueNode",
-    "io.netty.util.internal.shaded.org.jctools.queues.MpmcArrayQueueConsumerIndexField",
-    "io.netty.util.internal.shaded.org.jctools.queues.MpmcArrayQueueProducerIndexField",
-    "io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueueConsumerIndexField",
-    "io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueueProducerIndexField",
-    "io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueueProducerLimitField",
-    "io.netty.util.internal.shaded.org.jctools.util.UnsafeAccess",
-    "io.netty.util.internal.shaded.org.jctools.util.UnsafeLongArrayAccess",
-    "io.netty.util.internal.shaded.org.jctools.util.UnsafeRefArrayAccess",
-    "io.netty.util.internal.shaded.org.jctools.queues.unpadded.MpscUnpaddedArrayQueueConsumerIndexField",
-    "io.netty.util.internal.shaded.org.jctools.queues.unpadded.MpscUnpaddedArrayQueueProducerIndexField",
-    "io.netty.util.internal.shaded.org.jctools.queues.unpadded.MpscUnpaddedArrayQueueProducerLimitField",
-    "org.apache.arrow.memory.ArrowBuf",
-    "org.apache.arrow.memory.util.ByteFunctionHelpers",
-    "org.apache.arrow.memory.util.MemoryUtil",
-    "org.apache.arrow.memory.util.MemoryUtil\$1",
-    "org.apache.arrow.memory.util.hash.MurmurHasher",
-    "org.apache.arrow.memory.util.hash.SimpleHasher",
-    "org.apache.arrow.vector.BaseFixedWidthVector",
-    "org.apache.arrow.vector.BitVectorHelper",
-    "org.apache.arrow.vector.Decimal256Vector",
-    "org.apache.arrow.vector.DecimalVector",
-    "org.apache.arrow.vector.util.DecimalUtility",
-    "org.apache.arrow.vector.util.VectorAppender"
-  )
-}
diff --git a/libs/arrow-spi/licenses/arrow-format-17.0.0.jar.sha1 b/libs/arrow-spi/licenses/arrow-format-17.0.0.jar.sha1
diff --git a/libs/arrow-spi/licenses/arrow-memory-core-17.0.0.jar.sha1 b/libs/arrow-spi/licenses/arrow-memory-core-17.0.0.jar.sha1
diff --git a/libs/arrow-spi/licenses/arrow-memory-netty-17.0.0.jar.sha1 b/libs/arrow-spi/licenses/arrow-memory-netty-17.0.0.jar.sha1
diff --git a/libs/arrow-spi/licenses/arrow-memory-netty-buffer-patch-17.0.0.jar.sha1 b/libs/arrow-spi/licenses/arrow-memory-netty-buffer-patch-17.0.0.jar.sha1
diff --git a/libs/arrow-spi/licenses/arrow-vector-17.0.0.jar.sha1 b/libs/arrow-spi/licenses/arrow-vector-17.0.0.jar.sha1
diff --git a/libs/arrow-spi/licenses/jackson-databind-LICENSE.txt b/libs/arrow-spi/licenses/jackson-databind-LICENSE.txt
diff --git a/libs/arrow-spi/licenses/jackson-databind-NOTICE.txt b/libs/arrow-spi/licenses/jackson-databind-NOTICE.txt