Druid - Spark interoperation is problematic due to Netty dependency mismatch

[leventov]

Spark task (https://github.com/metamx/druid-spark-batch) which uses Druid fails with AbstractMethodError:

java.lang.AbstractMethodError
	at io.netty.util.ReferenceCountUtil.touch(ReferenceCountUtil.java:73) ~[druid-selfcontained-0.10.1-mmx17.jar:0.10.1-mmx17]
	at io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:107) ~[druid-selfcontained-0.10.1-mmx17.jar:0.10.1-mmx17]
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:810) ~[druid-selfcontained-0.10.1-mmx17.jar:0.10.1-mmx17]
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) [druid-selfcontained-0.10.1-mmx17.jar:0.10.1-mmx17]
	at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:111) ~[druid-selfcontained-0.10.1-mmx17.jar:0.10.1-mmx17]
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) [druid-selfcontained-0.10.1-mmx17.jar:0.10.1-mmx17]
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) [druid-selfcontained-0.10.1-mmx17.jar:0.10.1-mmx17]
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) [druid-selfcontained-0.10.1-mmx17.jar:0.10.1-mmx17]
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) [druid-selfcontained-0.10.1-mmx17.jar:0.10.1-mmx17]
	at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:305) [druid-selfcontained-0.10.1-mmx17.jar:0.10.1-mmx17]
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) [druid-selfcontained-0.10.1-mmx17.jar:0.10.1-mmx17]
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) [druid-selfcontained-0.10.1-mmx17.jar:0.10.1-mmx17]
	at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:38) [druid-selfcontained-0.10.1-mmx17.jar:0.10.1-mmx17]
	at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:1089) [druid-selfcontained-0.10.1-mmx17.jar:0.10.1-mmx17]
	at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1136) [druid-selfcontained-0.10.1-mmx17.jar:0.10.1-mmx17]
	at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1078) [druid-selfcontained-0.10.1-mmx17.jar:0.10.1-mmx17]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [druid-selfcontained-0.10.1-mmx17.jar:0.10.1-mmx17]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) [druid-selfcontained-0.10.1-mmx17.jar:0.10.1-mmx17]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) [druid-selfcontained-0.10.1-mmx17.jar:0.10.1-mmx17]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [druid-selfcontained-0.10.1-mmx17.jar:0.10.1-mmx17]
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) [druid-selfcontained-0.10.1-mmx17.jar:0.10.1-mmx17]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]

It doesn't seem that Spark will upgrade to Netty 4.1 soon, so proposed solution is to isolate Druid's usage of Netty somehow, e. g. via shading.

Modules which currently depend on Netty 4.1 in Druid:

• druid-sql
• druid-rocketmq
• druid-avro-extensions
• druid-services via druid-sql
• druid-historgram via druid-sql
• druid-indexing-service via hadoop-client
• druid-indexing-hadoop

But pretty much all everything in Druid is going to depend on Netty 4.1 via http-client, see metamx/http-client#29.

@gianm @drcrallen @himanshug any thoughts?

[drcrallen]

I think we should patch spark to make it work with netty 4.1 and just call out in the spark batch indexer for the druid version which pulls in 4.1 everywhere that it only works with patched spark.

[leventov]

Also @b-slim because it's related to Hadoop

[nishantmonu51]

it seems this will also break anyone using https://github.com/SparklineData/spark-druid-olap .
asking everyone to patch their spark installations may not be a good idea, we can probably shade netty jars for now until spark updates netty version.
Also, is there any specific fix/improvement in Netty 4.1 that we are updating for ? If not, can we use an older version that works well and skip upgrading Netty in 0.10.1 ?

[leventov]

Not sure about https://github.com/SparklineData/spark-druid-olap, but for https://github.com/metamx/druid-spark-batch publishing druid-processing alone would also help, because druid-spark-batch actually depends only on druid-processing, which doesn't depend on Netty.

[b-slim]

Looking quickly at https://github.com/SparklineData/spark-druid-olap does not seem to be impacted by this anyway (don't see dependency on druid processing).

[himanshug]

reiterating @nishantmonu51 's question , what is forcing us to upgrade Netty at this point ?

[leventov]

@himanshug probably nothing right now (not sure about druid-sql), but conceptually I'm against this approach. IMO huge projects should drive each other to update to the newer versions of libraries like Netty, not to stop each other from updating, because others are not updating. In comments here: https://issues.apache.org/jira/browse/SPARK-19552 a Spark committer Sean Owen said that they are not that much against updating Netty in Spark anymore, so IMO the solution is to update Spark to Netty 4.1, not to downgrade Druid to 4.0.

[himanshug]

@leventov yes, keeping dependency versions up-to-date is good in general. but in the current case, given that there is no definite need, I would let spark get updated first and then update Druid rather than trying to find workarounds for now.

[drcrallen]

@himanshug we're looking to upgrade the http-client metamx/http-client#29

[leventov]

for note Spark 2.3 updated it's Netty dep to 4.1

[stale]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the [email protected] list. Thank you for your contributions.

[stale]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.