sparklyr::enableMosaic() reference missing class IndexSystemID

[dabruehl]

Describe the bug
I want to enableMosaic Version 0.3.12 with sparklyr on a Databricks cluster DBR 12.2 LTS and get a java.lang.ClassNotFoundException: com.databricks.labs.mosaic.core.index.IndexSystemID

To Reproduce
Steps to reproduce:

1. Install sparklyrMosaic_0.3.12
2. Load library sparklyrMosaic
3. Enable Mosaic

Code to reproduce:

library(sparklyr)
install.packages("/dbfs/FileStore/mosaic/sparklyrMosaic_0.3.12.tar.gz", repos = NULL)

library(sparklyrMosaic)
sessionInfo()

sc <- sparklyr::spark_connect(method="databricks")
sparklyrMosaic::enableMosaic(sc)

Expected behavior
Mosaic enable functions and return without error

Error message
java.lang.ClassNotFoundException: com.databricks.labs.mosaic.core.index.IndexSystemID
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at com.databricks.backend.daemon.driver.ClassLoaders$LibraryClassLoader.loadClass(ClassLoaders.scala:151)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at sparklyr.StreamHandler.handleMethodCall(stream.scala:111)
at sparklyr.StreamHandler.read(stream.scala:62)
at sparklyr.BackendHandler.$anonfun$channelRead0$1(handler.scala:60)
at scala.util.control.Breaks.breakable(Breaks.scala:42)
at sparklyr.BackendHandler.channelRead0(handler.scala:41)
at sparklyr.BackendHandler.channelRead0(handler.scala:14)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:750)

Additional context
According the repo the class IndexSystemID is missing since Vesion 0.3.8
https://github.com/databrickslabs/mosaic/tree/v_0.3.8/src/main/scala/com/databricks/labs/mosaic/core/index

[sllynn]

Looks like we didn't update the boilerplate parts of the package before regenerating the bindings and publishing the package. I'm putting it right in #455 , bear with me.

[sllynn]

OK, I haven't yet had a chance to try this in a notebook but, if you're blocked, take a look at the "artefacts" package produced in this run. There should be a working version of the sparklyr wrapper inside. 👍

[dabruehl]

Hi @sllynn, we tested the created artefact (sparklyrMosaic) on our cluster and it works. Thank you for the fast fix and contribution