Failed to find data source: kinesis

[bijith]

I am getting following exception when trying to read from Kinesis. Any idea? using "spark-streaming-kinesis-asl_2.11"

Exception in thread "main" java.lang.ClassNotFoundException: Failed to find data source: kinesis. Please find packages at http://spark.apache.org/third-party-projects.html
at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:549)
at org.apache.spark.sql.execution.datasources.DataSource.providingClass$lzycompute(DataSource.scala:86)
at org.apache.spark.sql.execution.datasources.DataSource.providingClass(DataSource.scala:86)
at org.apache.spark.sql.execution.datasources.DataSource.sourceSchema(DataSource.scala:195)
at org.apache.spark.sql.execution.datasources.DataSource.sourceInfo$lzycompute(DataSource.scala:87)
at org.apache.spark.sql.execution.datasources.DataSource.sourceInfo(DataSource.scala:87)
at org.apache.spark.sql.execution.streaming.StreamingRelation$.apply(StreamingRelation.scala:30)
at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:150)
at nativo.data.hackathon.RealtimeAggregator$.main(RealtimeAggregator.scala:49)
at nativo.data.hackathon.RealtimeAggregator.main(RealtimeAggregator.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: java.lang.ClassNotFoundException: kinesis.DefaultSource

[alexmnyc]

Same here

[usufarif]

I am getting the same error message, when I try to run it from Scala object in Eclipse, via a Maven build. However, if I run the spark-shell and run the spar.readStrea.format("kinesis").option... in REPL, it works just fine. Any clues as to why that is the case? Here is the the Stack trace.

---

Exception in thread "main" java.lang.ClassNotFoundException: Failed to find data source: kinesis. Please find packages at http://spark.apache.org/third-party-projects.html
at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:549)
at org.apache.spark.sql.execution.datasources.DataSource.providingClass$lzycompute(DataSource.scala:86)
at org.apache.spark.sql.execution.datasources.DataSource.providingClass(DataSource.scala:86)
at org.apache.spark.sql.execution.datasources.DataSource.sourceSchema(DataSource.scala:195)
at org.apache.spark.sql.execution.datasources.DataSource.sourceInfo$lzycompute(DataSource.scala:87)
at org.apache.spark.sql.execution.datasources.DataSource.sourceInfo(DataSource.scala:87)
at org.apache.spark.sql.execution.streaming.StreamingRelation$.apply(StreamingRelation.scala:30)
at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:150)
at SparkKinesis$.delayedEndpoint$SparkKinesis$1(SparkKinesis.scala:54)
at SparkKinesis$delayedInit$body.apply(SparkKinesis.scala:12)
at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.collection.immutable.List.foreach(List.scala:392)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.App$class.main(App.scala:76)
at SparkKinesis$.main(SparkKinesis.scala:12)
at SparkKinesis.main(SparkKinesis.scala)
Caused by: java.lang.ClassNotFoundException: kinesis.DefaultSource

[bobdemp]

Hi

Did this ever get resolved?

Thanks

Rob

[imranshaikmuma]

Spark structured streaming is not supported for Kinesis currently
To use Spark structured streaming with Kinesis, use the jar located at https://github.com/qubole/kinesis-sql
and start the shell by spark-shell --jars /directory/spark-sql-kinesis_2.11-2.3.1.jar

[VibinD]

Do anyone know is this issue resolved or any work around for this?

[chadlagore]

com.qubole.spark:spark-sql-kinesis_2.11:1.1.3-spark_2.4 worked for me.

[manishkaushik29]

I tried with all the following combinations of jars

Still getting above error:

spark-tags_2.11-2.4.0.jar,
spark-streaming-kinesis-asl_2.11-2.4.0.jar,
spark-streaming_2.11-2.4.0.jar,
aws-java-sdk-sts-1.11.271.jar,
amazon-kinesis-client-1.8.10.jar,
spark-sql_2.11-2.4.0.jar

#####################################################################

spark-tags_2.11-2.4.3.jar,
spark-streaming-kinesis-asl_2.11-2.4.3.jar,
aws-java-sdk-sts-1.11.271.jar,
jackson-dataformat-cbor-2.6.7.jar,
unused-1.0.0.jar,
spark-sql_2.11-2.4.3.jar

##########################################
spark-sql-kinesis_2.11-1.1.3-spark_2.4.jar,
spark-tags_2.11-2.4.0.jar,
unused-1.0.0.jar,
scala-library-2.11.12.jar,
spark-sql_2.11-2.4.0.jar