The future of this project

[evkaky]

The activity of the project is super low (active PRs, issues etc). No roadmaps, no new features, almost nothing.
Is Jetbrains going to abandon this project at some point? If it is - why so?

[Jolanrensen]

Unfortunately the project at the moment has a low priority indeed. I took responsibility to keep it up to date, but my duties at Kotlin DataFrame need to come first. There is more user interest in that library and since Apache rejected the Kotlin API from becoming a first-class citizen, it's getting harder to justify my work here.
Also, keeping it up to date has become exceedingly hard, thanks to Spark 3.4+. I'm working on an a rebuild for supporting those Spark versions in my free time, so hopefully I can continue keeping the library up-to-date, however, new features are not planned.
That said, the project is open-source; anyone willing can contribute. It's what I did before joining JetBrains and I will be more than ready to help with contributions when needed.

[evkaky]

Apache rejected the Kotlin API from becoming a first-class citizen

That's terrible. Deeply disappointed about this Apache's decision. Kotlin would fit for Spark jobs/notebooks much better with its DSLs than Python or Scala or SQL

[Jolanrensen]

@evkaky I agree! Unfortunately Spark is written in Scala with only Java in mind, so coop with Kotlin in the same repo is difficult (hence this API). That said, there's also unofficial support for .NET and Spark and that seems to be doing fine, so if we can keep this API working we should be good to go.

[evkaky]

@Jolanrensen BTW, event if Apache declined Kotlin as a first class API like Scala / Java / Python, can we (JetBrains ?) just utilise this all-new Spark's "connect" feature to built a solid API?
https://spark.apache.org/docs/latest/spark-connect-overview.html

[Jolanrensen]

@evkaky It's actually something I'm looking into for 3.4+ support. One downside atm is that I cannot get spark-connect itself to work reliably at all. Running a Spark connect server with spark 3.5.1 and Scala 2.13 just breaks completely and when I target 2.12 I cannot get their simple standalone application sample to work due to various NoSuchMethodErrors etc. It would be very helpful if someone could give me a working sample with Spark-connect to check whether my 3.4+ implementation would actually work with it.

[andrija-s]

Umm, please live.

[Jolanrensen]

@andrija-s Just begging won't bring updates to the project. A lot of time and hard work does. I'm working full-time on Kotlin DataFrame. At the moment, we just don't have the time or resources needed to continue the 3.4+ path I started here. Plus, undoubtedly, Spark will break more stuff in the future, which will require even more time. Time which we may have in the future, but just not right now.

I don't want it to die either. When I found this project some time ago and I needed it myself, I contributed a lot of work to improve it. But so could you :) or anyone from the community. If you think the project is valuable enough and you're willing to invest the time and work I originally did, I'm not stopping you, in fact, I'm encouraging it :)