Enhancement request: Apply Custom Time Series Functions

[BenLBurke]

I've heard it is in development but wanted to add to the backlog.

I commonly find I am running multiple time-series related functions either with Numpy or utilizing tsfresh's library.

While these are useful, tsfresh's ability to run on spark is available, via their tsfresh.convenience.bindings.spark_feature_extraction_on_chunk feature, it is limited in it's capacity to truly run in SPARK as it performs an applyInPandas transformation.

Would love to see the ability to either utilize these functions or have custom created functions as well.

[rportilla-databricks]

@BenLBurke , this is definitely an interesting request. Can you explain what functions you are looking for in particular? The reason we ask is because applyInPandas is the go-to method if the function you want to apply is completely custom. However, if there is something specific (1 or 2 features from tsfresh) that you like, we can optimize this by using PySpark directly under the hood. Let me know what you're looking for and we can prioritize it.