Enhance input datasource capabilities

[mkuthan]

Right now druid-spark-batch reads data using sc.textFile from given locations. This is important limitation if data is stored in formats like Parquet (or any other data format supported by Spark).

Would you consider to enhance this tool with support for arbitrary Spark SQL expression to define input data? You'll get for free:

• support for any data format supported by Spark
• support for any UDF supported by Spark for data pre-processing
• support for joins before ingestion

[drcrallen]

I think it would make some sense to have a DataSupplierFactory (or a similar more-sparky-name) passed in the task definition, one of whose implementations effectively does sc.textFile(dataFiles mkString ",") and whose other implementations do other things, then have the chain at

val baseData

feed of the factory.

Such an improvement would be quite handy.

[Gauravshah]

wouldn't it be easier to make it accept a dataframe, and we do all the pre-processing before sending it to spark-batch ?

[drcrallen]

@Gauravshah similar to #10 ?

[Gauravshah]

👍