Versioning docs: DeltaLake

[ankatiyar]

Description

Partial solution #4468

Development notes

• Move DVC versioning docs under "integrations" header
• Documented the use of pandas.DeltaTableDataset to interact with Delta tables

Questions for reviewers:

• There's slightly more you can do with Delta lake using delta-rs for eg Load with datetime, restore to a prev version. Not sure how far I should go, for example, suggesting creating a custom Dataset on top of pandas.DeltaTableDataset? Update the pandas.DeltaTableDataset itself?
• How much should we document using delta tables with other datasets eg. spark.SparkDataset can save delta format tables. spark.DeltaTableDataset can simply read the datasets but not specific versions etc. We can also suggest building custom datasets to interact with spark but would that be overkill?
• Similarly, for other frameworks eg dask, polars for which we have no/limited support currently in kedro-datasets

Developer Certificate of Origin

We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a Signed-off-by line in the commit message. See our wiki for guidance.

If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.

Checklist

• Read the contributing guidelines
• Signed off each commit with a Developer Certificate of Origin (DCO)
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the RELEASE.md file
• Added tests to cover my changes
• Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

[ankatiyar]

cc @pascalwhoop

[astrojuanlu]

 sphinx.errors.SphinxWarning: /home/docs/checkouts/readthedocs.org/user_builds/kedro/checkouts/4483/docs/source/integrations/deltalake_versioning.md:15:'myst' cross-reference target not found: '' [myst.xref_missing] 

[astrojuanlu]

Similarly, for other frameworks eg dask, polars for which we have no/limited support currently in kedro-datasets

We do have support for Polars actually! There are a few unaddressed issues, that's true... https://github.com/kedro-org/kedro-plugins/issues?q=is%3Aissue%20state%3Aopen%20polars but it's very much an officially supported dataset

For Dask there's a dataset too but I don't see it mentioned very often https://docs.kedro.org/projects/kedro-datasets/en/kedro-datasets-6.0.0/api/kedro_datasets.dask.CSVDataset.html

[astrojuanlu]

Not sure how far I should go, for example, suggesting creating a custom Dataset on top of pandas.DeltaTableDataset? Update the pandas.DeltaTableDataset itself?
How much should we document using delta tables with other datasets eg. spark.SparkDataset can save delta format tables. spark.DeltaTableDataset can simply read the datasets but not specific versions etc. We can also suggest building custom datasets to interact with spark but would that be overkill?

Indeed... unfortunately kedro-org/kedro-plugins#542 is still unaddressed, so I think this doc page should limit itself to describing what the user can do today, and suggest creating custom datasets when appropriate. We might want to tackle kedro-org/kedro-plugins#542 later in the year and go back to these docs.

[astrojuanlu]

Gave this a first quick pass, thanks a lot @ankatiyar ! Dropped a few comments

[DimedS]

Thanks, @ankatiyar! I like the description - just left a small question.

[pascalwhoop]

No extra jar dependencies needed for spark? Eg when running delta on something that isn't a databricks runtime.

[ankatiyar]

I've limited the scope of this documentation page to delta-rs and pandas.DeltaTableDataset which deals with delta table and converts into pandas DataFrame. There's actually already a section for spark in the docs https://docs.kedro.org/en/stable/integrations/pyspark_integration.html#spark-and-delta-lake-interaction

[ankatiyar]

Similarly, for other frameworks eg dask, polars for which we have no/limited support currently in kedro-datasets

We do have support for Polars actually! There are a few unaddressed issues, that's true... https://github.com/kedro-org/kedro-plugins/issues?q=is%3Aissue%20state%3Aopen%20polars but it's very much an officially supported dataset

For Dask there's a dataset too but I don't see it mentioned very often https://docs.kedro.org/projects/kedro-datasets/en/kedro-datasets-6.0.0/api/kedro_datasets.dask.CSVDataset.html

I meant support for versioning with Delta Tables specifically

[astrojuanlu]

Oh, right kedro-org/kedro-plugins#444

[ankatiyar]

Actually turns out, there's already a section for Delta lake + Spark https://docs.kedro.org/en/stable/integrations/pyspark_integration.html#spark-and-delta-lake-interaction, shall I move it to this page? cc @astrojuanlu

[astrojuanlu]

First time I see that 😂 Yeah! Maybe bring that content to the page you're writing, and replace it by "if you want to work on Delta with PySpark, check out these docs [link]"

[astrojuanlu]

Rendered version https://kedro--4483.org.readthedocs.build/en/4483/integrations/deltalake_versioning.html

[astrojuanlu]

Made a few style comments, will hold off until you consolidate the Spark Delta stuff in here too

[lrcouto]

Markdown not rendering correctly here?

[lrcouto]

Looks good, easy enough to follow! Pointed out a couple minor details.

[ElenaKhaustova]

Looks good, nice job!

Answering questions for reviewers: I think for the first version of docs it is deep enough. What might be useful if we mention other use-cases with a note that can be done with custom datasets.

[astrojuanlu]

Added some minor suggestions, but this is good to go already! Thanks @ankatiyar 🙏🏼