Questions on Daft

[nauhziy]

Hey there - I just went through the quickstart and read the docs, must have been quite a grind to get all this done so quickly! Happy to share my thoughts and would be curious to understand how the team is thinking about these.

1. It's not immediately clear to me what the value prop is for this - at least based on the website and docs. I'd imagine that the target audience will be enterprise clients in the computer vision space, so the messaging should be obvious, given the inertia to migrate tech stacks and adopt is very high.

How were you thinking about your go to market strategy?

2. I like that it is python native - makes complete sense and is in line with market trends in terms of language adoption.

3. I spoke with Sieve data (I think they are the same YC batch w/ you) - thought there were some very interesting overlaps there in terms of use case.

4. I saw that there are methods around loading from datasets (and I think loading from DBs will be absolutely crucial). Will there be an export to / load into method?

5. Maybe it will be interesting to show side by side comparisons of how a workflow in daft looks like, vs something built in a competing DF method. And then showing the time / efficiency delta. That would be a very obvious metric to optimise for

That's it for now - I'd look at it again when I have more time, happy to discuss. And I think it's a super interesting problem space that the team here is working on!

[jaychia]

Thanks for the questions!

1. Great point - computer vision is indeed one of the biggest use-cases for a product such as Daft, but we also know of many other use-cases such as processing protobufs, audio, documents, ROSbags etc that Daft is great for. We're working on the messaging, but agree that "Complex Data" isn't the most well known term.
2. Indeed - Python is a big part of our strategy and we're glad this resonates!
3. Yes, we're in touch with the folks at Sieve as well and they're awesome folks to work with!
4. This is definitely on the roadmap, with plans for integration with Iceberg, Postgres, BigQuery, Snowflake etc - if you already have these databases with URLs pointing out to your files in S3, that's perfect for Daft.
5. Benchmarks are on the way! We will be making another big announcement once we have them.

Thanks for the feedback, really appreciate it! It's really encouraging for the team to see folks engaging with what we've built and we hope to keep up the momentum :)

[Matey-Tsilov]

Hey I also wanted to ask something about an error which pops out everytime I want to install Ludwig ai- Deep Machine Learning framework whose dependencies include Daft:

How can I resolve that issue, I am on windows OS is that the problem? Can you help me a little?

[samster25]

This should now be resolved now that be publish windows wheels

[jmakov]

Would be interesting to know how it compares to fugue on ray. Also I didn't find any docs on out of core processing. To me that's the missing part in the ecosystem.

[jaychia]

Hi @jmakov

Thanks for the questions! I just looked through the fugue docs, and it seems like the goal of the project is to present a unified interface across a few distributed execution engines (Pandas, Spark, Ray and Dask). For Ray, I believe it just uses Ray Datasets under the hood. In practice, this means that Daft could potentially be one of Fugue's execution engines and is complementary to the goals of the Fugue project.

However the interface that Fugue can provide is limited to the union of features across these backends. For instance, running computations through Fugue is done very generically using a Python function invoking Pandas code (see: transform). This is going to be orders of magnitude slower than running code native to each execution engine directly (e.g. using a PySpark/Daft expression).

If your goals are:

1. Code portability across different execution backends (i.e. your code needs to be able to run on Spark, Ray or Dask)
2. Simple backend-agnostic use-cases which Fugue and all of its supported backends support such as: read files, run a python function on your data, and write files back out

Then Fugue is a great solution!

However for more complex or performance-critical workloads, it might be worth directly using a DataFrame engine such as Daft instead.

I didn't find any docs on out of core processing

Thanks for the feedback! I'll add some documentation soon, but we really just leverage Ray's object spilling to achieve this -- if your data doesn't all fit in memory at once, Daft partitions will automatically spill onto disk based on memory pressure on the Ray cluster.

[jaychia]

PR incoming for documentation on out-of-core processing (and memory management in Daft in general!): #1736

[jmakov]

Thanks a lot for the detailed answer. I'm particularly interested in sorting larger than memory DFs since that's currently isn't possible in polars (sorting a .scan_parquet or .scan_ipc source loads the whole DF to /tmp/polars/sort and sorts there).

[jaychia]

Makes sense - you can actually try running Daft on a local Ray cluster which will run your operations while spilling to /tmp/ray/...

import daft

daft.context.set_runner_ray()

df = daft.read_parquet(...)
df = df.into_partitions(N)  # Split into multiple partitions for more efficient operations, I'd recommend ~256MB per partition
df = df.sort(...)
df.collect()

[jmakov]

Thanks for all the info. Till now my understanding of Daft is that it's improved Dask e.g. Dask on ray. But wondering what's underneath e.g. if I run polars on a single node, how will Daft compete? The first time I heard about Daft I somehow thought it's polars+smart partitioning on ray. Would be interesting to read an engineering blog about design decisions in Daft to understand if it's its own thing or a combination of already existing data frames + ray.