Summary of meeting 2

[jpivarski]

• We saw a demo of @douglasdavis’s user-level DAG (it worked, even without predictive type information; lazily loaded a Parquet file, computed getitem and flatten, visualizing the DAGs).
• We agreed that we are on the same page about the DAG being user-level with a kernel-level tracer to provide it with type information.
• Since Dask provides virtualness and partitioning, I'm considering descoping VirtualArray and PartitionedArray node types in v2, so that these features will have to come from Dask. (It's better than providing two features that do almost the same thing, but have subtle differences.)
• We also talked about whether the delayed array should be a distinct Python type from ak.Array; projects like dask.array and dask.frame are distinct from NumPy and Pandas, but the distinction is not maintained within xarray. @martindurant is of the opinion that it's better for them to be separate, so that calling .compute() or .persist() is a big deal, very visible to the user. I can be persuaded, I just want to think about it and be sure that we're not giving up an opportunity. This project is different from dask.array and dask.frame in that we can change Awkward's v2 development to fit, to make a more unified experience for users.
• As a corollary to that, there's a question of whether ak.from_parquet and dak.from_parquet (import dask_awkward as dak) should be separate functions (as they were in @douglasdavis’s demo, with each returning a different Python type: ak.Array vs dak.AwkwardDaskArray), or if there should be one function with a lazy=True flag. This is related to @agoose77's question on Gitter about ak.broadcast_arrays's arguments vs multiple functions, but it's addressing a more fundamental split if ak.Array is not the same Python type as dak.AwkwardDaskArray (even by inheritance).
• If calling .compute() or .persist() is a big deal, very user-visible, then we might be able to dispense with caching. Same-nodes within a DAG are not recomputed in one .compute() as part of Dask's infrastructure, but if a user calls .compute() twice, the file will be read twice and the statements will be computed twice. Users can explicitly set that to a variable. (Caching was important when computation was invoked implicitly.)

Questions about replacing Awkward's internal VirtualArray and PartitionedArray with Dask and an explicit, user-level .compute() would affect Coffea, especially NanoEvents, so @nsmith- will want to know that we're discussing it.

[martindurant]

On the parquet question, I've looked through the code, and see that in Awkward's use, all calls go through pyarrow.parquet.ParquetFile and pyarrow.parquet.ParquetWriter, which allows an arbitrary file-like object instead of a file-path, and this could be provided by fsspec.

The exception is pyarrow.parquet.write_metadata, which claims to need a pyarrow NativeFile, but works just fine in Dask anyway.

In short, we need to percolate a filesystem through the awkward code and replace os.open with fs.open and change all of the calls to os.path.

Note that the dask implementation essentially does all the filesystem and paths stuff and makes a graph before calling arrow_table.to_pandas at read time. We could reuse this stuff without the final step.

[jpivarski]

That's true, and the only thing preventing ak.from_parquet from reading remote files is in the Awkward code surrounding pyarrow.parquet.ParquetFile. I know I've looked at that to see what assumptions can be relaxed to allow reading from HTTPS, and there aren't many. (Might it be possible to write to remote filesystems, too?) I was thinking that should go in with a general change to use pyarrow's Dataset API (which would probably collapse some of the if-branches) and possibly also fsspec.

I'm in favor of doing that, but I'm just not familiar enough with the Dataset API and fsspec to do it right now. It would, however, fit nicely in the Dask project: "enabling Dask" would necessarily involve modifications to the file I/O. Then I could look at what you (pl.) do to from_parquet/to_parquet and use that as an example to update Uproot so that it's Dask-ready in the same way. If I can also learn by example how to fsspecify Uproot, to read ROOT files from arbitrary filesystems, that would be cool.

[martindurant]

Agreed, reworking of IO can be on our task list, but lower priority than the basic stuff we have started to do so far.