use arrow format as the memory model

[waynelapierre]

As the Python packages pandas and polars use arrow format as the memory model, any plan of having it in data.table?

[jangorecki]

I doubt. Algos in DT are designed for particular memory layout that R uses for data.frame. Switching this to arrow means that we cannot easily switch between DF and DT without doing a copy. And of course a lot of code rewriting. That of course could be considered, but what benefits do you expect? Without presenting a good reason for that it is rather unlikely.
Unfortunately portability for binary formats is a myth. At least till date, for feather arrow parquet etc.

[eddelbuettel]

Duplicate of #5656

A couple of months ago I looked into arrow -> data.table exporting. I couldn't find anything obvious as the two memory models are simply very different there is no way I can see around a one-time copying / materialization cost.

[eitsupi]

It should also be noted that the Arrow libraries (Arrow C++ or arrow-rs or the arrow2 crate) can be much more expensive to build than the current data.table.
The Pandas dev team discussed whether to include pyarrow as a required dependency (pyarrow is very difficult to build from source).

The polars R package has been removed from CRAN due to build time issue (In other words, building polars takes longer than any R package on CRAN.).

[jangorecki]

AFAIR @eddelbuettel can be good person to comment about building R's arrow package