join big tables takes too long

[jangorecki]

Detected by db-benchmark. Join 1e9 for data.table does not finish within 4h. It used to not finish this test because of OOM when run on 128GB mem machine. Now it runs on 250GB so I would expect it to finish. Considering that joins should scales pretty well in data.table it sounds abnormal that 4h is not enough.

c6id.metal machine
250 GB ram
128 cores
duckdblabs/db-benchmark#61

if someone has access to c6id.metal, it would be useful to run join script using verbose=TRUE, and also extend timeout, to see if maybe little more time is enough for completion.

[tdhock]

@Anirban166 @DorisAmoakohene this would be an interesting example to investigate, perhaps using atime

[HughParsonage]

I can't quite follow that repo. What's the actual minimal code for the join that the benchmark is measuring?

[jangorecki]

https://github.com/duckdblabs/db-benchmark/blob/master/datatable/join-datatable.R
this is the test script

data needs to be generated with

Rscript _data/join-datagen.R 1e9 NA 0 0