Reduce size of Timeseries task graph

[rjzamora]

Possible alternative to #263

While reviewing #263, I realized we were going a bit "overboard" in Timeseries.random_state. More specifically, we were generating/storing 624 * len(dtypes) 32-bit integers for every task. Given that the timeseries utility is generally meant for demonstration, testing and benchmarking, I don't see any reason to generate/store more than len(dtypes) integers for each task.

This PR reduces the number of random integers we store for each task by a factor of 624 (therefore, significantly reducing the size of the graph).

[phofl]

thx, this was on my todo list as well.

[phofl]

It's still twice as slow compared to dask/dask for a 70GB time series with 100 float columns on a 15 machine cluster. Any idea how we can speed this up without removing the seed?

[rjzamora]

It's still twice as slow compared to dask/dask for a 70GB time series with 100 float columns on a 15 machine cluster. Any idea how we can speed this up without removing the seed?

What takes twice as long? Is it generating the task graph or communicating it between client and scheduler?

[phofl]

It's mostly the sending of the graph, it's still significantly larger. Both are equally fast if I set a seed explicitly

[rjzamora]

Both are equally fast if I set a seed explicitly

Oh! I didn't realize that - very interesting.

[rjzamora]

Hm, I'm not finding that setting a seed changes the graph size in any way. However, I do see that we only store a single integer for each partition in dask/dask. If I recall correctly, this is because we always generate the data for every element of dtypes (even for columns that have been dropped). In dask-expr, we store a separate seed for each of the original columns. However, we could always adopt the dask/dask approach to reduce the graph size.

[phofl]

Setting the seed increases the graph size in dask/dask, that’s what makes the performance comparable. Sorry that was misleading earlier