[FEA] Support dask query planning

[ayushdg]

Is your feature request related to a problem? Please describe.
Currently many functionalities/tests do not work when dask query planning is enabled (Default dask behavior).

This is an issue to track the gaps for query planning to work with Curator

• FuzzyDuplicates
  • Melt (now supported in 24.06)
  • Custom blockwise/hlg logic in shuffle/merge steps
• TestShuffle
  • Specifically around dataset.df[self.rand_col] = dataset.df.map_partitions(self._add_rand_col)
    TypeError: Expected Pandas-like Index, Series, DataFrame, or scalar, got numpy.ndarray

[rjzamora]

@ayushdg - I'd like to start working on this. Do you want to support both query-planning "on" and "off" moving forward? It may be a bit hard to do this without a bunch of compatibly code.

Also, note that melt should now be supported with cudf-2406.

[sarahyurick]

Closed by #139.