ValueError: bytes object is too large

[nerdylinius]

Description

I've been getting a "ValueError: bytes object is too large" error while using lightgbm with dask. How can I fix it?

Reproducible example

from dask import array as da
train_df, val_df = get_train_val_data('2020-01-01', '2020-02-28', time_gap = 5, daily_fold=0, daily_k_fold=2, intraday_fold=0, intraday_k_fold=2)
train_chunks = min(100000, int(train_df.shape[0]/16))
val_chunks = min(100000, int(val_df.shape[0]/16))
Xy1 = da.from_array(train_df[selected_feats.ALL_FEATS+[selected_feats.TARGETS[0],]].values)
Xy2 = da.from_array(val_df[selected_feats.ALL_FEATS+[selected_feats.TARGETS[0],]].values)
X1, y1 = Xy1[:,:-1].rechunk(train_chunks), Xy1[:,-1].rechunk(train_chunks)
X2, y2 = Xy2[:,:-1].rechunk(val_chunks), Xy2[:,-1].rechunk(val_chunks)
print(X1.compute_chunk_sizes(), y1.compute_chunk_sizes())

dask_model = lg.DaskLGBMClassifier(num_leaves=1024, learning_rate=0.02, n_estimators=1000, colsample_bytree=0.7,
                                       n_jobs=300, silent=False)
dask_model.fit(X1, y1, eval_set=[(X2, y2)], eval_metric='auc', early_stopping_rounds=15)

###output###

dask.array<rechunk-merge, shape=(6107340, 1002), dtype=float32, chunksize=(100000, 1002), chunktype=numpy.ndarray> dask.array<rechunk-merge, shape=(6107340,), dtype=float32, chunksize=(100000,), chunktype=numpy.ndarray>
Finding random open ports for workers
Traceback (most recent call last):
  File "/media/st2000/workspace/darkquant/darkquant/auto_trader/ashare/alpha_v10c/tree/dask_lg_train.py", line 138, in <module>
    dask_model.fit(X1, y1, eval_set=[(X2, y2)], eval_metric='auc', early_stopping_rounds=15)
  File "/home/hadoop/anaconda2/envs/pytorchenv/lib/python3.6/site-packages/lightgbm/dask.py", line 736, in fit
    **kwargs
  File "/home/hadoop/anaconda2/envs/pytorchenv/lib/python3.6/site-packages/lightgbm/dask.py", line 626, in _lgb_dask_fit
    **kwargs
  File "/home/hadoop/anaconda2/envs/pytorchenv/lib/python3.6/site-packages/lightgbm/dask.py", line 447, in _train
    for worker, list_of_parts in worker_map.items()
  File "/home/hadoop/anaconda2/envs/pytorchenv/lib/python3.6/site-packages/lightgbm/dask.py", line 447, in <listcomp>
    for worker, list_of_parts in worker_map.items()
  File "/home/hadoop/anaconda2/envs/pytorchenv/lib/python3.6/site-packages/distributed/client.py", line 1596, in submit
    actors=actor,
  File "/home/hadoop/anaconda2/envs/pytorchenv/lib/python3.6/site-packages/distributed/client.py", line 2594, in _graph_to_futures
    dsk = highlevelgraph_pack(dsk, self, keyset)
  File "/home/hadoop/anaconda2/envs/pytorchenv/lib/python3.6/site-packages/distributed/protocol/highlevelgraph.py", line 124, in highlevelgraph_pack
    return dumps_msgpack({"layers": layers})
  File "/home/hadoop/anaconda2/envs/pytorchenv/lib/python3.6/site-packages/distributed/protocol/core.py", line 184, in dumps_msgpack
    payload = msgpack.dumps(msg, default=msgpack_encode_default, use_bin_type=True)
  File "/home/hadoop/anaconda2/envs/pytorchenv/lib/python3.6/site-packages/msgpack/__init__.py", line 35, in packb
    return Packer(**kwargs).pack(o)
  File "msgpack/_packer.pyx", line 286, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 258, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 196, in msgpack._cmsgpack.Packer._pack
ValueError: bytes object is too large

Environment info

lightgbm==3.2.1
dask==2021.01.1
python==3.6.2

[jameslamb]

Thanks for using LightGBM and for writing up this report! Could you please provide a few more details that will help me to reproduce this?

• definition of get_train_val_data()
• definition of selected_feats
• version of distributed (pip freeze | grep distributed)

If get_train_val_data() is something proprietary, then could you please try to provide a reproducible example using non-proprietary data (such as random data generated with dask.array.random.random())?

---

By the way, I noticed that you are passing eval_set, early_stopping_rounds, and eval_metric. Evaluation sets and early stopping are not yet supported in lightgbm.dask, so maybe those are causing this issue?

Refer to #4101 and #3712.

[nerdylinius]

thanks for your reply. It seem to be a dask bug, refer to dask/dask#7618 .

[jameslamb]

Ah ok, thanks for letting us know! Please come back any time if you run into issues with the Dask interface or if you'd like to contribute.

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.