Feature Request: customizable early_stopping_tolerance

[kryptonite0]

I have a situation where the default numerical tolerance (0.001) for early stopping is too large. My target has a gamma distribution and the LGB Regressor reaches convergence too early, when the numerous low target values are well approximated by the model, but the few large values are still underestimated. When I deactivate early stopping, I can see the loss metric still improving at the 4th or more decimal digit, past the best iteration reached during early stopping. It would be great to be able to set manually the tolerance.

[StrikerRUS]

@kryptonite0 Can you please provide a reproducible example or training logs at least?

As I know, we do not have any default numerical tolerance. Refer to:

LightGBM/python-package/lightgbm/callback.py

Line 227 in fc991c9

if best_score_list[i] is None or cmp_op[i](score, best_score[i]):

https://docs.python.org/3/library/operator.html#operator.lt

Linking dmlc/xgboost#4982 here.

[StrikerRUS]

Ping @kryptonite0

As you can see from examples, there is no "default numerical tolerance (0.001)":

LightGBM/examples/python-guide/advanced_example.py

Lines 52 to 59 in 785e477

	gbm = lgb.train(params,
	lgb_train,
	num_boost_round=10,
	valid_sets=lgb_train, # eval training data
	feature_name=feature_name,
	categorical_feature=[21])
	print('Finished first 10 rounds...')

[1]	training's binary_logloss: 0.680894
[2]	training's binary_logloss: 0.672151
[3]	training's binary_logloss: 0.664753
[4]	training's binary_logloss: 0.656185
[5]	training's binary_logloss: 0.648174
[6]	training's binary_logloss: 0.641671
[7]	training's binary_logloss: 0.635597
[8]	training's binary_logloss: 0.628874
[9]	training's binary_logloss: 0.622432
[10]	training's binary_logloss: 0.616403
Finished first 10 rounds...

[StrikerRUS]

At present we do not have any "default numerical tolerance". But having customizable early stopping tolerance might be useful in some cases.

[StrikerRUS]

Closed in favor of being in #2302. We decided to keep all feature requests in one place.

Welcome to contribute this feature! Please re-open this issue (or post a comment if you are not a topic starter) if you are actively working on implementing this feature.

[jmoralez]

Hi. I'm working on this, I'll make a PR soon.

[StrikerRUS]

Corresponding XGBoost experience:
Issue: dmlc/xgboost#4982
PR: dmlc/xgboost#6942
Further improvement: dmlc/xgboost#7137

[jmoralez]

Thanks for that. I believe my approach is the same as dmlc/xgboost#7137, basically the change I made was replacing operator.gt here:

LightGBM/python-package/lightgbm/callback.py

Line 200 in 99cc4f2

cmp_op.append(gt)

with:

def _gt_threshold(curr_score, best_score, threshold):
    return curr_score > best_score + threshold

and the opposite for the minimize case (curr_score < best_score - threshold). I named the parameter early_stopping_threshold in train, cv and still had to modify sklearn.py and the docs.

[StrikerRUS]

#4580 implemented this feature request for Python-package. Thank you very much @jmoralez !
I think we should reuse this issue as a feature request for adding the same functionality into R-package and core cpp code to not split the discussion. Refer to #4580 (comment). So, I'm not excluding this issue from #2302 for now.

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed.
To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues
including a reference to this.