[Question] Is it possible to execute the evaluation functions at an intervall?

[simpsus]

Bascially title, I want to evaluate only every n-th boosting round. Bonus: Can it not evaluate for the the first S rounds?

Why? My evaluation code is expensive and it will determine if my best iteration is 2300 or 2400, but for that purpose it can

• start evaluating only at iteration 1500 and not evaluate earlier
• evaluate every 10 iterations
  So my dream code would look like this

booster = lgb.train(
                {
                ...
                "eval_frequency":10,
                "start_eval_round":1500,
                },
                train_set,
                feval=fevals,
                valid_sets=[valid_set],
                callbacks=[lgb.record_evaluation(eval_result=eval_results),]
            )

[jmoralez]

Hey @simpsus, thanks for using LightGBM. About the evaluation frequency we have a feature request in #4107, so it may be implemented in the future. However, the starting at a specific iteration sounds too custom, so you may be better off trying to implement something yourself, you may find this discussion useful. Basically the idea is that you can implement your own training loop with something like the following:

params = {...}
cb = lgb.record_evaluation(eval_result=eval_results)
train_ds = lgb.Dataset(...)
valid_ds = train_ds.create_valid(...)
bst = lgb.Booster(params=params, train_set=train_ds)
bst.add_valid(valid_ds, 'valid')
# train the first iterations
for _ in range(1500):
    bst.update()
for i in range(1000): # some number of iterations here
    bst.update()
    if i % 10 == 0:
        evaluation_result_list = bst.eval_valid(feval)
        cb(lgb.callback.CallbackEnv(model=bst,
                                params=params,
                                iteration=1500+i,
                                begin_iteration=0,
                                end_iteration=1000,
                                evaluation_result_list=evaluation_result_list))

Please let us know if you run into any issues.

[simpsus]

Thank you very much. I did not know custom training loops are possible in lightgbm.

I thought I would have to resort to globally log the iteration and then use that in the eval function and other callbacks. I will report back if encounter problems. Closing for now.

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.