[R-package] [BUG] Using lgb.train() with booster = "dart" gives warning regardless of early_stopping_rounds value

[serkor1]

Hi,

I found this (trivial) bug - well, I assume it is a bug so feel free to close if it is not the case. Regardless of the set values of early_stopping_rounds the lgb.tran()-function throws a warning message when using the dart-booster.

Below is a minimal working example with three cases displaying the behavior, and the source of the (possible) bug. In all cases the warning occurs, and I have only changed the value of early_stopping_rounds!

MWE

# load library
library(lightgbm)

# define a simple dataset
DT <- list(
  x = as.matrix(
    mtcars[,
           grep(
             pattern = "mpg",
             value = TRUE,
             x = colnames(mtcars),
             invert = TRUE
           )
    ]

  ),
  y = mtcars$mpg
)

# run with early_stopping_rounds
# set to 0L
lightgbm::lgb.train(
  params    = list(
    objective = "regression",
    boosting  = "dart"
  ),
  data      = lightgbm::lgb.Dataset(
    data  = DT$x,
    label = DT$y
  ),
  verbose = -1,
  early_stopping_rounds = 0L
)
#> Warning in lightgbm::lgb.train(params = list(objective = "regression", boosting
#> = "dart"), : Early stopping is not available in 'dart' mode.
#> LightGBM Model (1 tree)
#> Objective: regression
#> Fitted to dataset with 10 columns

# run with early_stopping_rounds
# set to 1
lightgbm::lgb.train(
  params    = list(
    objective = "regression",
    boosting  = "dart"
  ),
  data      = lightgbm::lgb.Dataset(
    data  = DT$x,
    label = DT$y
  ),
  verbose = -1,
  early_stopping_rounds = 1
)
#> Warning in lightgbm::lgb.train(params = list(objective = "regression", boosting
#> = "dart"), : Early stopping is not available in 'dart' mode.
#> LightGBM Model (1 tree)
#> Objective: regression
#> Fitted to dataset with 10 columns

# run with early_stopping_rounds
# set to NULL
lightgbm::lgb.train(
  params    = list(
    objective = "regression",
    boosting  = "dart"
  ),
  data      = lightgbm::lgb.Dataset(
    data  = DT$x,
    label = DT$y
  ),
  verbose = -1,
  early_stopping_rounds = NULL
)
#> Warning in lightgbm::lgb.train(params = list(objective = "regression", boosting
#> = "dart"), : Early stopping is not available in 'dart' mode.
#> LightGBM Model (1 tree)
#> Objective: regression
#> Fitted to dataset with 10 columns

^{Created on 2024-08-15 with reprex v2.1.0}

Session info

sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.4.1 (2024-06-14)
#>  os       Zorin OS 17.1
#>  system   x86_64, linux-gnu
#>  ui       X11
#>  language en_US:en
#>  collate  en_US.UTF-8
#>  ctype    en_US.UTF-8
#>  tz       Europe/Copenhagen
#>  date     2024-08-15
#>  pandoc   3.1.11 @ /usr/lib/rstudio/resources/app/bin/quarto/bin/tools/x86_64/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version date (UTC) lib source
#>  cli           3.6.3   2024-06-21 [1] CRAN (R 4.4.1)
#>  data.table    1.15.4  2024-03-30 [1] CRAN (R 4.4.0)
#>  digest        0.6.36  2024-06-23 [1] CRAN (R 4.4.1)
#>  evaluate      0.24.0  2024-06-10 [1] CRAN (R 4.4.0)
#>  fastmap       1.2.0   2024-05-15 [1] CRAN (R 4.4.0)
#>  fs            1.6.4   2024-04-25 [1] CRAN (R 4.4.0)
#>  glue          1.7.0   2024-01-09 [1] CRAN (R 4.4.0)
#>  htmltools     0.5.8.1 2024-04-04 [1] CRAN (R 4.4.0)
#>  jsonlite      1.8.8   2023-12-04 [1] CRAN (R 4.4.0)
#>  knitr         1.47    2024-05-29 [1] CRAN (R 4.4.0)
#>  lattice       0.22-6  2024-03-20 [4] CRAN (R 4.4.0)
#>  lifecycle     1.0.4   2023-11-07 [1] CRAN (R 4.4.0)
#>  lightgbm    * 4.5.0   2024-07-26 [1] CRAN (R 4.4.1)
#>  magrittr      2.0.3   2022-03-30 [1] CRAN (R 4.4.0)
#>  Matrix        1.7-0   2024-04-26 [4] CRAN (R 4.4.0)
#>  purrr         1.0.2   2023-08-10 [1] CRAN (R 4.4.0)
#>  R.cache       0.16.0  2022-07-21 [1] CRAN (R 4.4.0)
#>  R.methodsS3   1.8.2   2022-06-13 [1] CRAN (R 4.4.0)
#>  R.oo          1.26.0  2024-01-24 [1] CRAN (R 4.4.0)
#>  R.utils       2.12.3  2023-11-18 [1] CRAN (R 4.4.0)
#>  R6            2.5.1   2021-08-19 [1] CRAN (R 4.4.0)
#>  reprex        2.1.0   2024-01-11 [3] CRAN (R 4.4.0)
#>  rlang         1.1.4   2024-06-04 [1] CRAN (R 4.4.0)
#>  rmarkdown     2.27    2024-05-17 [1] CRAN (R 4.4.0)
#>  rstudioapi    0.16.0  2024-03-24 [3] CRAN (R 4.4.0)
#>  sessioninfo   1.2.2   2021-12-06 [1] CRAN (R 4.4.0)
#>  styler        1.10.3  2024-04-07 [1] CRAN (R 4.4.0)
#>  vctrs         0.6.5   2023-12-01 [1] CRAN (R 4.4.0)
#>  withr         3.0.0   2024-01-16 [1] CRAN (R 4.4.0)
#>  xfun          0.45    2024-06-16 [1] CRAN (R 4.4.0)
#>  yaml          2.3.8   2023-12-11 [1] CRAN (R 4.4.0)
#> 
#>  [1] /home/serkan/R/x86_64-pc-linux-gnu-library/4.4
#>  [2] /usr/local/lib/R/site-library
#>  [3] /usr/lib/R/site-library
#>  [4] /usr/lib/R/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

Source of bug

In the code below the waning() will be triggered regardless of the early_stopping_rounds value.

LightGBM/R-package/R/lgb.train.R

Lines 260 to 271 in 6025283

	if (using_dart) {
	warning("Early stopping is not available in 'dart' mode.")
	using_early_stopping <- FALSE
	# Remove the cb_early_stop() function if it was passed in to callbacks
	callbacks <- Filter(
	f = function(cb_func) {
	!identical(attr(cb_func, "name"), "cb_early_stop")
	}
	, x = callbacks
	)
	}

Best,

Serkan

[jameslamb]

Thanks for using LightGBM, and for the excellent report with reproducible examples!

At first I thought this was similar to #6406, but now I think I see the bug and it's slightly different.

This warning should only be raised if early stopping was enabled:

LightGBM/R-package/R/lgb.train.R

Lines 260 to 261 in 874f6fb

	if (using_dart) {
	warning("Early stopping is not available in 'dart' mode.")

Would you like to contribute a fix? It'd mean:

• change that in lgb.train()
• change that in lgb.cv()
• add unit tests (or modify the existing tests) to ensure the warning is not re-introduced

[serkor1]

Hi @jameslamb,

It would be a privilege to contribute with a fix. Allow me some time to get more familiar with the source code so I can contribute optimally.

I have skimmed through the unittests, and I haven't stumbled upon any tests that uses boosting = 'dart' - So I guess I'll write one from scratch; I assume that a correctly specified lgb.train() and lgb.cv() would run without any message, warninng or error, right?

If that is the case then I could write a unit test that loops through the various boosters with testthat::expect_no_condition(). If not a simple testthat::expect_no_error() or testthat::expect_no_warning() would suffice. What do you say?

Also, I could update the NEWS.md-file accordingly if needed be!

Best,

[jameslamb]

You could modify this test:

LightGBM/R-package/tests/testthat/test_parameters.R

Line 94 in d67ecf9

test_that("training should warn if you use 'dart' boosting, specified with 'boosting' or aliases", {

So that it checks that the warning is only raised when early_stopping_rounds > 0 is provided.

That'd cover lightgbm() and lgb.train() (lightgbm() calls lgb.train(), we can rely on that for this case to minimize how many new tests we're adding just to check a warning).

Then write one other similar test (in that same file is fine, right after that test) using lgb.cv().

[serkor1]

Hi @jameslamb,

I am running into the following R CMD check-issue: Error: could not find function "expect_no_warning" on Windows R 3.6.X - My initial guess was that expect_no_warning() weren't available at all, but the other R CMD check passes on other OSes. The tests have been run on my forked repository (if that has anything to do with it).

Unless you want to opt for a mix of tryCatch() and expect_warning() (which clearly passes your checks), I have no immediate solution - might be that my skills are too limited for... 🤷‍♀️

[jameslamb]

Thanks for working on this. Let's please keep all conversation about the exact implementation you're proposing in the place where the code lives: #6619. I'll respond over there.