[ci] [R-package] R CI jobs on mac, linux do not always fail if unit tests fail

[jameslamb]

Look at this build from #3598 : https://github.com/microsoft/LightGBM/pull/3598/checks?check_run_id=1477128773. It was marked successful in GitHub Actions, even though a unit test is failing.

Status: 1 ERROR

Similar jobs on Mac have the same test error, but are marked as failures by GitHub Actions. For example, https://github.com/microsoft/LightGBM/pull/3598/checks?check_run_id=1477128812.

It looks like the ones that failed only failed because they also had NOTEs.

So I think that the current R CI scripts aren't correctly accounting for the possibility of an R CMD check run that yields ERRORs but no NOTEs or WARNINGs. R CMD check should yield a non-0 exit code on ERRORs.

Windows jobs all correctly fail under these conditions, with a message like this

ERRORs have been found by R CMD check!

How to Fix This

Something in this code is not correctly capturing ERRORs on Mac and Linux:

LightGBM/.ci/test_r_package.sh

Lines 156 to 163 in c02917e

	# fails tests if either ERRORs or WARNINGs are thrown by
	# R CMD CHECK
	check_succeeded="yes"
	(
	R CMD check ${PKG_TARBALL} \
	--as-cran \
	--run-donttest \
	|| check_succeeded="no"

.

You can add a stop("hello") in any of the R unit tests in R-package/tests/testthat to simulate the situation where R CMD check results in Status: 1 ERROR, and then try to fix that script. It might be the case that the trickery with putting R CMD check into the background and polling (to avoid timeouts) results in some mistakes, and the solution could be to just grep for ERROR in the output of R CMD check, just like we do for WARNING:

LightGBM/.ci/test_r_package.sh

Lines 184 to 187 in c02917e

	if grep -q -R "WARNING" "$LOG_FILE_NAME"; then
	echo "WARNINGS have been found by R CMD check!"
	exit -1
	fi

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.