[R-package] Installation with pre-compiled DLL fails: 'cannot create dataset handle'

[abhmodi]

Hi,

I tried other methods mentioned in earlier threads/issues about fixing such issues but none seem to work.

Apart from the below mentioned reproducible example, I had tried with code mentioned on the test code mentioned here: https://github.com/microsoft/LightGBM/tree/master/R-package#install

Running that code doesn't give "[LightGBM] [Fatal] Check failed: num_data > 0 at D:\a\1\s\src\io\dataset.cpp, line 30 ." error but only about "cannot create dataset handle"

Environment info

Operating System: Windows 10

CPU/GPU model:

C++/Python/R version: R-3.5.2

LightGBM version or commit hash: 2.3.2

Error message

[LightGBM] [Fatal] Check failed: num_data > 0 at D:\a\1\s\src\io\dataset.cpp, line 30 .

Error in dataset$construct() :
lgb.Dataset.construct: cannot create Dataset handle

Reproducible examples

lgb.unloader(wipe=TRUE)
require(lightgbm)
data(ChickWeight) # Built in data set

ChickWeight <- sapply(ChickWeight, function(x) as.numeric(as.character(x)))

y <- ChickWeight[,"weight"]
x <- ChickWeight[,!colnames(ChickWeight)=="weight"]

dtrain <- lgb.Dataset(data = as.matrix(x),
label = as.numeric(y),
free_raw_data=FALSE
)
lgb.Dataset.construct(dtrain)

Other methods tried from earlier threads/issues

1. Re-installed LightGBM using RGUI
2. Tried with restarting R-sessions; same issues with another code

[jameslamb]

Hello @abhmodi . I tried running this code from your example on the latest version of the lightgbm R package:

library(lightgbm)
lgb.unloader(wipe=TRUE)
require(lightgbm)
data(ChickWeight) # Built in data set

ChickWeight <- sapply(ChickWeight, function(x) as.numeric(as.character(x)))

y <- ChickWeight[,"weight"]
x <- ChickWeight[,!colnames(ChickWeight)=="weight"]

dtrain <- lgb.Dataset(
    data = as.matrix(x)
    , label = as.numeric(y)
    , free_raw_data = FALSE
)
lgb.Dataset.construct(dtrain)

The only thing I changed was adding a library(lightgbm) at the top, since lgb.unloader() won't work if you have no loaded the library yet. When I ran this code, I did not get an error and it seems (from visual inspection) that everything worked correctly.

I am on macOS Mojave and R 3.6.1.

Can you please give me some more information?

1. The full log produced by running the example code above in a brand new session
2. Exact description of how you installed the lightgbm R package

[abhmodi]

Hi @jameslamb, thank you for getting back. Please find the details below and do let me know if you need any other information.

I am using R-3.5.2 and Windows 10

Steps I used for installing lightgbm R package

1. Installed CMake and rtools 3.5
2. Added bin folders of both to the path variable
3. Used D:/something/something/LightGBM as the directory to clone the files
4. Downloaded pre-compiled DLL and copied in "D:/something/something/LightGBM/" folder
5. Moved CMakelist file to ...".\LightGBM\R-package\inst\bin"
6. Changed install.lib file to set use precompile = TRUE
7. Changed USE_D35 in CMakelist file to ON
8. Ran rbuild.R (from RGui)
   a. Received error saying lib_lightgbm.dll couldn't be found
   b. Added print statement in the install.lib file to find where it is looking for the DLL file (it was temp folder)
   c. Added src <- ".\LightGBM\lib_lightgbm.dll" in the install.lib file at line number 140

Please find below the log that I got from running the code above:

[jameslamb]

hmmmm ok. Thank you for the summary! I have a few more questions below. I'm not as familiar with the "install with a precompiled dll" part of this project, so if I can't figure it out after hearing your answers to these questions we'll need to get help from @Laurae2

4. Downloaded pre-compiled DLL and copied in "D:/something/something/LightGBM/" folder

How did you download it? With lgb.dl() like mentioned in the https://github.com/microsoft/LightGBM/tree/master/R-package#installation-with-precompiled-dlllib-from-r-using-github? Can you show the exact command you ran?

5. Moved CMakelist file to ...".\LightGBM\R-package\inst\bin"

Why did you have to do this? That is already handled in https://github.com/microsoft/LightGBM/blob/master/build_r.R#L62-L67

7. Changed USE_D35 in CMakelist file to ON

Do you mean USE_R35? Can you check if you made a mistake.

8. Ran rbuild.R (from RGui)
   a. Received error saying lib_lightgbm.dll couldn't be found
   b. Added print statement in the install.lib file to find where it is looking for the DLL file (it was temp folder)
   c. Added src <- ".\LightGBM\lib_lightgbm.dll" in the install.lib file at line number 140

Do you mean build_r.R at the root of this repository? Can you share the full log produced when you ran this script (including the error message)?

[abhmodi]

Thank you @jameslamb for helping :)

How did you download it? With lgb.dl() like mentioned in the https://github.com/microsoft/LightGBM/tree/master/R-package#installation-with-precompiled-dlllib-from-r-using-github? Can you show the exact command you ran?\

I downloaded the precompiled DLL from here

Why did you have to do this? That is already handled in https://github.com/microsoft/LightGBM/blob/master/build_r.R#L62-L67

I did just reading a StackOverflow answer

Do you mean USE_R35? Can you check if you made a mistake.

Yes, I mean USE_R35. Sorry for the trouble

Do you mean build_r.R at the root of this repository? Can you share the full log produced when you ran this script (including the error message)?

Yes, I got the below message when I hadn't hardcoded the src variable (in install.libs.R file)

After manually updating SRC variable in install.libs.R file:

Again, thank you for all the help :)

[ningistine7]

hi guys, any solution for this problem, I have the same exact error although I installed and loaded it successfully ?

[jameslamb]

@Laurae2 Is this something you can help with? I haven't been able to produce this on Mac or Linux so I feel it might be Windows-specific, and I don't have an easily-available Windows environment to test in.

Sorry for the delay @ningistine7 and @abhmodi . Until we figure this out, could you install from source instead of the precompiled dll?

[Edimer]

Hi,

I have the same problem. I have followed these recommendations I was able to install it but I have this error:

Error in data $ construct ():
lgb.Dataset.construct: cannot create Dataset handle

I have R version 3.6.3 (windows).

Thanks for the help!

Question: Will lightgbm ever be available from the CRAN?

[jameslamb]

@Edimer thanks for the report. Sorry, but I cannot completely read those instructions (I only speak English). I do see that they're almost a year old...could you try with our current installation instructions.

As far as I can tell from your comment and the previous ones, maybe there is a problem with building from a precompiled lib_lightgbm.dll. I will look into it, but to try to unblock you...could you try building from source? We test installation from source frequently in our CI on Linux, Mac, and Windows (in progress, #2936), with all popular compilers, so I'm very confident that that path works.

Question: Will lightgbm ever be available from the CRAN?

Yes definitely! My primary focus right now is getting the package to CRAN. You can follow the progress in #629. I left the most recent update in #629 (comment).

[jameslamb]

@abhmodi hope it is ok, I changed the issue title based on the new evidence from @Edimer and @ningistine7

[Edimer]

@Edimer thanks for the report. Sorry, but I cannot completely read those instructions (I only speak English). I do see that they're almost a year old...could you try with our current installation instructions.

As far as I can tell from your comment and the previous ones, maybe there is a problem with building from a precompiled lib_lightgbm.dll. I will look into it, but to try to unblock you...could you try building from source? We test installation from source frequently in our CI on Linux, Mac, and Windows (in progress, #2936), with all popular compilers, so I'm very confident that that path works.

Question: Will lightgbm ever be available from the CRAN?

Yes definitely! My primary focus right now is getting the package to CRAN. You can follow the progress in #629. I left the most recent update in #629 (comment).

Ok, thanks @jameslamb for your help. I'll keep trying and keep an eye out for LightGBM in CRAN.

[Edimer]

Hi @jameslamb, I hope you are very well.

I have overcome the problem of 'cannot create dataset handle', however when I try to replicate the example with agaricus.train I have the following error:

Error in .Call (fun_name, ..., ret, call_state, PACKAGE = "lib_lightgbm"):
"LGBM_DatasetCreateFromCSC_R" not available for .Call () for package "lib_lightgbm"

Any ideas regarding this error?

I followed the installation instructions...but I still can't get it on Windows (on Linux yes).

Thank you.

[jameslamb]

Can you please clarify (copy and paste the exact code) how you installed the package? The README link you provided offers multiple alternatives for installing the R package.

[Edimer]

Hi @jameslamb, It worked!

I used the following code (which is in the installation instructions in R):

git clone --recursive https://github.com/microsoft/LightGBM
cd LightGBM
Rscript build_r.R

It worked for me with the following specs:

• Windows 10
• CMake 3.17
• Visual Studio 16 2019
• R 3.6.3
• Note: I think (I'm not completely sure) that it is very important not to have blank spaces in the installation folder, in order to avoid leaks.

I really appreciate your help and patience ... Your work is excellent!

[jameslamb]

Thanks @Edimer , glad it is working! I'm going to close this issue since the last response from other posters was a while ago.

For those finding this issue from search engines...please comment if you face an issue building the R package with a pre-compiled DLL and we will re-open it. If not, please open a new issue and we will be happy to help you!

[RaminZi]

Hi @jameslamb, It worked!
I used the following code (which is in the installation instructions in R):
git clone --recursive https://github.com/microsoft/LightGBM
cd LightGBM
Rscript build_r.R
It worked for me with the following specs:

Windows 10
CMake 3.17
Visual Studio 16 2019
R 3.6.3
Note: I think (I'm not completely sure) that it is very important not to have blank spaces in the installation folder, in order to avoid leaks.

I really appreciate your help and patience ... Your work is excellent!

Could you please explain how you solved the problem of this error?

Error in .Call(fun_name, ..., ret, call_state, PACKAGE = "lib_lightgbm") :
"LGBM_DatasetCreateFromMat_R" not available for .Call() for package "lib_lightgbm"

It was a real pain to install the package itself on my machine! But, after trying it for several days, it finally got installed. However, when I try to run anything, I get the same exact error. I have:
Windows 10, x64
Cmake 3.17.0
R 3.6.1
Visual Studio 16 2019

[jameslamb]

@RaminZi can you please tell me the exact code you ran to install LightGBM? When was the last time that you cloned from this repository?

Can you please run the steps from our installation documentation for the R package and share the log with me?

git clone --recursive https://github.com/microsoft/LightGBM
cd LightGBM
Rscript build_r.R

[RaminZi]

@RaminZi can you please tell me the exact code you ran to install LightGBM? When was the last time that you cloned from this repository?
Can you please run the steps from our installation documentation for the R package and share the log with me?
git clone --recursive https://github.com/microsoft/LightGBM
cd LightGBM
Rscript build_r.R

Wow! Thanks for your super fast response James! So, I followed the exact steps from the installation guide you just mentioned, and it did not work. It was not just me though! I was taking ML course this semester, and 3 other friends of mine also tried to install it but they could not (I think they all were using Windows). Anyway, after searching for a solution online for a few days, I tried almost everything I found but still no luck. Lastly, it was mentioned somewhere that it would not work if both versions of R (x32, x64) are installed on a machine. The solution was to use this "--no-multiarch", so it would not use both versions (?) (it's just my understanding). After that, I got it to work, but then I would get that error.
Honestly, Right after commenting here, I upgraded both my R (to 4) and Rstudio (to 1.2.5042). I'll try to follow the link you sent and will get back to you with my result.

Thanks again.
Ramin

[jameslamb]

When was the last time that you cloned from this repository?

^ this question is important. We know we had some recent issues with Visual Studio installations of the R package, which were just fixed (#2963).

I upgraded both my R (to 4) and Rstudio (to 1.2.5042).

We haven't tested with R4.0 yet so I am curious to see what you find. Please do let me know if our installation instructions work for you, and thanks for using LightGBM!

[RaminZi]

When was the last time that you cloned from this repository?

^ this question is important. We know we had some recent issues with Visual Studio installations of the R package, which were just fixed (#2963).

I upgraded both my R (to 4) and Rstudio (to 1.2.5042).

We haven't tested with R4.0 yet so I am curious to see what you find. Please do let me know if our installation instructions work for you, and thanks for using LightGBM!

It must have been before April 7th.
Sure. I'll get back to you soon.

Ramin

[RaminZi]

When was the last time that you cloned from this repository?

^ this question is important. We know we had some recent issues with Visual Studio installations of the R package, which were just fixed (#2963).

I upgraded both my R (to 4) and Rstudio (to 1.2.5042).

We haven't tested with R4.0 yet so I am curious to see what you find. Please do let me know if our installation instructions work for you, and thanks for using LightGBM!

Ok. So, this is what happened. I installed only the x64 version of R 4.0.0 (unchecked the x32 during the installation). Then, I ran those 3 lines you mentioned.

git clone --recursive https://github.com/microsoft/LightGBM
cd LightGBM
Rscript build_r.R

However, when was running the last line with that Rscript, it gives me an error. I'm attaching the error here.

After that, I tried the following process I got to install the package.

git clone --recursive https://github.com/microsoft/LightGBM
cd LightGBM
mkdir build
cd build
cmake -A x64 ..
cmake --build . --target ALL_BUILD --config Release

copy everything from release folder into Lightgbm root folder
copy cmakeList from root folder into R-package/inst/bin and change (USE_R35) to ON
change use_precompile in install.libs.R to TRUE

Run R.exe CMD INSTALL --build . inside lightgbm_r folder [Passed]

After all this, it was installed on my machine. However, when I run a code to see if it was installed correctly, I get the error I mentioned yesterday.

These are what I have on my machine:

Visual Studio 2019
Rtools35
cmake 3.17.0
boost_1_72_0-msvc-14.2-64

Thanks for your time.
Ramin

[jameslamb]

@RaminZi thanks for the thorough report! Now that you've updated to R4.0, I'd like to split the conversation. For your report about running the steps I suggested but with R4.0, please follow along at #3064

For the issue of using a precompiled DLL, you can stay here in this issue and we'll keep it open.

I think that #3064 is a higher priority right now, since that is a more common and better-supported way to build the R package, so that is going to be what I work on first. The approach you tried is not documented in our installation documentation for the R package because it is not recommended. You likely got the error you got there because building lib_lightgbm requires some R-specific flags passed to CMake, which is handled inside the R package in https://github.com/microsoft/LightGBM/blob/master/R-package/src/install.libs.R.

The CMake command you need to run is closer to

cmake -DBUILD_FOR_R=ON  -DUSE_R35=ON -DCMAKE_R_VERSION='4.0.0' -G "Visual Studio 16 2019" -A x64 ..

That's kind of complex, which is why we don't ask people to run it directly and instead include it inside code that eventually gets called when you run Rscript build_r.R.

Also, in the future could you please provide logs in text instead of screenshots? That makes it easier for others to find these issues from search engines, for example by searching for the error message "LGBM_DatasetCreateFromCSC_R".

Thanks again for using LightGBM!

[jameslamb]

I'm going to close this issue due to inactivity. A lot of improvement have been made to the R package since May, which I hope have made it easier to install.

Because of this, the ability to install from a pre-compiled DLL will be removed soon: #3360

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.