[docs][R] added R-package docs generation routines #2176
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
16 tasks
|
I'm not sure whether non-admins on RTD are able to observe protected branch Seems that logs are protected for wide access, so I'm pasting them here: Logs (clickable)From them it's seems that LightGBM is compiled twice despite the fact that I cut |
|
Seems that some examples in API references crash:
UPD: Fixed by marking |
StrikerRUS
commented
May 15, 2019
docs/conf.py
Outdated
| commands = """ | ||
| export PATH="/home/docs/.conda/bin:$PATH" | ||
| echo 'options(repos = "https://cran.rstudio.com")' > $HOME/.Rprofile | ||
| conda create -q -y -n r_env r-base r-devtools r-data.table r-jsonlite r-magrittr r-matrix r-testthat cmake |
There was a problem hiding this comment.
Without testthat documentation process crashes:
Found library file: /home/docs/checkouts/readthedocs.org/user_builds/lightgbm/checkouts/docs/lightgbm_r/src/lib_lightgbm.so to move to /tmp/RtmpgL11IH/devtools_install_4d06ff4063f/lightgbm/libs
Warning: unknown option ‘--keep_empty-dirs’
* installing to library ‘/home/docs/.conda/envs/r_env/lib/R/library’
* installing *source* package ‘lightgbm’ ...
** libs
installing via 'install.libs.R' to /home/docs/.conda/envs/r_env/lib/R/library/lightgbm
** R
** data
** demo
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (lightgbm)
Updating lightgbm documentation
Loading lightgbm
Re-compiling lightgbm
'/home/docs/.conda/envs/r_env/lib/R/bin/R' --no-site-file --no-environ \
--no-save --no-restore --quiet CMD INSTALL \
'/home/docs/checkouts/readthedocs.org/user_builds/lightgbm/checkouts/docs/lightgbm_r' \
--library='/tmp/RtmpgL11IH/devtools_install_4d06ff4063f' --no-R --no-data \
--no-help --no-demo --no-inst --no-docs --no-exec --no-multiarch \
--no-test-load --preclean
* installing *source* package ‘lightgbm’ ...
** libs
installing via 'install.libs.R' to /tmp/RtmpgL11IH/devtools_install_4d06ff4063f/lightgbm
* DONE (lightgbm)
Loading required package: R6
Error in loadNamespace(name) : there is no package called ‘testthat’
Calls: build_reference ... tryCatch -> tryCatchList -> tryCatchOne -> <Anonymous>
Execution halted
Is it expected?
There was a problem hiding this comment.
A user reported a similar error in #2153 (also using conda as an R installer, just like you are here). We haven't found a resolution there yet.
The reporter on that issue solved their problem by completely removing their conda and reinstalling, but that is less than ideal 😂
It's not expected that you should need testthat to build the package or its docs. Just so I understand...are you saying "I had to add r-testthat to my install command and that worked but it's weird"? Or is this currently not working?
There was a problem hiding this comment.
@jameslamb The first one:
"I had to add r-testthat to my install command and that worked but it's weird"
Everything is fine with testthat. You can observe it on the site right now.
I just wanted to know is there a workaround to drop this dependency (who knows, due to the build time limits, maybe in the future even several seconds will matter...). Thanks for the link - now I know that's I'm not alone with this problem. 😃
FYI #1702 (comment) first compile: second compile: |
|
I think we can modify the appveyor.yml in https://github.com/BruceZhaoR/lightgbm_r/blob/master/appveyor.yml, add it to https://github.com/microsoft/LightGBM/blob/master/.appveyor.yml, then push the R docs to the |
Thanks! However, I use much less code here, and the only place where compilation occurs in my opinion is
I think that keeping all docs in one place is good idea. What's why I decided to accept the challenge of R installation on RTD. The main idea of this PR is to get rid of maintaining of one more site (GH pages) and the overhead produced by communication with GitHub. FYI, Appveyor probably is bad candidate for this, because it offers only one concurrent job for open source projects in free plan and is extremely slow. |
StrikerRUS
force-pushed
the
docs
branch
2 times, most recently
from
ccf5c35 to
8275d7e
Compare
|
Problem about that builds with multiple builders exceeded time limit has been solved by generating docs for |
|
According to the logs, it seems to be that all broken examples (#2176 (comment)) appear after a crash in All examples which are executed before this function are fine. And all further ones are broken: Here is the place where crash happens: |
StrikerRUS
force-pushed
the
docs
branch
7 times, most recently
from
e75a650 to
824651d
Compare
|
@StrikerRUS Make sure to comment all the Lines like this: https://github.com/microsoft/LightGBM/pull/1208/files#diff-c3a2935b3c14bfb7c9669eac2d0f26ffR31 (example from an old PR) |
|
Thanks a lot @Laurae2 ! Indeed, |
StrikerRUS
force-pushed
the
docs
branch
3 times, most recently
from
d995a17 to
524a99b
Compare
StrikerRUS
commented
May 16, 2019
build_r_site.R
Outdated
| clean_site() | ||
| init_site() | ||
| build_home(preview = FALSE, quiet = FALSE) | ||
| build_reference(lazy = FALSE, document = FALSE, |
There was a problem hiding this comment.
It's quite strange, but removing devtools::document() and setting document = TRUE result in broken page:
Log (clickable):
── Initialising site ───────────────────────────────────────────────────────────
Copying '../../../../../../../.conda/envs/r_env/lib/R/library/pkgdown/assets/docsearch.css' to 'docsearch.css'
Copying '../../../../../../../.conda/envs/r_env/lib/R/library/pkgdown/assets/docsearch.js' to 'docsearch.js'
Copying '../../../../../../../.conda/envs/r_env/lib/R/library/pkgdown/assets/link.svg' to 'link.svg'
Copying '../../../../../../../.conda/envs/r_env/lib/R/library/pkgdown/assets/pkgdown.css' to 'pkgdown.css'
Copying '../../../../../../../.conda/envs/r_env/lib/R/library/pkgdown/assets/pkgdown.js' to 'pkgdown.js'
── Building home ───────────────────────────────────────────────────────────────
Writing 'authors.html'
Writing 'LICENSE-text.html'
Writing 'index.html'
── Building function reference ─────────────────────────────────────────────────
-- The C compiler identification is GNU 7.2.0
-- The CXX compiler identification is GNU 7.2.0
-- Check for working C compiler: /home/docs/.conda/envs/r_env/bin/x86_64-conda_cos6-linux-gnu-cc
-- Check for working C compiler: /home/docs/.conda/envs/r_env/bin/x86_64-conda_cos6-linux-gnu-cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /home/docs/.conda/envs/r_env/bin/x86_64-conda_cos6-linux-gnu-c++
-- Check for working CXX compiler: /home/docs/.conda/envs/r_env/bin/x86_64-conda_cos6-linux-gnu-c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found OpenMP_C: -fopenmp (found version "4.5")
-- Found OpenMP_CXX: -fopenmp (found version "4.5")
-- Found OpenMP: TRUE (found version "4.5")
-- Configuring done
-- Generating done
-- Build files have been written to: /home/docs/checkouts/readthedocs.org/user_builds/lightgbm/checkouts/docs/lightgbm_r/src/build
Scanning dependencies of target _lightgbm
[ 3%] Building CXX object CMakeFiles/_lightgbm.dir/src/c_api.cpp.o
[ 6%] Building CXX object CMakeFiles/_lightgbm.dir/src/lightgbm_R.cpp.o
[ 9%] Building CXX object CMakeFiles/_lightgbm.dir/src/application/application.cpp.o
[ 12%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/boosting.cpp.o
[ 15%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt.cpp.o
[ 18%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt_model_text.cpp.o
[ 21%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt_prediction.cpp.o
[ 25%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/prediction_early_stop.cpp.o
[ 28%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/bin.cpp.o
[ 31%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/config.cpp.o
[ 34%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/config_auto.cpp.o
[ 37%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/dataset.cpp.o
[ 40%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/dataset_loader.cpp.o
[ 43%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/file_io.cpp.o
[ 46%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/json11.cpp.o
[ 50%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/metadata.cpp.o
[ 53%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/parser.cpp.o
[ 56%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/tree.cpp.o
[ 59%] Building CXX object CMakeFiles/_lightgbm.dir/src/metric/dcg_calculator.cpp.o
[ 62%] Building CXX object CMakeFiles/_lightgbm.dir/src/metric/metric.cpp.o
[ 65%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linker_topo.cpp.o
[ 68%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linkers_mpi.cpp.o
[ 71%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linkers_socket.cpp.o
[ 75%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/network.cpp.o
[ 78%] Building CXX object CMakeFiles/_lightgbm.dir/src/objective/objective_function.cpp.o
[ 81%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/data_parallel_tree_learner.cpp.o
[ 84%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/feature_parallel_tree_learner.cpp.o
[ 87%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/gpu_tree_learner.cpp.o
[ 90%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/serial_tree_learner.cpp.o
[ 93%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/tree_learner.cpp.o
[ 96%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/voting_parallel_tree_learner.cpp.o
[100%] Linking CXX shared library ../lib_lightgbm.so
[100%] Built target _lightgbm
Found library file: /home/docs/checkouts/readthedocs.org/user_builds/lightgbm/checkouts/docs/lightgbm_r/src/lib_lightgbm.so to move to /tmp/RtmpFiudjw/devtools_install_325eec6510/lightgbm/libsWriting NAMESPACE
Writing NAMESPACE
Writing predict.lgb.Booster.Rd
Writing lgb.load.Rd
Writing lgb.save.Rd
Writing lgb.dump.Rd
Writing lgb.get.eval.result.Rd
Writing lgb.Dataset.Rd
Writing lgb.Dataset.create.valid.Rd
Writing lgb.Dataset.construct.Rd
Writing dim.Rd
Writing dimnames.lgb.Dataset.Rd
Writing slice.Rd
Writing getinfo.Rd
Writing setinfo.Rd
Writing lgb.Dataset.set.categorical.Rd
Writing lgb.Dataset.set.reference.Rd
Writing lgb.Dataset.save.Rd
Writing lgb.cv.Rd
Writing lgb.importance.Rd
Writing lgb.interprete.Rd
Writing lgb.model.dt.tree.Rd
Writing lgb.plot.importance.Rd
Writing lgb.plot.interpretation.Rd
Writing lgb.prepare.Rd
Writing lgb.prepare2.Rd
Writing lgb.prepare_rules.Rd
Writing lgb.prepare_rules2.Rd
Writing lgb.train.Rd
Writing lgb.unloader.Rd
Writing lgb_shared_params.Rd
Writing lightgbm.Rd
Writing agaricus.train.Rd
Writing agaricus.test.Rd
Writing bank.Rd
Writing readRDS.lgb.Booster.Rd
Writing saveRDS.lgb.Booster.Rd
Writing 'reference/index.html'
─ installing *source* package ‘lightgbm’ ...
** libs
installing via 'install.libs.R' to /tmp/RtmpFiudjw/devtools_install_32548d1273d/lightgbm
-- Found OpenMP_C: -fopenmp
-- Found OpenMP_CXX: -fopenmp
-- Found OpenMP: TRUE
-- Configuring done
-- Generating done
-- Build files have been written to: /home/docs/checkouts/readthedocs.org/user_builds/lightgbm/checkouts/docs/lightgbm_r/src/build
[ 3%] Linking CXX shared library ../lib_lightgbm.so
[100%] Built target _lightgbm
Found library file: /home/docs/checkouts/readthedocs.org/user_builds/lightgbm/checkouts/docs/lightgbm_r/src/lib_lightgbm.so to move to /tmp/RtmpFiudjw/devtools_install_32548d1273d/lightgbm/libs
Found library file: /home/docs/checkouts/readthedocs.org/user_builds/lightgbm/checkouts/docs/lightgbm_r/src/lib_lightgbm.so to move to /tmp/RtmpFiudjw/devtools_install_32548d1273d/lightgbm/libs* DONE (lightgbm)
Updating lightgbm documentation
Loading lightgbm
Re-compiling lightgbm
'/home/docs/.conda/envs/r_env/lib/R/bin/R' --no-site-file --no-environ \
--no-save --no-restore --quiet CMD INSTALL \
'/home/docs/checkouts/readthedocs.org/user_builds/lightgbm/checkouts/docs/lightgbm_r' \
--library='/tmp/RtmpFiudjw/devtools_install_325eec6510' --no-R --no-data \
--no-help --no-demo --no-inst --no-docs --no-exec --no-multiarch \
--no-test-load --preclean
* installing *source* package ‘lightgbm’ ...
** libs
installing via 'install.libs.R' to /tmp/RtmpFiudjw/devtools_install_325eec6510/lightgbm
* DONE (lightgbm)
Loading required package: R6
Updating roxygen version in /home/docs/checkouts/readthedocs.org/user_builds/lightgbm/checkouts/docs/lightgbm_r/DESCRIPTION
Warning: In '_pkgdown.yml', topic must be a valid R expression.
Problem topic: `\`agaricus.test\``
Warning: In '_pkgdown.yml', topic must be a valid R expression.
Problem topic: `\`agaricus.train\``
Warning: In '_pkgdown.yml', topic must be a valid R expression.
Problem topic: `\`bank\``
Warning: In '_pkgdown.yml', topic must evaluate to a numeric vector.
Problem topic: `\`dim.lgb.Dataset\``
Actual value: function (x, ...)
{
if (!lgb.is.Dataset(x)) {
stop("dim.lgb.Dataset: input data should be an lgb.Dataset object")
}
x$dim()
}
Warning: In '_pkgdown.yml', topic must evaluate to a numeric vector.
Problem topic: `\`dimnames.lgb.Dataset\``
Actual value: function (x)
{
if (!lgb.is.Dataset(x)) {
stop("dimnames.lgb.Dataset: input data should be an lgb.Dataset object")
}
list(NULL, x$get_colnames())
}
Warning: In '_pkgdown.yml', topic must evaluate to a numeric vector.
Problem topic: `\`getinfo\``
Actual value: function (dataset, ...)
{
UseMethod("getinfo")
}
Warning: In '_pkgdown.yml', topic must evaluate to a numeric vector.
Problem topic: `\`setinfo\``
Actual value: function (dataset, ...)
{
UseMethod("setinfo")
}
Warning: In '_pkgdown.yml', topic must evaluate to a numeric vector.
Problem topic: `\`slice\``
Actual value: function (dataset, ...)
{
UseMethod("slice")
}
Warning: In '_pkgdown.yml', topic must evaluate to a numeric vector.
Problem topic: `\`lgb.Dataset.construct\``
Actual value: function (dataset)
{
if (!lgb.is.Dataset(dataset)) {
stop("lgb.Dataset.construct: input data should be an lgb.Dataset object")
}
invisible(dataset$construct())
}
Warning: In '_pkgdown.yml', topic must evaluate to a numeric vector.
Problem topic: `\`lgb.Dataset.create.valid\``
Actual value: function (dataset, data, info = list(), ...)
{
if (!lgb.is.Dataset(dataset)) {
stop("lgb.Dataset.create.valid: input data should be an lgb.Dataset object")
}
invisible(dataset$create_valid(data, info, ...))
}
Warning: In '_pkgdown.yml', topic must evaluate to a numeric vector.
Problem topic: `\`lgb.Dataset\``
Actual value: function (data, params = list(), reference = NULL, colnames = NULL,
categorical_feature = NULL, free_raw_data = TRUE, info = list(),
...)
{
invisible(Dataset$new(data, params, reference, colnames,
categorical_feature, NULL, free_raw_data, NULL, info,
...))
}
Warning: In '_pkgdown.yml', topic must evaluate to a numeric vector.
Problem topic: `\`lgb.Dataset.save\``
Actual value: function (dataset, fname)
{
if (!lgb.is.Dataset(dataset)) {
stop("lgb.Dataset.set: input dataset should be an lgb.Dataset object")
}
if (!is.character(fname)) {
stop("lgb.Dataset.set: fname should be a character or a file connection")
}
invisible(dataset$save_binary(fname))
}
...
<A lot of errors for every function>
There was a problem hiding this comment.
@StrikerRUS Can you use roxygen2::roxygenize("mypackagefolder", roclets = "rd") instead of building the package with devtools::document() to update the Rd documentation?
Note that the rd documents you removed must be generated and still stored in LightGBM repository. They are essential for the R (offline) documentation and should not be generated by the user.
There was a problem hiding this comment.
Note that the rd documents you removed must be generated and still stored in LightGBM repository.
Thanks! I understand this clearly. At this moment this PR is in "Draft" stage and I removed all Rd files just to ensure that they can be generated on the RTD site, but not reused.
However, I'd like to ask someone else to regenerate these files (I have no R installed on my machine) right after this PR or I can place them back but they will be outdated because I modified R source files a little bit.
There was a problem hiding this comment.
Just as you should never stare directly into the sun, you should never look directly at the devtools or pkgdown source code
StrikerRUS
force-pushed
the
docs
branch
3 times, most recently
from
43b30c3 to
12992c1
Compare
|
Thanks a lot @jameslamb for your re-review! |
|
@Laurae2 @guolinke @henry0312 Your turn, guys! |
guolinke
approved these changes
Jun 2, 2019
|
Something has been updated and now build is failing. So, I pinned all versions which allow to build successfully. Log (clickable): |
|
@henry0312 Can you please give a brief glance at RTD new config file and |
|
ping @henry0312 |
guolinke
approved these changes
Aug 4, 2019
|
@henry0312 I'm going to merge this PR this weekend. Do you have some time to drop your review? |
|
Merging this. @henry0312 Feel free to submit your review any time you can. I'll be happy to consider changes (if any) in a separate PR! |
Merged
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is based on https://github.com/BruceZhaoR/lightgbm_r.
CI is temporarily disabled due to frequent commints.
The main problem here is that our docs builds are limited to 900 seconds. Currently, common build takes approximately 380 seconds. It's without
build_articles(),build_tutorials()andbuild_news().Due to the same reason I moved RTD configuration from site admin panel to
.readthedocs.ymlfile, because it's the only way to disable building archive with HTML pages (htmlzip). I think that at the end we should leave onlyformats: [pdf], because I can't imagine who needs EPUB or archived version of our docs. Moreover, configuration file is easier to maintain and gives a way to propose configuration updates not only by admins on RTD.https://docs.readthedocs.io/en/stable/config-file/v2.html#formats
We cannot use
apt-getto install R because RTD's docker is launched under the user without root access:https://github.com/rtfd/readthedocs-docker-images/blob/bc963e9c634d35daa6ab1eddac2e681a99b8553e/Dockerfile#L138-L143
I tried to compile R from sources (and succeeded in it), but of course it's not enough time for other stuff! So, conda is the only way to setup R.