Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[R-package] move more finalizer logic into C++ side to address memory leaks #4353

Merged
merged 7 commits into from
Jun 13, 2021

Conversation

jameslamb
Copy link
Collaborator

@jameslamb jameslamb commented Jun 8, 2021

Contributes to #4282 (fixing memory leaks in the R package). Specifically, this addresses the following comment from that issue:

Functions that return an R external pointer are wrapped into R6 classes, with the finalizer for the external pointer being under the R6 finalizer. If an error occurs during initialization (when calling Class$new(...), like it is done when creating a dataset), the finalizer will not be called, thus leaking the C++ object (e.g. a dataset or a model). This could be solved by setting the finalizer in the R external pointer object itself.

From https://cran.r-project.org/doc/manuals/r-release/R-exts.html#External-pointers-and-weak-references:

An external pointer object can have a finalizer, a piece of code to be run when the object is garbage collected. This can be R code or C code, and the various interfaces are, respectively.

Some examples:

Changes in this PR

  • Adds finalizers for Booster and Dataset objects at the C++ size, using R_RegisterCFinalizerEx()
  • removes checks like !lgb.is.null.handle() in R6 objects' finalize() calls, and moved that logic into the corresponding C++ functions
  • removes uses of testthat::skip() on tests where valgrind detects memory leaks

@jameslamb jameslamb added the fix label Jun 8, 2021
@jameslamb
Copy link
Collaborator Author

jameslamb commented Jun 8, 2021

/gha run r-valgrind

Workflow R valgrind tests has been triggered! 🚀
https://github.com/microsoft/LightGBM/actions/runs/917119658

Status: failure ❌.

@jameslamb
Copy link
Collaborator Author

jameslamb commented Jun 9, 2021

Alright I still think this change is worth making, but it does not allow those skipped tests to pass valgrind. Probably because those tests expose code that suffers from several of the memory leak sources mentioned in #4282.

==2689== LEAK SUMMARY:
==2689==    definitely lost: 1,058,175 bytes in 15 blocks
==2689==    indirectly lost: 1,702 bytes in 11 blocks
==2689==      possibly lost: 336 bytes in 1 blocks
==2689==    still reachable: 299,629,382 bytes in 56,133 blocks
==2689==                       of which reachable via heuristic:
==2689==                         newarray           : 4,264 bytes in 1 blocks
==2689==         suppressed: 0 bytes in 0 blocks
full logs (click me)
==2689== 
==2689== HEAP SUMMARY:
==2689==     in use at exit: 300,689,595 bytes in 56,160 blocks
==2689==   total heap usage: 4,060,849 allocs, 4,004,689 frees, 6,242,399,693 bytes allocated
==2689== 
==2689== 16 bytes in 1 blocks are definitely lost in loss record 30 of 2,832
==2689==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2689==    by 0x14E812A9: __gnu_cxx::new_allocator<unsigned long>::allocate(unsigned long, void const*) (new_allocator.h:115)
==2689==    by 0x14E7F25D: std::allocator_traits<std::allocator<unsigned long> >::allocate(std::allocator<unsigned long>&, unsigned long) (alloc_traits.h:460)
==2689==    by 0x14E95E61: std::_Vector_base<unsigned long, std::allocator<unsigned long> >::_M_allocate(unsigned long) (stl_vector.h:346)
==2689==    by 0x14E9555E: std::_Vector_base<unsigned long, std::allocator<unsigned long> >::_M_create_storage(unsigned long) (stl_vector.h:361)
==2689==    by 0x14E93904: std::_Vector_base<unsigned long, std::allocator<unsigned long> >::_Vector_base(unsigned long, std::allocator<unsigned long> const&) (stl_vector.h:305)
==2689==    by 0x14E92532: std::vector<unsigned long, std::allocator<unsigned long> >::vector(unsigned long, std::allocator<unsigned long> const&) (stl_vector.h:511)
==2689==    by 0x14E89E27: LightGBM::GBDT::SaveModelToString[abi:cxx11](int, int, int) const (gbdt_model_text.cpp:355)
==2689==    by 0x151A8D05: LightGBM::Booster::SaveModelToString[abi:cxx11](int, int, int) const (c_api.cpp:732)
==2689==    by 0x1519BF8C: LGBM_BoosterSaveModelToString (c_api.cpp:2225)
==2689==    by 0x151C8B9E: LGBM_BoosterSaveModelToString_R (lightgbm_R.cpp:684)
==2689==    by 0x494214E: R_doDotCall (dotcode.c:604)
==2689== 
==2689== 19 bytes in 1 blocks are definitely lost in loss record 32 of 2,832
==2689==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2689==    by 0x737286C: void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==2689==    by 0x151876CA: LightGBM::Config::Config() (config.h:34)
==2689==    by 0x1519691A: LGBM_DatasetUpdateParamChecking (c_api.cpp:1496)
==2689==    by 0x151C5E70: LGBM_DatasetUpdateParamChecking_R (lightgbm_R.cpp:308)
==2689==    by 0x4942116: R_doDotCall (dotcode.c:601)
==2689==    by 0x494D2C8: do_dotcall (dotcode.c:1281)
==2689==    by 0x498C952: Rf_eval (eval.c:830)
==2689==    by 0x499173F: do_begin (eval.c:2517)
==2689==    by 0x498C6F9: Rf_eval (eval.c:802)
==2689==    by 0x498BDEE: forcePromise (eval.c:555)
==2689==    by 0x4996F6B: FORCE_PROMISE (eval.c:5136)
==2689== 
==2689== 19 bytes in 1 blocks are definitely lost in loss record 33 of 2,832
==2689==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2689==    by 0x737286C: void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==2689==    by 0x151876CA: LightGBM::Config::Config() (config.h:34)
==2689==    by 0x151A436C: LightGBM::Booster::CheckDatasetResetConfig(LightGBM::Config const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) (c_api.cpp:187)
==2689==    by 0x15196965: LGBM_DatasetUpdateParamChecking (c_api.cpp:1499)
==2689==    by 0x151C5E70: LGBM_DatasetUpdateParamChecking_R (lightgbm_R.cpp:308)
==2689==    by 0x4942116: R_doDotCall (dotcode.c:601)
==2689==    by 0x494D2C8: do_dotcall (dotcode.c:1281)
==2689==    by 0x498C952: Rf_eval (eval.c:830)
==2689==    by 0x499173F: do_begin (eval.c:2517)
==2689==    by 0x498C6F9: Rf_eval (eval.c:802)
==2689==    by 0x498BDEE: forcePromise (eval.c:555)
==2689== 
==2689== 20 bytes in 1 blocks are definitely lost in loss record 34 of 2,832
==2689==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2689==    by 0x737286C: void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==2689==    by 0x15187A14: LightGBM::Config::Config() (config.h:34)
==2689==    by 0x1519691A: LGBM_DatasetUpdateParamChecking (c_api.cpp:1496)
==2689==    by 0x151C5E70: LGBM_DatasetUpdateParamChecking_R (lightgbm_R.cpp:308)
==2689==    by 0x4942116: R_doDotCall (dotcode.c:601)
==2689==    by 0x494D2C8: do_dotcall (dotcode.c:1281)
==2689==    by 0x498C952: Rf_eval (eval.c:830)
==2689==    by 0x499173F: do_begin (eval.c:2517)
==2689==    by 0x498C6F9: Rf_eval (eval.c:802)
==2689==    by 0x498BDEE: forcePromise (eval.c:555)
==2689==    by 0x4996F6B: FORCE_PROMISE (eval.c:5136)
==2689== 
==2689== 20 bytes in 1 blocks are definitely lost in loss record 35 of 2,832
==2689==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2689==    by 0x737286C: void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==2689==    by 0x15187A14: LightGBM::Config::Config() (config.h:34)
==2689==    by 0x151A436C: LightGBM::Booster::CheckDatasetResetConfig(LightGBM::Config const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) (c_api.cpp:187)
==2689==    by 0x15196965: LGBM_DatasetUpdateParamChecking (c_api.cpp:1499)
==2689==    by 0x151C5E70: LGBM_DatasetUpdateParamChecking_R (lightgbm_R.cpp:308)
==2689==    by 0x4942116: R_doDotCall (dotcode.c:601)
==2689==    by 0x494D2C8: do_dotcall (dotcode.c:1281)
==2689==    by 0x498C952: Rf_eval (eval.c:830)
==2689==    by 0x499173F: do_begin (eval.c:2517)
==2689==    by 0x498C6F9: Rf_eval (eval.c:802)
==2689==    by 0x498BDEE: forcePromise (eval.c:555)
==2689== 
==2689== 28 bytes in 1 blocks are definitely lost in loss record 51 of 2,832
==2689==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2689==    by 0x737286C: void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==2689==    by 0x151879A2: LightGBM::Config::Config() (config.h:34)
==2689==    by 0x1519691A: LGBM_DatasetUpdateParamChecking (c_api.cpp:1496)
==2689==    by 0x151C5E70: LGBM_DatasetUpdateParamChecking_R (lightgbm_R.cpp:308)
==2689==    by 0x4942116: R_doDotCall (dotcode.c:601)
==2689==    by 0x494D2C8: do_dotcall (dotcode.c:1281)
==2689==    by 0x498C952: Rf_eval (eval.c:830)
==2689==    by 0x499173F: do_begin (eval.c:2517)
==2689==    by 0x498C6F9: Rf_eval (eval.c:802)
==2689==    by 0x498BDEE: forcePromise (eval.c:555)
==2689==    by 0x4996F6B: FORCE_PROMISE (eval.c:5136)
==2689== 
==2689== 28 bytes in 1 blocks are definitely lost in loss record 52 of 2,832
==2689==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2689==    by 0x737286C: void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==2689==    by 0x151879A2: LightGBM::Config::Config() (config.h:34)
==2689==    by 0x151A436C: LightGBM::Booster::CheckDatasetResetConfig(LightGBM::Config const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) (c_api.cpp:187)
==2689==    by 0x15196965: LGBM_DatasetUpdateParamChecking (c_api.cpp:1499)
==2689==    by 0x151C5E70: LGBM_DatasetUpdateParamChecking_R (lightgbm_R.cpp:308)
==2689==    by 0x4942116: R_doDotCall (dotcode.c:601)
==2689==    by 0x494D2C8: do_dotcall (dotcode.c:1281)
==2689==    by 0x498C952: Rf_eval (eval.c:830)
==2689==    by 0x499173F: do_begin (eval.c:2517)
==2689==    by 0x498C6F9: Rf_eval (eval.c:802)
==2689==    by 0x498BDEE: forcePromise (eval.c:555)
==2689== 
==2689== 32 (24 direct, 8 indirect) bytes in 1 blocks are definitely lost in loss record 56 of 2,832
==2689==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2689==    by 0x14E663A8: __gnu_cxx::new_allocator<std::vector<double, std::allocator<double> > >::allocate(unsigned long, void const*) (new_allocator.h:115)
==2689==    by 0x14E64CC2: std::allocator_traits<std::allocator<std::vector<double, std::allocator<double> > > >::allocate(std::allocator<std::vector<double, std::allocator<double> > >&, unsigned long) (alloc_traits.h:460)
==2689==    by 0x14E62BE3: std::_Vector_base<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >::_M_allocate(unsigned long) (stl_vector.h:346)
==2689==    by 0x14E5FB7C: std::_Vector_base<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >::_M_create_storage(unsigned long) (stl_vector.h:361)
==2689==    by 0x14E5B226: std::_Vector_base<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >::_Vector_base(unsigned long, std::allocator<std::vector<double, std::allocator<double> > > const&) (stl_vector.h:305)
==2689==    by 0x14F1CB7A: std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >::vector(unsigned long, std::vector<double, std::allocator<double> > const&, std::allocator<std::vector<double, std::allocator<double> > > const&) (stl_vector.h:524)
==2689==    by 0x14F16A77: LightGBM::Config::GetAucMuWeights() (config.cpp:161)
==2689==    by 0x14F17282: LightGBM::Config::Set(std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) (config.cpp:221)
==2689==    by 0x15196933: LGBM_DatasetUpdateParamChecking (c_api.cpp:1497)
==2689==    by 0x151C5E70: LGBM_DatasetUpdateParamChecking_R (lightgbm_R.cpp:308)
==2689==    by 0x4942116: R_doDotCall (dotcode.c:601)
==2689== 
==2689== 32 bytes in 1 blocks are definitely lost in loss record 57 of 2,832
==2689==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2689==    by 0x14E65CAB: __gnu_cxx::new_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::allocate(unsigned long, void const*) (new_allocator.h:115)
==2689==    by 0x14E64277: std::allocator_traits<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::allocate(std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >&, unsigned long) (alloc_traits.h:460)
==2689==    by 0x14E61D25: std::_Vector_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_allocate(unsigned long) (stl_vector.h:346)
==2689==    by 0x14F1D504: void std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_realloc_insert<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>(__gnu_cxx::__normal_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (vector.tcc:440)
==2689==    by 0x14F1CB03: std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::push_back(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (stl_vector.h:1198)
==2689==    by 0x14F15334: LightGBM::ParseMetrics(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*) (config.cpp:74)
==2689==    by 0x14F15635: LightGBM::GetMetricType(std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*) (config.cpp:92)
==2689==    by 0x14F171B0: LightGBM::Config::Set(std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) (config.cpp:211)
==2689==    by 0x151A4385: LightGBM::Booster::CheckDatasetResetConfig(LightGBM::Config const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) (c_api.cpp:188)
==2689==    by 0x15196965: LGBM_DatasetUpdateParamChecking (c_api.cpp:1499)
==2689==    by 0x151C5E70: LGBM_DatasetUpdateParamChecking_R (lightgbm_R.cpp:308)
==2689== 
==2689== 32 (24 direct, 8 indirect) bytes in 1 blocks are definitely lost in loss record 58 of 2,832
==2689==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2689==    by 0x14E663A8: __gnu_cxx::new_allocator<std::vector<double, std::allocator<double> > >::allocate(unsigned long, void const*) (new_allocator.h:115)
==2689==    by 0x14E64CC2: std::allocator_traits<std::allocator<std::vector<double, std::allocator<double> > > >::allocate(std::allocator<std::vector<double, std::allocator<double> > >&, unsigned long) (alloc_traits.h:460)
==2689==    by 0x14E62BE3: std::_Vector_base<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >::_M_allocate(unsigned long) (stl_vector.h:346)
==2689==    by 0x14E5FB7C: std::_Vector_base<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >::_M_create_storage(unsigned long) (stl_vector.h:361)
==2689==    by 0x14E5B226: std::_Vector_base<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >::_Vector_base(unsigned long, std::allocator<std::vector<double, std::allocator<double> > > const&) (stl_vector.h:305)
==2689==    by 0x14F1CB7A: std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >::vector(unsigned long, std::vector<double, std::allocator<double> > const&, std::allocator<std::vector<double, std::allocator<double> > > const&) (stl_vector.h:524)
==2689==    by 0x14F16A77: LightGBM::Config::GetAucMuWeights() (config.cpp:161)
==2689==    by 0x14F17282: LightGBM::Config::Set(std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) (config.cpp:221)
==2689==    by 0x151A4385: LightGBM::Booster::CheckDatasetResetConfig(LightGBM::Config const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) (c_api.cpp:188)
==2689==    by 0x15196965: LGBM_DatasetUpdateParamChecking (c_api.cpp:1499)
==2689==    by 0x151C5E70: LGBM_DatasetUpdateParamChecking_R (lightgbm_R.cpp:308)
==2689== 
==2689== 336 bytes in 1 blocks are possibly lost in loss record 164 of 2,832
==2689==    at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2689==    by 0x40149CA: allocate_dtv (dl-tls.c:286)
==2689==    by 0x40149CA: _dl_allocate_tls (dl-tls.c:532)
==2689==    by 0x5721322: allocate_stack (allocatestack.c:622)
==2689==    by 0x5721322: pthread_create@@GLIBC_2.2.5 (pthread_create.c:660)
==2689==    by 0x56EFDDA: ??? (in /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0)
==2689==    by 0x56E78E0: GOMP_parallel (in /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0)
==2689==    by 0x151953CB: LGBM_DatasetCreateFromCSC (c_api.cpp:1303)
==2689==    by 0x151C4611: LGBM_DatasetCreateFromCSC_R (lightgbm_R.cpp:94)
==2689==    by 0x4942314: R_doDotCall (dotcode.c:624)
==2689==    by 0x494D2C8: do_dotcall (dotcode.c:1281)
==2689==    by 0x498C952: Rf_eval (eval.c:830)
==2689==    by 0x4992CBC: do_set (eval.c:2969)
==2689==    by 0x498C6F9: Rf_eval (eval.c:802)
==2689== 
==2689== 664 (104 direct, 560 indirect) bytes in 1 blocks are definitely lost in loss record 208 of 2,832
==2689==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2689==    by 0x14E85335: __gnu_cxx::new_allocator<std::__detail::_Hash_node_base*>::allocate(unsigned long, void const*) (new_allocator.h:115)
==2689==    by 0x14E84B46: std::allocator_traits<std::allocator<std::__detail::_Hash_node_base*> >::allocate(std::allocator<std::__detail::_Hash_node_base*>&, unsigned long) (alloc_traits.h:460)
==2689==    by 0x14E9A2A2: std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, true> > >::_M_allocate_buckets(unsigned long) (hashtable_policy.h:2085)
==2689==    by 0x14E9959A: std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_allocate_buckets(unsigned long) (hashtable.h:398)
==2689==    by 0x14E97FEE: std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_rehash_aux(unsigned long, std::integral_constant<bool, true>) (hashtable.h:2091)
==2689==    by 0x14E966E3: std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_rehash(unsigned long, unsigned long const&) (hashtable.h:2070)
==2689==    by 0x14E94E09: std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_insert_unique_node(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, unsigned long, std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, true>*, unsigned long) (hashtable.h:1728)
==2689==    by 0x14F1E1C5: std::pair<std::__detail::_Node_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, false, true>, bool> std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_emplace<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(std::integral_constant<bool, true>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (hashtable.h:1682)
==2689==    by 0x14F1D26E: std::pair<std::__detail::_Node_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, false, true>, bool> std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::emplace<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (hashtable.h:788)
==2689==    by 0x14F1C9DB: std::pair<std::__detail::_Node_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, false, true>, bool> std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::emplace<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (unordered_map.h:388)
==2689==    by 0x14F14845: LightGBM::Config::KV2Map(std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >*, char const*) (config.cpp:27)
==2689== 
==2689== 1,008 bytes in 1 blocks are definitely lost in loss record 236 of 2,832
==2689==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2689==    by 0x14E647C7: __gnu_cxx::new_allocator<double>::allocate(unsigned long, void const*) (new_allocator.h:115)
==2689==    by 0x14E625C2: std::allocator_traits<std::allocator<double> >::allocate(std::allocator<double>&, unsigned long) (alloc_traits.h:460)
==2689==    by 0x14E5F3AD: std::_Vector_base<double, std::allocator<double> >::_M_allocate(unsigned long) (stl_vector.h:346)
==2689==    by 0x14E5F1D6: std::_Vector_base<double, std::allocator<double> >::_M_create_storage(unsigned long) (stl_vector.h:361)
==2689==    by 0x14E5A37C: std::_Vector_base<double, std::allocator<double> >::_Vector_base(unsigned long, std::allocator<double> const&) (stl_vector.h:305)
==2689==    by 0x14E580BC: std::vector<double, std::allocator<double> >::vector(unsigned long, double const&, std::allocator<double> const&) (stl_vector.h:524)
==2689==    by 0x14E8C722: LightGBM::GBDT::FeatureImportance(int, int) const (gbdt_model_text.cpp:607)
==2689==    by 0x14E89FF7: LightGBM::GBDT::SaveModelToString[abi:cxx11](int, int, int) const (gbdt_model_text.cpp:374)
==2689==    by 0x151A8D05: LightGBM::Booster::SaveModelToString[abi:cxx11](int, int, int) const (c_api.cpp:732)
==2689==    by 0x1519BF8C: LGBM_BoosterSaveModelToString (c_api.cpp:2225)
==2689==    by 0x151C8B9E: LGBM_BoosterSaveModelToString_R (lightgbm_R.cpp:684)
==2689== 
==2689== 1,190 (64 direct, 1,126 indirect) bytes in 1 blocks are definitely lost in loss record 253 of 2,832
==2689==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2689==    by 0x14E65CAB: __gnu_cxx::new_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::allocate(unsigned long, void const*) (new_allocator.h:115)
==2689==    by 0x14E64277: std::allocator_traits<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::allocate(std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >&, unsigned long) (alloc_traits.h:460)
==2689==    by 0x14E61D25: std::_Vector_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_allocate(unsigned long) (stl_vector.h:346)
==2689==    by 0x14E5EB64: std::_Vector_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_create_storage(unsigned long) (stl_vector.h:361)
==2689==    by 0x14E59B32: std::_Vector_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_Vector_base(unsigned long, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&) (stl_vector.h:305)
==2689==    by 0x14E80CAA: std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::vector(unsigned long, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&) (stl_vector.h:511)
==2689==    by 0x14E89DE1: LightGBM::GBDT::SaveModelToString[abi:cxx11](int, int, int) const (gbdt_model_text.cpp:354)
==2689==    by 0x151A8D05: LightGBM::Booster::SaveModelToString[abi:cxx11](int, int, int) const (c_api.cpp:732)
==2689==    by 0x1519BF8C: LGBM_BoosterSaveModelToString (c_api.cpp:2225)
==2689==    by 0x151C8B9E: LGBM_BoosterSaveModelToString_R (lightgbm_R.cpp:684)
==2689==    by 0x494214E: R_doDotCall (dotcode.c:604)
==2689== 
==2689== 8,193 bytes in 1 blocks are definitely lost in loss record 1,302 of 2,832
==2689==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2689==    by 0x73703AF: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::reserve(unsigned long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==2689==    by 0x7365E4B: std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::overflow(int) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==2689==    by 0x736E679: std::basic_streambuf<char, std::char_traits<char> >::xsputn(char const*, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==2689==    by 0x73609B3: std::basic_ostream<char, std::char_traits<char> >& std::__ostream_insert<char, std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==2689==    by 0x14E89F6F: LightGBM::GBDT::SaveModelToString[abi:cxx11](int, int, int) const (gbdt_model_text.cpp:369)
==2689==    by 0x151A8D05: LightGBM::Booster::SaveModelToString[abi:cxx11](int, int, int) const (c_api.cpp:732)
==2689==    by 0x1519BF8C: LGBM_BoosterSaveModelToString (c_api.cpp:2225)
==2689==    by 0x151C8B9E: LGBM_BoosterSaveModelToString_R (lightgbm_R.cpp:684)
==2689==    by 0x494214E: R_doDotCall (dotcode.c:604)
==2689==    by 0x494D2C8: do_dotcall (dotcode.c:1281)
==2689==    by 0x498C952: Rf_eval (eval.c:830)
==2689== 
==2689== 1,048,576 bytes in 1 blocks are definitely lost in loss record 2,788 of 2,832
==2689==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2689==    by 0x14E61D81: __gnu_cxx::new_allocator<char>::allocate(unsigned long, void const*) (new_allocator.h:115)
==2689==    by 0x14E5E385: std::allocator_traits<std::allocator<char> >::allocate(std::allocator<char>&, unsigned long) (alloc_traits.h:460)
==2689==    by 0x14E61DB1: std::_Vector_base<char, std::allocator<char> >::_M_allocate(unsigned long) (stl_vector.h:346)
==2689==    by 0x14E5E44E: std::_Vector_base<char, std::allocator<char> >::_M_create_storage(unsigned long) (stl_vector.h:361)
==2689==    by 0x14E5911C: std::_Vector_base<char, std::allocator<char> >::_Vector_base(unsigned long, std::allocator<char> const&) (stl_vector.h:305)
==2689==    by 0x14E55BB0: std::vector<char, std::allocator<char> >::vector(unsigned long, std::allocator<char> const&) (stl_vector.h:511)
==2689==    by 0x151C8B4D: LGBM_BoosterSaveModelToString_R (lightgbm_R.cpp:683)
==2689==    by 0x494214E: R_doDotCall (dotcode.c:604)
==2689==    by 0x494D2C8: do_dotcall (dotcode.c:1281)
==2689==    by 0x498C952: Rf_eval (eval.c:830)
==2689==    by 0x4992CBC: do_set (eval.c:2969)
==2689== 
==2689== LEAK SUMMARY:
==2689==    definitely lost: 1,058,175 bytes in 15 blocks
==2689==    indirectly lost: 1,702 bytes in 11 blocks
==2689==      possibly lost: 336 bytes in 1 blocks
==2689==    still reachable: 299,629,382 bytes in 56,133 blocks
==2689==                       of which reachable via heuristic:
==2689==                         newarray           : 4,264 bytes in 1 blocks
==2689==         suppressed: 0 bytes in 0 blocks
==2689== Reachable blocks (those to which a pointer was found) are not shown.
==2689== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==2689== 
==2689== For lists of detected and suppressed errors, rerun with: -s
==2689== ERROR SUMMARY: 16 errors from 16 contexts (suppressed: 0 from 0)
writing valgrind output to valgrind-logs.log
valgrind found 1058175 bytes definitely lost

@jameslamb
Copy link
Collaborator Author

jameslamb commented Jun 9, 2021

/gha run r-valgrind

Workflow R valgrind tests has been triggered! 🚀
https://github.com/microsoft/LightGBM/actions/runs/920235595

Status: failure ❌.

@jameslamb
Copy link
Collaborator Author

jameslamb commented Jun 9, 2021

/gha run r-valgrind

Workflow R valgrind tests has been triggered! 🚀
https://github.com/microsoft/LightGBM/actions/runs/920572360

Status: success ✔️.

@jameslamb
Copy link
Collaborator Author

jameslamb commented Jun 9, 2021

I just ran another test in #4352 (comment). That looks at the valgrind output from removing the two uses of testthat::skip() but not changing anything about finalizers.

==2690== LEAK SUMMARY:
==2690==    definitely lost: 1,057,135 bytes in 13 blocks
==2690==    indirectly lost: 1,702 bytes in 11 blocks
==2690==      possibly lost: 1,376 bytes in 3 blocks
==2690==    still reachable: 300,641,977 bytes in 56,109 blocks
==2690==                       of which reachable via heuristic:
==2690==                         newarray           : 4,264 bytes in 1 blocks
==2690==         suppressed: 0 bytes in 0 blocks
full logs (click me)
==2690== HEAP SUMMARY:
==2690==     in use at exit: 301,702,190 bytes in 56,136 blocks
==2690==   total heap usage: 4,576,344 allocs, 4,520,208 frees, 6,284,154,585 bytes allocated
==2690== 
==2690== 16 bytes in 1 blocks are definitely lost in loss record 30 of 2,862
==2690==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2690==    by 0x14E81249: __gnu_cxx::new_allocator<unsigned long>::allocate(unsigned long, void const*) (new_allocator.h:115)
==2690==    by 0x14E7F1FD: std::allocator_traits<std::allocator<unsigned long> >::allocate(std::allocator<unsigned long>&, unsigned long) (alloc_traits.h:460)
==2690==    by 0x14E95E01: std::_Vector_base<unsigned long, std::allocator<unsigned long> >::_M_allocate(unsigned long) (stl_vector.h:346)
==2690==    by 0x14E954FE: std::_Vector_base<unsigned long, std::allocator<unsigned long> >::_M_create_storage(unsigned long) (stl_vector.h:361)
==2690==    by 0x14E938A4: std::_Vector_base<unsigned long, std::allocator<unsigned long> >::_Vector_base(unsigned long, std::allocator<unsigned long> const&) (stl_vector.h:305)
==2690==    by 0x14E924D2: std::vector<unsigned long, std::allocator<unsigned long> >::vector(unsigned long, std::allocator<unsigned long> const&) (stl_vector.h:511)
==2690==    by 0x14E89DC7: LightGBM::GBDT::SaveModelToString[abi:cxx11](int, int, int) const (gbdt_model_text.cpp:355)
==2690==    by 0x151A8CA5: LightGBM::Booster::SaveModelToString[abi:cxx11](int, int, int) const (c_api.cpp:732)
==2690==    by 0x1519BF2C: LGBM_BoosterSaveModelToString (c_api.cpp:2225)
==2690==    by 0x151C8A0D: LGBM_BoosterSaveModelToString_R (lightgbm_R.cpp:669)
==2690==    by 0x494214E: R_doDotCall (dotcode.c:604)
==2690== 
==2690== 19 bytes in 1 blocks are definitely lost in loss record 32 of 2,862
==2690==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2690==    by 0x737286C: void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==2690==    by 0x1518766A: LightGBM::Config::Config() (config.h:34)
==2690==    by 0x151968BA: LGBM_DatasetUpdateParamChecking (c_api.cpp:1496)
==2690==    by 0x151C5D6A: LGBM_DatasetUpdateParamChecking_R (lightgbm_R.cpp:300)
==2690==    by 0x4942116: R_doDotCall (dotcode.c:601)
==2690==    by 0x494D2C8: do_dotcall (dotcode.c:1281)
==2690==    by 0x498C952: Rf_eval (eval.c:830)
==2690==    by 0x499173F: do_begin (eval.c:2517)
==2690==    by 0x498C6F9: Rf_eval (eval.c:802)
==2690==    by 0x498BDEE: forcePromise (eval.c:555)
==2690==    by 0x4996F6B: FORCE_PROMISE (eval.c:5136)
==2690== 
==2690== 19 bytes in 1 blocks are definitely lost in loss record 33 of 2,862
==2690==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2690==    by 0x737286C: void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==2690==    by 0x1518766A: LightGBM::Config::Config() (config.h:34)
==2690==    by 0x151A430C: LightGBM::Booster::CheckDatasetResetConfig(LightGBM::Config const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) (c_api.cpp:187)
==2690==    by 0x15196905: LGBM_DatasetUpdateParamChecking (c_api.cpp:1499)
==2690==    by 0x151C5D6A: LGBM_DatasetUpdateParamChecking_R (lightgbm_R.cpp:300)
==2690==    by 0x4942116: R_doDotCall (dotcode.c:601)
==2690==    by 0x494D2C8: do_dotcall (dotcode.c:1281)
==2690==    by 0x498C952: Rf_eval (eval.c:830)
==2690==    by 0x499173F: do_begin (eval.c:2517)
==2690==    by 0x498C6F9: Rf_eval (eval.c:802)
==2690==    by 0x498BDEE: forcePromise (eval.c:555)
==2690== 
==2690== 20 bytes in 1 blocks are definitely lost in loss record 34 of 2,862
==2690==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2690==    by 0x737286C: void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==2690==    by 0x151879B4: LightGBM::Config::Config() (config.h:34)
==2690==    by 0x151968BA: LGBM_DatasetUpdateParamChecking (c_api.cpp:1496)
==2690==    by 0x151C5D6A: LGBM_DatasetUpdateParamChecking_R (lightgbm_R.cpp:300)
==2690==    by 0x4942116: R_doDotCall (dotcode.c:601)
==2690==    by 0x494D2C8: do_dotcall (dotcode.c:1281)
==2690==    by 0x498C952: Rf_eval (eval.c:830)
==2690==    by 0x499173F: do_begin (eval.c:2517)
==2690==    by 0x498C6F9: Rf_eval (eval.c:802)
==2690==    by 0x498BDEE: forcePromise (eval.c:555)
==2690==    by 0x4996F6B: FORCE_PROMISE (eval.c:5136)
==2690== 
==2690== 20 bytes in 1 blocks are definitely lost in loss record 35 of 2,862
==2690==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2690==    by 0x737286C: void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==2690==    by 0x151879B4: LightGBM::Config::Config() (config.h:34)
==2690==    by 0x151A430C: LightGBM::Booster::CheckDatasetResetConfig(LightGBM::Config const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) (c_api.cpp:187)
==2690==    by 0x15196905: LGBM_DatasetUpdateParamChecking (c_api.cpp:1499)
==2690==    by 0x151C5D6A: LGBM_DatasetUpdateParamChecking_R (lightgbm_R.cpp:300)
==2690==    by 0x4942116: R_doDotCall (dotcode.c:601)
==2690==    by 0x494D2C8: do_dotcall (dotcode.c:1281)
==2690==    by 0x498C952: Rf_eval (eval.c:830)
==2690==    by 0x499173F: do_begin (eval.c:2517)
==2690==    by 0x498C6F9: Rf_eval (eval.c:802)
==2690==    by 0x498BDEE: forcePromise (eval.c:555)
==2690== 
==2690== 28 bytes in 1 blocks are definitely lost in loss record 51 of 2,862
==2690==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2690==    by 0x737286C: void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==2690==    by 0x15187942: LightGBM::Config::Config() (config.h:34)
==2690==    by 0x151968BA: LGBM_DatasetUpdateParamChecking (c_api.cpp:1496)
==2690==    by 0x151C5D6A: LGBM_DatasetUpdateParamChecking_R (lightgbm_R.cpp:300)
==2690==    by 0x4942116: R_doDotCall (dotcode.c:601)
==2690==    by 0x494D2C8: do_dotcall (dotcode.c:1281)
==2690==    by 0x498C952: Rf_eval (eval.c:830)
==2690==    by 0x499173F: do_begin (eval.c:2517)
==2690==    by 0x498C6F9: Rf_eval (eval.c:802)
==2690==    by 0x498BDEE: forcePromise (eval.c:555)
==2690==    by 0x4996F6B: FORCE_PROMISE (eval.c:5136)
==2690== 
==2690== 28 bytes in 1 blocks are definitely lost in loss record 52 of 2,862
==2690==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2690==    by 0x737286C: void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==2690==    by 0x15187942: LightGBM::Config::Config() (config.h:34)
==2690==    by 0x151A430C: LightGBM::Booster::CheckDatasetResetConfig(LightGBM::Config const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) (c_api.cpp:187)
==2690==    by 0x15196905: LGBM_DatasetUpdateParamChecking (c_api.cpp:1499)
==2690==    by 0x151C5D6A: LGBM_DatasetUpdateParamChecking_R (lightgbm_R.cpp:300)
==2690==    by 0x4942116: R_doDotCall (dotcode.c:601)
==2690==    by 0x494D2C8: do_dotcall (dotcode.c:1281)
==2690==    by 0x498C952: Rf_eval (eval.c:830)
==2690==    by 0x499173F: do_begin (eval.c:2517)
==2690==    by 0x498C6F9: Rf_eval (eval.c:802)
==2690==    by 0x498BDEE: forcePromise (eval.c:555)
==2690== 
==2690== 32 bytes in 1 blocks are possibly lost in loss record 56 of 2,862
==2690==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2690==    by 0x14E65C4B: __gnu_cxx::new_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::allocate(unsigned long, void const*) (new_allocator.h:115)
==2690==    by 0x14E64217: std::allocator_traits<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::allocate(std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >&, unsigned long) (alloc_traits.h:460)
==2690==    by 0x14E61CC5: std::_Vector_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_allocate(unsigned long) (stl_vector.h:346)
==2690==    by 0x14F1D4A4: void std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_realloc_insert<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>(__gnu_cxx::__normal_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (vector.tcc:440)
==2690==    by 0x14F1CAA3: std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::push_back(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (stl_vector.h:1198)
==2690==    by 0x14F152D4: LightGBM::ParseMetrics(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*) (config.cpp:74)
==2690==    by 0x14F155D5: LightGBM::GetMetricType(std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*) (config.cpp:92)
==2690==    by 0x14F17150: LightGBM::Config::Set(std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) (config.cpp:211)
==2690==    by 0x151A4325: LightGBM::Booster::CheckDatasetResetConfig(LightGBM::Config const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) (c_api.cpp:188)
==2690==    by 0x15196905: LGBM_DatasetUpdateParamChecking (c_api.cpp:1499)
==2690==    by 0x151C5D6A: LGBM_DatasetUpdateParamChecking_R (lightgbm_R.cpp:300)
==2690== 
==2690== 32 (24 direct, 8 indirect) bytes in 1 blocks are definitely lost in loss record 57 of 2,862
==2690==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2690==    by 0x14E66348: __gnu_cxx::new_allocator<std::vector<double, std::allocator<double> > >::allocate(unsigned long, void const*) (new_allocator.h:115)
==2690==    by 0x14E64C62: std::allocator_traits<std::allocator<std::vector<double, std::allocator<double> > > >::allocate(std::allocator<std::vector<double, std::allocator<double> > >&, unsigned long) (alloc_traits.h:460)
==2690==    by 0x14E62B83: std::_Vector_base<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >::_M_allocate(unsigned long) (stl_vector.h:346)
==2690==    by 0x14E5FB1C: std::_Vector_base<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >::_M_create_storage(unsigned long) (stl_vector.h:361)
==2690==    by 0x14E5B1C6: std::_Vector_base<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >::_Vector_base(unsigned long, std::allocator<std::vector<double, std::allocator<double> > > const&) (stl_vector.h:305)
==2690==    by 0x14F1CB1A: std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >::vector(unsigned long, std::vector<double, std::allocator<double> > const&, std::allocator<std::vector<double, std::allocator<double> > > const&) (stl_vector.h:524)
==2690==    by 0x14F16A17: LightGBM::Config::GetAucMuWeights() (config.cpp:161)
==2690==    by 0x14F17222: LightGBM::Config::Set(std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) (config.cpp:221)
==2690==    by 0x151968D3: LGBM_DatasetUpdateParamChecking (c_api.cpp:1497)
==2690==    by 0x151C5D6A: LGBM_DatasetUpdateParamChecking_R (lightgbm_R.cpp:300)
==2690==    by 0x4942116: R_doDotCall (dotcode.c:601)
==2690== 
==2690== 32 (24 direct, 8 indirect) bytes in 1 blocks are definitely lost in loss record 58 of 2,862
==2690==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2690==    by 0x14E66348: __gnu_cxx::new_allocator<std::vector<double, std::allocator<double> > >::allocate(unsigned long, void const*) (new_allocator.h:115)
==2690==    by 0x14E64C62: std::allocator_traits<std::allocator<std::vector<double, std::allocator<double> > > >::allocate(std::allocator<std::vector<double, std::allocator<double> > >&, unsigned long) (alloc_traits.h:460)
==2690==    by 0x14E62B83: std::_Vector_base<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >::_M_allocate(unsigned long) (stl_vector.h:346)
==2690==    by 0x14E5FB1C: std::_Vector_base<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >::_M_create_storage(unsigned long) (stl_vector.h:361)
==2690==    by 0x14E5B1C6: std::_Vector_base<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >::_Vector_base(unsigned long, std::allocator<std::vector<double, std::allocator<double> > > const&) (stl_vector.h:305)
==2690==    by 0x14F1CB1A: std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >::vector(unsigned long, std::vector<double, std::allocator<double> > const&, std::allocator<std::vector<double, std::allocator<double> > > const&) (stl_vector.h:524)
==2690==    by 0x14F16A17: LightGBM::Config::GetAucMuWeights() (config.cpp:161)
==2690==    by 0x14F17222: LightGBM::Config::Set(std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) (config.cpp:221)
==2690==    by 0x151A4325: LightGBM::Booster::CheckDatasetResetConfig(LightGBM::Config const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) (c_api.cpp:188)
==2690==    by 0x15196905: LGBM_DatasetUpdateParamChecking (c_api.cpp:1499)
==2690==    by 0x151C5D6A: LGBM_DatasetUpdateParamChecking_R (lightgbm_R.cpp:300)
==2690== 
==2690== 336 bytes in 1 blocks are possibly lost in loss record 166 of 2,862
==2690==    at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2690==    by 0x40149CA: allocate_dtv (dl-tls.c:286)
==2690==    by 0x40149CA: _dl_allocate_tls (dl-tls.c:532)
==2690==    by 0x5721322: allocate_stack (allocatestack.c:622)
==2690==    by 0x5721322: pthread_create@@GLIBC_2.2.5 (pthread_create.c:660)
==2690==    by 0x56EFDDA: ??? (in /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0)
==2690==    by 0x56E78E0: GOMP_parallel (in /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0)
==2690==    by 0x1519536B: LGBM_DatasetCreateFromCSC (c_api.cpp:1303)
==2690==    by 0x151C4577: LGBM_DatasetCreateFromCSC_R (lightgbm_R.cpp:89)
==2690==    by 0x4942314: R_doDotCall (dotcode.c:624)
==2690==    by 0x494D2C8: do_dotcall (dotcode.c:1281)
==2690==    by 0x498C952: Rf_eval (eval.c:830)
==2690==    by 0x4992CBC: do_set (eval.c:2969)
==2690==    by 0x498C6F9: Rf_eval (eval.c:802)
==2690== 
==2690== 664 (104 direct, 560 indirect) bytes in 1 blocks are definitely lost in loss record 209 of 2,862
==2690==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2690==    by 0x14E852D5: __gnu_cxx::new_allocator<std::__detail::_Hash_node_base*>::allocate(unsigned long, void const*) (new_allocator.h:115)
==2690==    by 0x14E84AE6: std::allocator_traits<std::allocator<std::__detail::_Hash_node_base*> >::allocate(std::allocator<std::__detail::_Hash_node_base*>&, unsigned long) (alloc_traits.h:460)
==2690==    by 0x14E9A242: std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, true> > >::_M_allocate_buckets(unsigned long) (hashtable_policy.h:2085)
==2690==    by 0x14E9953A: std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_allocate_buckets(unsigned long) (hashtable.h:398)
==2690==    by 0x14E97F8E: std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_rehash_aux(unsigned long, std::integral_constant<bool, true>) (hashtable.h:2091)
==2690==    by 0x14E96683: std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_rehash(unsigned long, unsigned long const&) (hashtable.h:2070)
==2690==    by 0x14E94DA9: std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_insert_unique_node(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, unsigned long, std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, true>*, unsigned long) (hashtable.h:1728)
==2690==    by 0x14F1E165: std::pair<std::__detail::_Node_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, false, true>, bool> std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_emplace<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(std::integral_constant<bool, true>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (hashtable.h:1682)
==2690==    by 0x14F1D20E: std::pair<std::__detail::_Node_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, false, true>, bool> std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::emplace<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (hashtable.h:788)
==2690==    by 0x14F1C97B: std::pair<std::__detail::_Node_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, false, true>, bool> std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::emplace<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (unordered_map.h:388)
==2690==    by 0x14F147E5: LightGBM::Config::KV2Map(std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >*, char const*) (config.cpp:27)
==2690== 
==2690== 1,008 bytes in 1 blocks are possibly lost in loss record 237 of 2,862
==2690==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2690==    by 0x14E64767: __gnu_cxx::new_allocator<double>::allocate(unsigned long, void const*) (new_allocator.h:115)
==2690==    by 0x14E62562: std::allocator_traits<std::allocator<double> >::allocate(std::allocator<double>&, unsigned long) (alloc_traits.h:460)
==2690==    by 0x14E5F34D: std::_Vector_base<double, std::allocator<double> >::_M_allocate(unsigned long) (stl_vector.h:346)
==2690==    by 0x14E5F176: std::_Vector_base<double, std::allocator<double> >::_M_create_storage(unsigned long) (stl_vector.h:361)
==2690==    by 0x14E5A31C: std::_Vector_base<double, std::allocator<double> >::_Vector_base(unsigned long, std::allocator<double> const&) (stl_vector.h:305)
==2690==    by 0x14E5805C: std::vector<double, std::allocator<double> >::vector(unsigned long, double const&, std::allocator<double> const&) (stl_vector.h:524)
==2690==    by 0x14E8C6C2: LightGBM::GBDT::FeatureImportance(int, int) const (gbdt_model_text.cpp:607)
==2690==    by 0x14E89F97: LightGBM::GBDT::SaveModelToString[abi:cxx11](int, int, int) const (gbdt_model_text.cpp:374)
==2690==    by 0x151A8CA5: LightGBM::Booster::SaveModelToString[abi:cxx11](int, int, int) const (c_api.cpp:732)
==2690==    by 0x1519BF2C: LGBM_BoosterSaveModelToString (c_api.cpp:2225)
==2690==    by 0x151C8A0D: LGBM_BoosterSaveModelToString_R (lightgbm_R.cpp:669)
==2690== 
==2690== 1,190 (64 direct, 1,126 indirect) bytes in 1 blocks are definitely lost in loss record 256 of 2,862
==2690==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2690==    by 0x14E65C4B: __gnu_cxx::new_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::allocate(unsigned long, void const*) (new_allocator.h:115)
==2690==    by 0x14E64217: std::allocator_traits<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::allocate(std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >&, unsigned long) (alloc_traits.h:460)
==2690==    by 0x14E61CC5: std::_Vector_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_allocate(unsigned long) (stl_vector.h:346)
==2690==    by 0x14E5EB04: std::_Vector_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_create_storage(unsigned long) (stl_vector.h:361)
==2690==    by 0x14E59AD2: std::_Vector_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_Vector_base(unsigned long, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&) (stl_vector.h:305)
==2690==    by 0x14E80C4A: std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::vector(unsigned long, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&) (stl_vector.h:511)
==2690==    by 0x14E89D81: LightGBM::GBDT::SaveModelToString[abi:cxx11](int, int, int) const (gbdt_model_text.cpp:354)
==2690==    by 0x151A8CA5: LightGBM::Booster::SaveModelToString[abi:cxx11](int, int, int) const (c_api.cpp:732)
==2690==    by 0x1519BF2C: LGBM_BoosterSaveModelToString (c_api.cpp:2225)
==2690==    by 0x151C8A0D: LGBM_BoosterSaveModelToString_R (lightgbm_R.cpp:669)
==2690==    by 0x494214E: R_doDotCall (dotcode.c:604)
==2690== 
==2690== 8,193 bytes in 1 blocks are definitely lost in loss record 1,352 of 2,862
==2690==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2690==    by 0x73703AF: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::reserve(unsigned long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==2690==    by 0x7365E4B: std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::overflow(int) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==2690==    by 0x736E679: std::basic_streambuf<char, std::char_traits<char> >::xsputn(char const*, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==2690==    by 0x73609B3: std::basic_ostream<char, std::char_traits<char> >& std::__ostream_insert<char, std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==2690==    by 0x14E89F0F: LightGBM::GBDT::SaveModelToString[abi:cxx11](int, int, int) const (gbdt_model_text.cpp:369)
==2690==    by 0x151A8CA5: LightGBM::Booster::SaveModelToString[abi:cxx11](int, int, int) const (c_api.cpp:732)
==2690==    by 0x1519BF2C: LGBM_BoosterSaveModelToString (c_api.cpp:2225)
==2690==    by 0x151C8A0D: LGBM_BoosterSaveModelToString_R (lightgbm_R.cpp:669)
==2690==    by 0x494214E: R_doDotCall (dotcode.c:604)
==2690==    by 0x494D2C8: do_dotcall (dotcode.c:1281)
==2690==    by 0x498C952: Rf_eval (eval.c:830)
==2690== 
==2690== 1,048,576 bytes in 1 blocks are definitely lost in loss record 2,818 of 2,862
==2690==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2690==    by 0x14E61D21: __gnu_cxx::new_allocator<char>::allocate(unsigned long, void const*) (new_allocator.h:115)
==2690==    by 0x14E5E325: std::allocator_traits<std::allocator<char> >::allocate(std::allocator<char>&, unsigned long) (alloc_traits.h:460)
==2690==    by 0x14E61D51: std::_Vector_base<char, std::allocator<char> >::_M_allocate(unsigned long) (stl_vector.h:346)
==2690==    by 0x14E5E3EE: std::_Vector_base<char, std::allocator<char> >::_M_create_storage(unsigned long) (stl_vector.h:361)
==2690==    by 0x14E590BC: std::_Vector_base<char, std::allocator<char> >::_Vector_base(unsigned long, std::allocator<char> const&) (stl_vector.h:305)
==2690==    by 0x14E55B50: std::vector<char, std::allocator<char> >::vector(unsigned long, std::allocator<char> const&) (stl_vector.h:511)
==2690==    by 0x151C89BC: LGBM_BoosterSaveModelToString_R (lightgbm_R.cpp:668)
==2690==    by 0x494214E: R_doDotCall (dotcode.c:604)
==2690==    by 0x494D2C8: do_dotcall (dotcode.c:1281)
==2690==    by 0x498C952: Rf_eval (eval.c:830)
==2690==    by 0x4992CBC: do_set (eval.c:2969)
==2690== 
==2690== LEAK SUMMARY:
==2690==    definitely lost: 1,057,135 bytes in 13 blocks
==2690==    indirectly lost: 1,702 bytes in 11 blocks
==2690==      possibly lost: 1,376 bytes in 3 blocks
==2690==    still reachable: 300,641,977 bytes in 56,109 blocks
==2690==                       of which reachable via heuristic:
==2690==                         newarray           : 4,264 bytes in 1 blocks
==2690==         suppressed: 0 bytes in 0 blocks
==2690== Reachable blocks (those to which a pointer was found) are not shown.
==2690== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==2690== 
==2690== For lists of detected and suppressed errors, rerun with: -s
==2690== ERROR SUMMARY: 16 errors from 16 contexts (suppressed: 0 from 0)
writing valgrind output to valgrind-logs.log
valgrind found 1057135 bytes definitely lost

commit: 2714777
build: https://github.com/microsoft/LightGBM/runs/2784480910?check_suite_focus=true

As of the change in this PR, valgrind now thinks 1040 bytes are "definitely lost" which it previously said were "possibly lost".

Namely, the two findings below say "possibly lost" on master (with testthat::skip() removed) and "definitely lost" on this branch (with testthat::skip() removed).

==2690== 1,008 bytes in 1 blocks are possibly lost in loss record 237 of 2,862
==2690==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
...
==2690==    by 0x14E8C6C2: LightGBM::GBDT::FeatureImportance(int, int) const (gbdt_model_text.cpp:607)
==2690==    by 0x14E89F97: LightGBM::GBDT::SaveModelToString[abi:cxx11](int, int, int) const (gbdt_model_text.cpp:374)
==2690==    by 0x151A8CA5: LightGBM::Booster::SaveModelToString[abi:cxx11](int, int, int) const (c_api.cpp:732)
==2690==    by 0x1519BF2C: LGBM_BoosterSaveModelToString (c_api.cpp:2225)
==2690==    by 0x151C8A0D: LGBM_BoosterSaveModelToString_R (lightgbm_R.cpp:669)
==2690== 
...
...
==2690== 32 bytes in 1 blocks are possibly lost in loss record 56 of 2,862
==2690==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
....
std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) (c_api.cpp:188)
==2690==    by 0x15196905: LGBM_DatasetUpdateParamChecking (c_api.cpp:1499)
==2690==    by 0x151C5D6A: LGBM_DatasetUpdateParamChecking_R (lightgbm_R.cpp:300)

I know this is kind of complicated and confusing. My conclusion from this is:

  1. This PR should not remove testthat::skip() calls in tests, because it seems that the memory leaks from those tests suffer from other memory leaks described in [R-package] R memory leaks #4282, not just leaks from the way handles to Boosters / Datasets are finalized.
  2. This PR should be merged, as it passes all CI and does not introduce new valgrind issues ([R-package] move more finalizer logic into C++ side to address memory leaks #4353 (comment)), and for the reasons given in the PR description.

@jameslamb jameslamb changed the title WIP: [R-package] move more finalizer logic into C++ side to address memory leaks [R-package] move more finalizer logic into C++ side to address memory leaks Jun 9, 2021
@jameslamb jameslamb marked this pull request as ready for review June 9, 2021 16:56
@jameslamb jameslamb requested a review from Laurae2 as a code owner June 9, 2021 16:56
@jameslamb jameslamb requested review from StrikerRUS and shiyu1994 June 9, 2021 16:56
Copy link
Collaborator

@StrikerRUS StrikerRUS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Glad to see R-package is getting more "standard" and rich in terms of working with underlying cpp code 👍 .

R-package/tests/testthat/test_lgb.Booster.R Outdated Show resolved Hide resolved
@jameslamb jameslamb merged commit 53ffba7 into master Jun 13, 2021
@jameslamb jameslamb deleted the r/finalize branch June 13, 2021 04:13
@github-actions
Copy link

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants