Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Msgpack may clean up data twice #404

Closed
mgovers opened this issue Oct 16, 2023 · 3 comments · Fixed by #420
Closed

[BUG] Msgpack may clean up data twice #404

mgovers opened this issue Oct 16, 2023 · 3 comments · Fixed by #420
Labels
bug Something isn't working

Comments

@mgovers
Copy link
Member

mgovers commented Oct 16, 2023

Describe the bug

According to clang-tidy (full run on all targets, e.g. https://github.com/PowerGridModel/power-grid-model/actions/runs/6529848243/job/17728134248 ), msgpack cleans up data twice

  • once using std::unique_ptr::~unique_ptr
  • once using std::free

This is very dangerous, because it may clean up memory that another (part of the) program may already be using

Data Dump

/home/linuxbrew/.linuxbrew/include/msgpack/v1/detail/cpp11_zone.hpp:197:9: error: Attempt to free released memory [clang-analyzer-cplusplus.NewDelete,-warnings-as-errors]
        ::free(p);
        ^
/home/runner/work/power-grid-model/power-grid-model/tests/cpp_validation_tests/test_validation.cpp:589:9: note: Assuming the condition is true
        SUBCASE(param.case_name.c_str()) {
        ^
/home/linuxbrew/.linuxbrew/include/doctest/doctest.h:2945:23: note: expanded from macro 'SUBCASE'
#define SUBCASE(name) DOCTEST_SUBCASE(name)
                      ^~~~~~~~~~~~~~~~~~~~~
/home/linuxbrew/.linuxbrew/include/doctest/doctest.h:2279:41: note: expanded from macro 'DOCTEST_SUBCASE'
    if(const doctest::detail::Subcase & DOCTEST_ANONYMOUS(DOCTEST_ANON_SUBCASE_) DOCTEST_UNUSED =  \
                                        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/linuxbrew/.linuxbrew/include/doctest/doctest.h:422:30: note: expanded from macro 'DOCTEST_ANONYMOUS'
#define DOCTEST_ANONYMOUS(x) DOCTEST_CAT(x, __COUNTER__)
                             ^~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/linuxbrew/.linuxbrew/include/doctest/doctest.h:420:29: note: expanded from macro 'DOCTEST_CAT'
#define DOCTEST_CAT(s1, s2) DOCTEST_CAT_IMPL(s1, s2)
                            ^~~~~~~~~~~~~~~~~~~~~~~~
/home/linuxbrew/.linuxbrew/include/doctest/doctest.h:419:34: note: expanded from macro 'DOCTEST_CAT_IMPL'
#define DOCTEST_CAT_IMPL(s1, s2) s1##s2
                                 ^~~~~~
note: expanded from here
/home/runner/work/power-grid-model/power-grid-model/tests/cpp_validation_tests/test_validation.cpp:589:9: note: Taking true branch
        SUBCASE(param.case_name.c_str()) {
        ^
/home/linuxbrew/.linuxbrew/include/doctest/doctest.h:2945:23: note: expanded from macro 'SUBCASE'
#define SUBCASE(name) DOCTEST_SUBCASE(name)
                      ^
/home/linuxbrew/.linuxbrew/include/doctest/doctest.h:2279:5: note: expanded from macro 'DOCTEST_SUBCASE'
    if(const doctest::detail::Subcase & DOCTEST_ANONYMOUS(DOCTEST_ANON_SUBCASE_) DOCTEST_UNUSED =  \
    ^
/home/runner/work/power-grid-model/power-grid-model/tests/cpp_validation_tests/test_validation.cpp:591:17: note: Calling 'validate_batch_case'
                validate_batch_case(param);
                ^~~~~~~~~~~~~~~~~~~~~~~~~~
/home/runner/work/power-grid-model/power-grid-model/tests/cpp_validation_tests/test_validation.cpp:532:5: note: Calling 'execute_test<(lambda at /home/runner/work/power-grid-model/power-grid-model/tests/cpp_validation_tests/test_validation.cpp:532:25)>'
    execute_test(param, [&]() {
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/runner/work/power-grid-model/power-grid-model/tests/cpp_validation_tests/test_validation.cpp:507:9: note: Assuming the condition is false
    if (should_skip_test(param)) {
        ^~~~~~~~~~~~~~~~~~~~~~~
/home/runner/work/power-grid-model/power-grid-model/tests/cpp_validation_tests/test_validation.cpp:507:5: note: Taking false branch
    if (should_skip_test(param)) {
    ^
/home/runner/work/power-grid-model/power-grid-model/tests/cpp_validation_tests/test_validation.cpp:511:9: note: Calling 'operator()'
        func();
        ^~~~~~
/home/runner/work/power-grid-model/power-grid-model/tests/cpp_validation_tests/test_validation.cpp:533:38: note: Calling 'create_validation_case'
        auto const validation_case = create_validation_case(param);
                                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/runner/work/power-grid-model/power-grid-model/tests/cpp_validation_tests/test_validation.cpp:451:29: note: Calling 'load_dataset'
    validation_case.input = load_dataset(param.case_dir / "input.json");
                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/runner/work/power-grid-model/power-grid-model/tests/cpp_validation_tests/test_validation.cpp:114:12: note: Calling defaulted destructor for 'Deserializer'
    return dataset;
           ^
/home/runner/work/power-grid-model/power-grid-model/power_grid_model_c/power_grid_model/include/power_grid_model/auxiliary/serialization/deserializer.hpp:109:29: note: Calling implicit destructor for 'object_handle'
    ~Deserializer() = default;
                            ^
/home/runner/work/power-grid-model/power-grid-model/power_grid_model_c/power_grid_model/include/power_grid_model/auxiliary/serialization/deserializer.hpp:109:29: note: Calling '~unique_ptr'
/usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/unique_ptr.h:395:6: note: Assuming the condition is true
        if (__ptr != nullptr)
            ^~~~~~~~~~~~~~~~
/usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/unique_ptr.h:395:2: note: Taking true branch
        if (__ptr != nullptr)
        ^
/usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/unique_ptr.h:396:4: note: Calling 'default_delete::operator()'
          get_deleter()(std::move(__ptr));
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/unique_ptr.h:95:2: note: Memory is released
        delete __ptr;
        ^~~~~~~~~~~~
/usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/unique_ptr.h:95:2: note: Calling 'zone::operator delete'
        delete __ptr;
        ^~~~~~~~~~~~
/home/linuxbrew/.linuxbrew/include/msgpack/v1/detail/cpp11_zone.hpp:197:9: note: Attempt to free released memory
        ::free(p);
        ^~~~~~~~~

To Reproduce

Run clang-tidy on all targets, e.g. in CI (manual workflow dispatch). See e.g. https://github.com/PowerGridModel/power-grid-model/actions/runs/6529848243/job/17728134248

Expected behavior

clang-tidy run on all targets passes as expected.

Additional context

It looks like the root cause of the issue lies in a cleanup within msgpack when moving objects. Moving of msgpack::object_handle already cleans up the data, meaning that calling the destructor of the msgpack::object_handle will attempt to cleanup the data again, which is a bug.

We may want to move to the visitor pattern implementation of msgpack to work around this. That way, there is no unpacker object with extended lifetime (no member variables) and no moving needs to happen.

We already seek to do this because of the better memory performance

@mgovers mgovers added the bug Something isn't working label Oct 16, 2023
@mgovers mgovers changed the title [BUG] *descriptive_name* [BUG] Msgpack may clean up data twice Oct 16, 2023
@mgovers
Copy link
Member Author

mgovers commented Oct 16, 2023

It appears that the root cause is the fact that msgpack::zone is overloading the delete operator.

Minimal repro case (using Debug configuration without additional optimizations):

// SPDX-FileCopyrightText: 2022 Contributors to the Power Grid Model project <[email protected]>
//
// SPDX-License-Identifier: MPL-2.0

#include <msgpack.hpp>

namespace {
struct Foo {
    std::unique_ptr<msgpack::zone> zone;
};
} // namespace

int main() {
    Foo const foo{};
    return 0;
}
[build] "<...>/cmake.exe" -E __run_co_compile --tidy=clang-tidy.exe;--extra-arg=/EHsc;--extra-arg-before=--driver-mode=cl --source=<source_dir>/source.cpp -- <...>/clang-cl.exe  /nologo -TP  -imsvc<std_lib> /DWIN32 /D_WINDOWS /EHsc /Zi /Ob0 /Od /RTC1 -std:c++20 -MDd /showIncludes /Fo<build_dir>/test_tmp.cpp.obj /Fd<build_dir>\ -c -- <source_dir>/source.cpp
[build] <msgpack_dir>/include/msgpack/v1/detail/cpp11_zone.hpp:197:9: error: Attempt to free released memory [clang-analyzer-cplusplus.NewDelete,-warnings-as-errors]
[build]         ::free(p);
[build]         ^
[build] <source_dir>/source.cpp:15:12: note: Calling implicit destructor for 'Foo'
[build]     return 0;
[build]            ^
[build] <source_dir>/source.cpp:15:12: note: Calling '~unique_ptr'
[build] <std_lib>/memory:3289:13: note: Assuming field '_Myval2' is non-null
[build]         if (_Mypair._Myval2) {
[build]             ^~~~~~~~~~~~~~~
[build] <std_lib>/memory:3289:9: note: Taking true branch
[build]         if (_Mypair._Myval2) {
[build]         ^
[build] <std_lib>/memory:3290:13: note: Calling 'default_delete::operator()'
[build]             _Mypair._Get_first()(_Mypair._Myval2);
[build]             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[build] <std_lib>/memory:3180:9: note: Memory is released
[build]         delete _Ptr;
[build]         ^~~~~~~~~~~
[build] <std_lib>/memory:3180:9: note: Calling 'zone::operator delete'
[build]         delete _Ptr;
[build]         ^~~~~~~~~~~
[build] <msgpack_dir>/include/msgpack/v1/detail/cpp11_zone.hpp:197:9: note: Attempt to free released memory
[build]         ::free(p);
[build]         ^~~~~~~~~

@mgovers
Copy link
Member Author

mgovers commented Oct 17, 2023

Bug reported to msgpack-cxx in msgpack/msgpack-c#1098

@mgovers
Copy link
Member Author

mgovers commented Nov 8, 2023

Full clang-tidy run passed. See also #420 (review)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

1 participant