Isn't the correct term for what happens when you hit the high watermark ... #18
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…rk is ejections and not evictions?
ns-codereview
referenced
this pull request
in couchbase/ep-engine
Oct 8, 2015
- As part of the cancelling the processer task in ExecutorPool's stopTaskGroup, the consumer's destructor will be invoked as the reference has been released. - The consumer's destructor will try to cancel the processer task, which tries to acquire ExecutorPool::tMutex, but the same lock was already acquired by this very thread as part of stopTaskGroup. #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007fea5d5f5657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007fea5d5f5480 in __GI___pthread_mutex_lock (mutex=0x7d5000018120) at ../nptl/pthread_mutex_lock.c:79 #3 0x00007fea5e0c1cd5 in pthread_mutex_lock () #4 0x00007fea5da10ea9 in cb_mutex_enter (mutex=0x7d5000018120) at /home/abhinav/couchbase/platform/src/cb_pthreads.c:115 #5 0x00007fea588bce2a in Mutex::acquire (this=0x7d5000018118) at /home/abhinav/couchbase/ep-engine/src/mutex.cc:31 #6 0x00007fea58892277 in lock (this=<optimized out>) at /home/abhinav/couchbase/ep-engine/src/locks.h:70 #7 LockHolder (m=..., tryLock=false, this=<optimized out>) at /home/abhinav/couchbase/ep-engine/src/locks.h:47 #8 ExecutorPool::_cancel (this=0x7d5000018000, taskId=41, eraseTask=<optimized out>) at /home/abhinav/couchbase/ep-engine/src/executorpool.cc:295 #9 0x00007fea58892a38 in ExecutorPool::cancel (this=0x7d5000018120, taskId=128, eraseTask=false) at /home/abhinav/couchbase/ep-engine/src/executorpool.cc:328 #10 0x00007fea587fe9c7 in cancelTask (this=0x7d5000016800) at /home/abhinav/couchbase/ep-engine/src/dcp/consumer.cc:114 #11 DcpConsumer::~DcpConsumer (this=0x7d5000016800) at /home/abhinav/couchbase/ep-engine/src/dcp/consumer.cc:104 #12 0x00007fea587ff126 in DcpConsumer::~DcpConsumer (this=0x7d5000016800) at /home/abhinav/couchbase/ep-engine/src/dcp/consumer.cc:103 #13 0x00007fea58804e14 in ~SingleThreadedRCPtr (this=<optimized out>) at /home/abhinav/couchbase/ep-engine/src/atomic.h:271 #14 ~Processer (this=0x7d140000e600) at /home/abhinav/couchbase/ep-engine/src/dcp/consumer.cc:35 #15 Processer::~Processer (this=0x7d140000e600) at /home/abhinav/couchbase/ep-engine/src/dcp/consumer.cc:35 #16 0x00007fea58897596 in ~SingleThreadedRCPtr (this=0x7d140000e600) at /home/abhinav/couchbase/ep-engine/src/atomic.h:271 #17 ExecutorPool::_stopTaskGroup (this=<optimized out>, taskGID=<optimized out>, taskType=<optimized out>, force=<optimized out>) at /home/abhinav/couchbase/ep-engine/src/executorpool.cc:576 #18 0x00007fea58897d7f in ExecutorPool::stopTaskGroup (this=0x7d5000018120, taskGID=128, taskType=WRITER_TASK_IDX, force=false) at /home/abhinav/couchbase/ep-engine/src/executorpool.cc:585 #19 0x00007fea588283a7 in EventuallyPersistentStore::~EventuallyPersistentStore (this=0x7d540000fc80) at /home/abhinav/couchbase/ep-engine/src/ep.cc:468 #20 0x00007fea58878e4b in EventuallyPersistentEngine::~EventuallyPersistentEngine (this=0x7d780000f000) at /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:6336 #21 0x00007fea58858aad in EvpDestroy (handle=0x7d780000f000, force=<optimized out>) at /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:142 #22 0x00007fea5e1070e8 in mock_destroy (handle=<optimized out>, force=false) at /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:98 #23 0x00007fea5e1065ef in destroy_bucket (handle=0x7d480000b880, handle_v1=0x7d480000b880, force=false) at /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:995 #24 execute_test (engine=<optimized out>, default_cfg=<optimized out>, test=...) at /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1112 #25 main (argc=<optimized out>, argv=<optimized out>) at /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1439 Change-Id: Iaebb3d0ed8d0cfbcef13dafd689e7c08f94069d8 Reviewed-on: http://review.couchbase.org/55884 Reviewed-by: Chiyoung Seo <[email protected]> Tested-by: buildbot <[email protected]>
ns-codereview
referenced
this pull request
in couchbase/ep-engine
Apr 7, 2016
16:50:07 WARNING: ThreadSanitizer: data race (pid=36686) 16:50:07 Write of size 8 at 0x7d0c00003ad8 by main thread (mutexes: write M130807): 16:50:07 #0 operator delete(void*) <null> (ep-engine_dcp_test+0x00000047357b) 16:50:07 #1 std::_Rb_tree<void const*, std::pair<void const* const, DcpConsumer*>, std::_Select1st<std::pair<void const* const, DcpConsumer*> >, std::less<void const*>, std::allocator<std::pair<void const* const, DcpConsumer*> > >::_M_erase_aux(std::_Rb_tree_const_iterator<std::pair<void const* const, DcpConsumer*> >) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/ext/new_allocator.h:110 (ep-engine_dcp_test+0x00000052ad65) 16:50:07 #2 FlowControl::~FlowControl() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/dcp/flow-control.cc:44 (ep-engine_dcp_test+0x000000528ea1) 16:50:07 #3 DcpConsumer::~DcpConsumer() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/dcp/consumer.cc:133 (ep-engine_dcp_test+0x00000051b223) 16:50:07 #4 DcpConsumer::~DcpConsumer() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/dcp/consumer.cc:130 (ep-engine_dcp_test+0x00000051b815) 16:50:07 #5 Processer::~Processer() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/atomic.h:272 (ep-engine_dcp_test+0x00000051a51c) 16:50:07 #6 ~SingleThreadedRCPtr /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/atomic.h:272 (ep-engine_dcp_test+0x0000005ba662) 16:50:07 #7 ExecutorPool::stopTaskGroup(unsigned long, task_type_t, bool) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorpool.cc:594 (ep-engine_dcp_test+0x0000005badee) 16:50:07 #8 EventuallyPersistentStore::~EventuallyPersistentStore() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep.cc:565 (ep-engine_dcp_test+0x00000054f302) 16:50:07 #9 EventuallyPersistentEngine::~EventuallyPersistentEngine() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep_engine.cc:6349 (ep-engine_dcp_test+0x00000059bf7a) 16:50:07 #10 EvpDestroy(engine_interface*, bool) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep_engine.cc:148 (ep-engine_dcp_test+0x00000057e2b7) 16:50:07 #11 DCPTest::TearDown() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/tests/module_tests/dcp_test.cc:126 (ep-engine_dcp_test+0x0000004e5737) 16:50:07 #12 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/googletest/googletest/src/gtest.cc:2402 (ep-engine_dcp_test+0x0000006d7792) 16:50:07 #13 testing::Test::Run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/googletest/googletest/src/gtest.cc:2482 (ep-engine_dcp_test+0x0000006a4e31) 16:50:07 #14 testing::TestInfo::Run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/googletest/googletest/src/gtest.cc:2656 (ep-engine_dcp_test+0x0000006a6e0b) 16:50:07 #15 testing::TestCase::Run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/googletest/googletest/src/gtest.cc:2774 (ep-engine_dcp_test+0x0000006a78ba) 16:50:07 #16 testing::internal::UnitTestImpl::RunAllTests() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/googletest/googletest/src/gtest.cc:4649 (ep-engine_dcp_test+0x0000006b84f3) 16:50:07 #17 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/googletest/googletest/src/gtest.cc:2402 (ep-engine_dcp_test+0x0000006d8902) 16:50:07 #18 testing::UnitTest::Run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/googletest/googletest/src/gtest.cc:4257 (ep-engine_dcp_test+0x0000006b7ad0) 16:50:07 #19 main /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/googletest/googletest/include/gtest/gtest.h:2237 (ep-engine_dcp_test+0x0000004e2cc0) 16:50:07 16:50:07 Previous read of size 8 at 0x7d0c00003ad8 by thread T5 (mutexes: write M130950): 16:50:07 #0 DcpFlowControlManagerAggressive::handleDisconnect(DcpConsumer*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/dcp/flow-control-manager.cc:211 (ep-engine_dcp_test+0x00000052aea8) 16:50:07 #1 FlowControl::~FlowControl() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/dcp/flow-control.cc:44 (ep-engine_dcp_test+0x000000528ea1) 16:50:07 #2 DcpConsumer::~DcpConsumer() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/dcp/consumer.cc:133 (ep-engine_dcp_test+0x00000051b223) 16:50:07 #3 DcpConsumer::~DcpConsumer() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/dcp/consumer.cc:130 (ep-engine_dcp_test+0x00000051b815) 16:50:07 #4 Processer::~Processer() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/atomic.h:272 (ep-engine_dcp_test+0x00000051a51c) 16:50:07 #5 ExecutorThread::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/atomic.h:325 (ep-engine_dcp_test+0x0000005bfa5b) 16:50:07 #6 launch_executor_thread(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorthread.cc:33 (ep-engine_dcp_test+0x0000005bf885) 16:50:07 #7 platform_thread_wrap(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/platform/src/cb_pthreads.cc:54 (libplatform.so.0.1.0+0x00000000568b) Change-Id: I84860030d78d0bc2e5010255e8ba30bec6109719 Reviewed-on: http://review.couchbase.org/62527 Tested-by: buildbot <[email protected]> Reviewed-by: Manu Dhundi <[email protected]>
cb-robot
pushed a commit
that referenced
this pull request
Jul 7, 2016
Regression test for MB-20054 - the following abort is encountered when
a DCPBackfill task is still running when a bucket is deleted:
Assertion failed: (engine), function verifyEngine, file
ep-engine/src/objectregistry.cc, line 58.
This issue occurs because the DCPBackfill object (and associated
objects ActiveStream and importantly ActiveStreams' readyQ of Items)
is not deleted earlier in the shutdown sequence (via EvpDestroy), as
we use ref-counted pointers for it and there is a still an outstanding
reference by the AuxIO Thread which is running the task. Hence the
DCPBackfill object is only deleted when we finally unregister the
deleted bucket from the shared ExecutorPool - see the following
backtrace:
#1 0x00007f513b75a085 in abort () from /lib64/libc.so.6
#2 0x00007f51337034e2 in ObjectRegistry::onDeleteItem (pItem=<value optimized out>) at ep-engine/src/objectregistry.cc:157
#3 0x00007f5133652094 in ~Item (this=<value optimized out>) at ep-engine/src/item.h:352
#4 SingleThreadedRCPtr<Item>::~SingleThreadedRCPtr (this=<value optimized out>) at ep-engine/src/atomic.h:430
#5 0x00007f51336c7f47 in ~MutationResponse (this=0x3cd87880) at ep-engine/src/dcp-response.h:275
#6 MutationResponse::~MutationResponse (this=0x3cd87880) at ep-engine/src/dcp-response.h:275
#7 0x00007f51336d86aa in clear_UNLOCKED (this=0x7a3f5fa0) at ep-engine/src/dcp-stream.cc:201
#8 ~ActiveStream (this=0x7a3f5fa0) at ep-engine/src/dcp-stream.h:178
#9 ActiveStream::~ActiveStream (this=0x7a3f5fa0) at ep-engine/src/dcp-stream.h:179
#10 0x00007f51336cc808 in RCPtr<Stream>::~RCPtr (this=0xb1823780) at ep-engine/src/atomic.h:348
#11 0x00007f51336d77c7 in ~DCPBackfill (this=0xb1823740) at ep-engine/src/dcp-stream.cc:114
#12 DCPBackfill::~DCPBackfill (this=0xb1823740) at ep-engine/src/dcp-stream.cc:114
#13 0x00007f513368d95f in ~SingleThreadedRCPtr (this=0x5b55a20, e=0x59c4000, taskType=NO_TASK_TYPE) at ep-engine/src/atomic.h:430
#14 ExecutorPool::_stopTaskGroup (this=0x5b55a20, e=0x59c4000, taskType=NO_TASK_TYPE) at ep-engine/src/executorpool.cc:532
#15 0x00007f513368dad3 in ExecutorPool::_unregisterBucket (this=0x5b55a20, engine=0x59c4000) at ep-engine/src/executorpool.cc:551
#16 0x00007f513368e143 in ExecutorPool::unregisterBucket (this=0x5b55a20, engine=0x59c4000) at ep-engine/src/executorpool.cc:602
#17 0x00007f5133655f82 in EventuallyPersistentStore::~EventuallyPersistentStore (this=0x59e6000)
at ep-engine/src/ep.cc:365
#18 0x00007f5133672a25 in EventuallyPersistentEngine::~EventuallyPersistentEngine (this=0x59c4000)
at ep-engine/src/ep_engine.cc:5791
#19 0x00007f5133672c95 in EvpDestroy (handle=0x59c4000, force=<value optimized out>) at ep-engine/src/ep_engine.cc:143
To actually reproduce the issue is somewhat involved - we need to
orchestrate the world such that we delete the engine while a
DCPBackfill task is still running. We spin up a separate thread which
will run the DCPBackfill task concurrently with destroy - specifically
DCPBackfill must start running (and add items to the readyQ) before
destroy(), it must then continue running (stop after) _stopTaskGroup
is invoked. To achieve this we use a couple of condition variables to
synchronise between the two threads - the timeline needs to look like:
auxIO thread: [------- DCPBackfill ----------]
main thread: [--destroy()--] [ExecutorPool::_stopTaskGroup]
--------------------------------------------------------> time
Change-Id: Ic64c419cb8e4e0af2378efba9711b121aacee15b
Reviewed-on: http://review.couchbase.org/65520
Well-Formed: buildbot <[email protected]>
Tested-by: buildbot <[email protected]>
Reviewed-by: Jim Walker <[email protected]>
cb-robot
pushed a commit
that referenced
this pull request
Oct 20, 2016
When developing additional stats test for ep-engine_perfsuite, the following data race was uncovered. The patch ensures that accesses to vals are protected with a mutex. Also get_stat and get_histo_stat can only be called one at a time as they use the three global variables (requested_stat_name, actual_stat_value and histogram_stat_int_value). Therefore a secondary mutex is required to enforce this. It has been confirmed that with this patch the ThreadSanitizer issue no longer occurs. 30: WARNING: ThreadSanitizer: data race (pid=42065) 30: Read of size 8 at 0x7d1800060a38 by thread T18 (mutexes: write M1704): 30: #0 std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_S_right(std::_Rb_tree_node_base*) <null> (engine_testapp+0x00000049c503) 30: #1 std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_M_lower_bound(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*, std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (engine_testapp+0x0000004a30db) 30: #2 std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::lower_bound(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (engine_testapp+0x00000049c5f6) 30: #3 std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::lower_bound(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (engine_testapp+0x000000499047) 30: #4 std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::operator[](std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (engine_testapp+0x00000048e3ba) 30: #5 add_stats /home/owend/master/ep-engine/tests/ep_test_apis.cc:236 (ep_perfsuite.so+0x00000008f5b9) 30: #6 STATWRITER_NAMESPACE::add_casted_stat(char const*, char const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/owend/master/ep-engine/src/statwriter.h:39 (ep.so+0x0000000e5673) 30: #7 StatCheckpointVisitor::addCheckpointStat(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), EventuallyPersistentStore*, RCPtr<VBucket>&) <null> (ep.so+0x0000001c120c) 30: #8 StatCheckpointVisitor::visitBucket(RCPtr<VBucket>&) <null> (ep.so+0x0000001c10e5) 30: #9 EventuallyPersistentStore::visit(VBucketVisitor&) /home/owend/master/ep-engine/src/ep.cc:3796 (ep.so+0x000000174559) 30: #10 StatCheckpointTask::run() <null> (ep.so+0x0000001c155b) 30: #11 ExecutorThread::run() /home/owend/master/ep-engine/src/executorthread.cc:113 (ep.so+0x0000001ea96d) 30: #12 launch_executor_thread /home/owend/master/ep-engine/src/executorthread.cc:31 (ep.so+0x0000001ea0d0) 30: #13 CouchbaseThread::run() /home/owend/master/platform/src/cb_pthreads.cc:58 (libplatform_so.so.0.1.0+0x00000000c9c7) 30: #14 platform_thread_wrap /home/owend/master/platform/src/cb_pthreads.cc:71 (libplatform_so.so.0.1.0+0x00000000affe) 30: #15 <null> <null> (libtsan.so.0+0x0000000230d9) 30: 30: Previous write of size 8 at 0x7d1800060a38 by main thread (mutexes: write M1575): 30: #0 operator new(unsigned long) <null> (libtsan.so.0+0x000000025a33) 30: #1 __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::allocate(unsigned long, void const*) <null> (engine_testapp+0x0000004a6ca7) 30: #2 std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >::allocate(std::allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, unsigned long) <null> (engine_testapp+0x0000004a5168) 30: #3 std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_M_get_node() <null> (engine_testapp+0x0000004a316f) 30: #4 std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >* std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_M_create_node<std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>, std::tuple<> >(std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&, std::tuple<>&&) <null> (engine_testapp+0x00000049c6b4) 30: #5 std::_Rb_tree_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_M_emplace_hint_unique<std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>, std::tuple<> >(std::_Rb_tree_const_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&, std::tuple<>&&) <null> (engine_testapp+0x00000049928e) 30: #6 std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::operator[](std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (engine_testapp+0x00000048e488) 30: #7 add_stats /home/owend/master/ep-engine/tests/ep_test_apis.cc:236 (ep_perfsuite.so+0x00000008f5b9) 30: #8 STATWRITER_NAMESPACE::add_casted_stat(char const*, char const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/owend/master/ep-engine/src/statwriter.h:39 (ep.so+0x0000000e5673) 30: #9 void STATWRITER_NAMESPACE::add_casted_stat<unsigned long>(char const*, unsigned long const&, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/owend/master/ep-engine/src/statwriter.h:48 (ep.so+0x0000000e96ae) 30: #10 EventuallyPersistentEngine::addSeqnoVbStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), RCPtr<VBucket> const&) /home/owend/master/ep-engine/src/ep_engine.cc:4445 (ep.so+0x0000001ad56d) 30: #11 EventuallyPersistentEngine::doSeqnoStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), char const*, int) /home/owend/master/ep-engine/src/ep_engine.cc:4490 (ep.so+0x0000001ad95f) 30: #12 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/owend/master/ep-engine/src/ep_engine.cc:4595 (ep.so+0x0000001ae4de) 30: #13 EvpGetStats /home/owend/master/ep-engine/src/ep_engine.cc:232 (ep.so+0x00000019851e) 30: #14 ENGINE_ERROR_CODE std::_Bind<ENGINE_ERROR_CODE (*(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)))(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*))>::__call<ENGINE_ERROR_CODE, , 0ul, 1ul, 2ul, 3ul, 4ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul, 2ul, 3ul, 4ul>) <null> (engine_testapp+0x00000049f211) 30: #15 ENGINE_ERROR_CODE std::_Bind<ENGINE_ERROR_CODE (*(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)))(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*))>::operator()<, ENGINE_ERROR_CODE>() <null> (engine_testapp+0x00000049aa02) 30: #16 std::_Function_handler<ENGINE_ERROR_CODE (), std::_Bind<ENGINE_ERROR_CODE (*(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)))(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*))> >::_M_invoke(std::_Any_data const&) <null> (engine_testapp+0x00000049145b) 30: #17 std::function<ENGINE_ERROR_CODE ()>::operator()() const <null> (engine_testapp+0x000000484e28) 30: #18 call_engine_and_handle_EWOULDBLOCK /home/owend/master/memcached/programs/engine_testapp/engine_testapp.cc:127 (engine_testapp+0x00000047cf00) 30: #19 mock_get_stats /home/owend/master/memcached/programs/engine_testapp/engine_testapp.cc:220 (engine_testapp+0x00000047d896) 30: #20 perf_stat_latency_core /home/owend/master/ep-engine/tests/ep_perfsuite.cc:1238 (ep_perfsuite.so+0x000000063d3d) 30: #21 perf_stat_latency /home/owend/master/ep-engine/tests/ep_perfsuite.cc:1306 (ep_perfsuite.so+0x00000006427a) 30: #22 perf_slow_stat_latency_100vb /home/owend/master/ep-engine/tests/ep_perfsuite.cc:1365 (ep_perfsuite.so+0x00000006498c) 30: #23 execute_test /home/owend/master/memcached/programs/engine_testapp/engine_testapp.cc:962 (engine_testapp+0x000000481776) 30: #24 main /home/owend/master/memcached/programs/engine_testapp/engine_testapp.cc:1359 (engine_testapp+0x000000482ab1) Change-Id: I7bdc847c0913244409fa044e312d53b484dc2dab Reviewed-on: http://review.couchbase.org/68813 Reviewed-by: Dave Rigby <[email protected]> Tested-by: buildbot <[email protected]>
cb-robot
pushed a commit
that referenced
this pull request
Nov 1, 2016
…kfills
The existing code was safe, however ThreadSanitizer doesn't know about
our own Spinlocks. Given this shouldn't be a hot path, switch to
normal std:mutex.
ThreadSanitizer report:
WARNING: ThreadSanitizer: data race (pid=23569)
Read of size 2 at 0x7d840000eca2 by thread T8 (mutexes: write M294, read M27095, write M66205, write M102676, write M95235):
#0 DcpConnMap::canAddBackfillToActiveQ() ep-engine/src/connmap.cc:1308 (ep.so+0x000000045ac5)
#1 BackfillManager::schedule() ep-engine/src/dcp/backfill-manager.cc:142 (ep.so+0x00000005b0eb)
#2 DcpProducer::scheduleBackfillManager() ep-engine/src/dcp/producer.cc:702 (ep.so+0x000000078fe3)
#3 ActiveStream::scheduleBackfill_UNLOCKED(bool) ep-engine/src/dcp/stream.cc:1016 (ep.so+0x00000008f280)
#4 ActiveStream::transitionState(stream_state_t) ep-engine/src/dcp/stream.cc:1145 (ep.so+0x000000090589)
#5 ActiveStream::setActive() ep-engine/src/dcp/stream.h:204 (ep.so+0x00000009958e)
#6 DcpProducer::streamRequest() ep-engine/src/dcp/producer.cc:327 (ep.so+0x00000007f85d)
#7 EvpDcpStreamReq ep-engine/src/ep_engine.cc:1573 (ep.so+0x0000000cea78)
#8 dcp_stream_req_executor memcached/daemon/mcbp_executors.cc:2272 (memcached+0x00000045925c)
#9 process_bin_packet memcached/daemon/mcbp_executors.cc:4650 (memcached+0x00000046481d)
#10 mcbp_complete_nread(McbpConnection*) memcached/daemon/mcbp_executors.cc:4759 (memcached+0x00000046481d)
#11 conn_nread(McbpConnection*) memcached/daemon/statemachine_mcbp.cc:314 (memcached+0x000000472678)
#12 McbpStateMachine::execute(McbpConnection&) memcached/daemon/statemachine_mcbp.h:43 (memcached+0x000000447054)
#13 McbpConnection::runStateMachinery() memcached/daemon/connection_mcbp.cc:1003 (memcached+0x000000447054)
#14 McbpConnection::runEventLoop(short) memcached/daemon/connection_mcbp.cc:1274 (memcached+0x0000004470dd)
#15 run_event_loop memcached/daemon/connections.cc:147 (memcached+0x00000044b9e9)
#16 event_handler(int, short, void*) memcached/daemon/memcached.cc:851 (memcached+0x00000041466c)
#17 event_persist_closure src/libevent/event.c:1319 (libevent_core-2.0.so.5+0x00000000b6b7)
#18 event_process_active_single_queue src/libevent/event.c:1363 (libevent_core-2.0.so.5+0x00000000b6b7)
#19 event_process_active src/libevent/event.c:1438 (libevent_core-2.0.so.5+0x00000000b6b7)
#20 event_base_loop src/libevent/event.c:1639 (libevent_core-2.0.so.5+0x00000000b6b7)
#21 CouchbaseThread::run() platform/src/cb_pthreads.cc:54 (libplatform.so.0.1.0+0x0000000057a5)
#22 platform_thread_wrap platform/src/cb_pthreads.cc:66 (libplatform.so.0.1.0+0x0000000057a5)
Previous write of size 2 at 0x7d840000eca2 by thread T55:
#0 DcpConnMap::decrNumActiveSnoozingBackfills() ep-engine/src/connmap.cc:1319 (ep.so+0x000000045b7b)
#1 BackfillManager::backfill() ep-engine/src/dcp/backfill-manager.cc:273 (ep.so+0x00000005a783)
#2 BackfillManagerTask::run() ep-engine/src/dcp/backfill-manager.cc:62 (ep.so+0x00000005ac1c)
#3 ExecutorThread::run() ep-engine/src/executorthread.cc:115 (ep.so+0x000000108d96)
#4 launch_executor_thread ep-engine/src/executorthread.cc:33 (ep.so+0x000000109675)
#5 CouchbaseThread::run() platform/src/cb_pthreads.cc:54 (libplatform.so.0.1.0+0x0000000057a5)
#6 platform_thread_wrap platform/src/cb_pthreads.cc:66 (libplatform.so.0.1.0+0x0000000057a5)
Change-Id: I88df57b268c1e615b7c5d2b7caf5f038a53692ba
Reviewed-on: http://review.couchbase.org/69235
Reviewed-by: Trond Norbye <[email protected]>
Tested-by: buildbot <[email protected]>
cb-robot
pushed a commit
that referenced
this pull request
Nov 2, 2016
As identified by ThreadSantizer:
WARNING: ThreadSanitizer: data race (pid=23560)
Write of size 8 at 0x7ff76a2e7e50 by thread T52 (mutexes: write M64345, write M85935):
#0 VBucket::adjustCheckpointFlushTimeout(unsigned long) ep-engine/src/vbucket.cc:393 (ep.so+0x00000016c806)
#1 VBucket::notifyOnPersistence(EventuallyPersistentEngine&, unsigned long, bool) ep-engine/src/vbucket.cc:332 (ep.so+0x00000016c806)
#2 EventuallyPersistentStore::flushVBucket(unsigned short) ep-engine/src/ep.cc:3475 (ep.so+0x0000000be736)
#3 Flusher::flushVB() ep-engine/src/flusher.cc:284 (ep.so+0x0000001100d1)
#4 Flusher::step(GlobalTask*) ep-engine/src/flusher.cc:183 (ep.so+0x0000001110b7)
#5 FlusherTask::run() ep-engine/src/tasks.cc:138 (ep.so+0x000000156642)
#6 ExecutorThread::run() ep-engine/src/executorthread.cc:115 (ep.so+0x000000108d96)
#7 launch_executor_thread ep-engine/src/executorthread.cc:33 (ep.so+0x000000109675)
#8 CouchbaseThread::run() platform/src/cb_pthreads.cc:54 (libplatform.so.0.1.0+0x0000000057a5)
#9 platform_thread_wrap platform/src/cb_pthreads.cc:66 (libplatform.so.0.1.0+0x0000000057a5)
Previous read of size 8 at 0x7ff76a2e7e50 by thread T9 (mutexes: write M295):
#0 VBucket::getCheckpointFlushTimeout() ep-engine/src/vbucket.cc:406 (ep.so+0x0000001683e6)
#1 EventuallyPersistentEngine::doEngineStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) ep-engine/src/ep_engine.cc:3628 (ep.so+0x0000000e9678)
#2 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) ep-engine/src/ep_engine.cc:4680 (ep.so+0x0000000ea930)
#3 EvpGetStats ep-engine/src/ep_engine.cc:235 (ep.so+0x0000000eb9fe)
#4 stat_executor memcached/daemon/mcbp_executors.cc:3367 (memcached+0x00000046c94b)
#5 process_bin_packet memcached/daemon/mcbp_executors.cc:4650 (memcached+0x00000046481d)
#6 mcbp_complete_nread(McbpConnection*) memcached/daemon/mcbp_executors.cc:4759 (memcached+0x00000046481d)
#7 conn_nread(McbpConnection*) memcached/daemon/statemachine_mcbp.cc:314 (memcached+0x000000472678)
#8 McbpStateMachine::execute(McbpConnection&) memcached/daemon/statemachine_mcbp.h:43 (memcached+0x000000447054)
#9 McbpConnection::runStateMachinery() memcached/daemon/connection_mcbp.cc:1003 (memcached+0x000000447054)
#10 McbpConnection::runEventLoop(short) memcached/daemon/connection_mcbp.cc:1274 (memcached+0x0000004470dd)
#11 run_event_loop memcached/daemon/connections.cc:147 (memcached+0x00000044b9e9)
#12 event_handler(int, short, void*) memcached/daemon/memcached.cc:851 (memcached+0x00000041466c)
#13 event_persist_closure /home/couchbase/serverjenkins/workspace/cbdeps-platform-build/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1319 (libevent_core-2.0.so.5+0x00000000b6b7)
#14 event_process_active_single_queue /home/couchbase/serverjenkins/workspace/cbdeps-platform-build/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1363 (libevent_core-2.0.so.5+0x00000000b6b7)
#15 event_process_active /home/couchbase/serverjenkins/workspace/cbdeps-platform-build/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1438 (libevent_core-2.0.so.5+0x00000000b6b7)
#16 event_base_loop /home/couchbase/serverjenkins/workspace/cbdeps-platform-build/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1639 (libevent_core-2.0.so.5+0x00000000b6b7)
#17 CouchbaseThread::run() platform/src/cb_pthreads.cc:54 (libplatform.so.0.1.0+0x0000000057a5)
#18 platform_thread_wrap platform/src/cb_pthreads.cc:66 (libplatform.so.0.1.0+0x0000000057a5)
Location is global 'VBucket::chkFlushTimeout' of size 8 at 0x7ff76a2e7e50 (ep.so+0x000000431e50)
Change-Id: I2d6f928b8a5552cf08a91c6134ad5134810966c2
Reviewed-on: http://review.couchbase.org/69234
Reviewed-by: Trond Norbye <[email protected]>
Tested-by: buildbot <[email protected]>
cb-robot
pushed a commit
that referenced
this pull request
Dec 5, 2016
The CTest test names don't match the executable names for the
ep_testsuite_* series of tests. This can be confusing, and complicates
test management, particulary as we are about to add a third variant
(ephemeral) to the tests.
Rationalise the test names, so they are simply <suite_name>.<variant>
Examples:
$ ctest -N -R ep_testsuite
Test project /Users/dave/repos/couchbase/server/source/build/ep-engine
Test #18: ep_testsuite.value_eviction
Test #19: ep_testsuite.full_eviction
Test #20: ep_testsuite_basic.value_eviction
Test #21: ep_testsuite_basic.full_eviction
Test #22: ep_testsuite_dcp.value_eviction
Test #23: ep_testsuite_dcp.full_eviction
Test #24: ep_testsuite_tap.value_eviction
Test #25: ep_testsuite_tap.full_eviction
Test #26: ep_testsuite_checkpoint.value_eviction
Test #27: ep_testsuite_checkpoint.full_eviction
Test #28: ep_testsuite_xdcr.value_eviction
Test #29: ep_testsuite_xdcr.full_eviction
Change-Id: I227b669497ad2c8114d631f331cf77180774ae30
Reviewed-on: http://review.couchbase.org/70611
Tested-by: buildbot <[email protected]>
Reviewed-by: Jim Walker <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
...is ejections and not evictions?