default bucket ignored #5
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ns-codereview
pushed a commit
that referenced
this pull request
Jul 7, 2016
Regression test for MB-20054 - the following abort is encountered when
a DCPBackfill task is still running when a bucket is deleted:
Assertion failed: (engine), function verifyEngine, file
ep-engine/src/objectregistry.cc, line 58.
This issue occurs because the DCPBackfill object (and associated
objects ActiveStream and importantly ActiveStreams' readyQ of Items)
is not deleted earlier in the shutdown sequence (via EvpDestroy), as
we use ref-counted pointers for it and there is a still an outstanding
reference by the AuxIO Thread which is running the task. Hence the
DCPBackfill object is only deleted when we finally unregister the
deleted bucket from the shared ExecutorPool - see the following
backtrace:
#1 0x00007f513b75a085 in abort () from /lib64/libc.so.6
#2 0x00007f51337034e2 in ObjectRegistry::onDeleteItem (pItem=<value optimized out>) at ep-engine/src/objectregistry.cc:157
#3 0x00007f5133652094 in ~Item (this=<value optimized out>) at ep-engine/src/item.h:352
#4 SingleThreadedRCPtr<Item>::~SingleThreadedRCPtr (this=<value optimized out>) at ep-engine/src/atomic.h:430
#5 0x00007f51336c7f47 in ~MutationResponse (this=0x3cd87880) at ep-engine/src/dcp-response.h:275
#6 MutationResponse::~MutationResponse (this=0x3cd87880) at ep-engine/src/dcp-response.h:275
#7 0x00007f51336d86aa in clear_UNLOCKED (this=0x7a3f5fa0) at ep-engine/src/dcp-stream.cc:201
#8 ~ActiveStream (this=0x7a3f5fa0) at ep-engine/src/dcp-stream.h:178
#9 ActiveStream::~ActiveStream (this=0x7a3f5fa0) at ep-engine/src/dcp-stream.h:179
#10 0x00007f51336cc808 in RCPtr<Stream>::~RCPtr (this=0xb1823780) at ep-engine/src/atomic.h:348
#11 0x00007f51336d77c7 in ~DCPBackfill (this=0xb1823740) at ep-engine/src/dcp-stream.cc:114
#12 DCPBackfill::~DCPBackfill (this=0xb1823740) at ep-engine/src/dcp-stream.cc:114
#13 0x00007f513368d95f in ~SingleThreadedRCPtr (this=0x5b55a20, e=0x59c4000, taskType=NO_TASK_TYPE) at ep-engine/src/atomic.h:430
#14 ExecutorPool::_stopTaskGroup (this=0x5b55a20, e=0x59c4000, taskType=NO_TASK_TYPE) at ep-engine/src/executorpool.cc:532
#15 0x00007f513368dad3 in ExecutorPool::_unregisterBucket (this=0x5b55a20, engine=0x59c4000) at ep-engine/src/executorpool.cc:551
#16 0x00007f513368e143 in ExecutorPool::unregisterBucket (this=0x5b55a20, engine=0x59c4000) at ep-engine/src/executorpool.cc:602
#17 0x00007f5133655f82 in EventuallyPersistentStore::~EventuallyPersistentStore (this=0x59e6000)
at ep-engine/src/ep.cc:365
#18 0x00007f5133672a25 in EventuallyPersistentEngine::~EventuallyPersistentEngine (this=0x59c4000)
at ep-engine/src/ep_engine.cc:5791
#19 0x00007f5133672c95 in EvpDestroy (handle=0x59c4000, force=<value optimized out>) at ep-engine/src/ep_engine.cc:143
To actually reproduce the issue is somewhat involved - we need to
orchestrate the world such that we delete the engine while a
DCPBackfill task is still running. We spin up a separate thread which
will run the DCPBackfill task concurrently with destroy - specifically
DCPBackfill must start running (and add items to the readyQ) before
destroy(), it must then continue running (stop after) _stopTaskGroup
is invoked. To achieve this we use a couple of condition variables to
synchronise between the two threads - the timeline needs to look like:
auxIO thread: [------- DCPBackfill ----------]
main thread: [--destroy()--] [ExecutorPool::_stopTaskGroup]
--------------------------------------------------------> time
Change-Id: Ic64c419cb8e4e0af2378efba9711b121aacee15b
Reviewed-on: http://review.couchbase.org/65520
Well-Formed: buildbot <[email protected]>
Tested-by: buildbot <[email protected]>
Reviewed-by: Jim Walker <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Jul 7, 2016
A mistake in 495e00a means that no variable is created for the ReaderLockHolder, the compiler either optimises away the lock constructor/destructor or the lock scope is wrong. Either way we need to create a variable. Includes some lock ordering changes as per ThreadSantitiser warnings. (Reviewed-on: http://review.couchbase.org/56978) This will address the following lock inversion: 11:56:19 WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=51509) 11:56:19 Cycle in lock order graph: M21441 (0x7d780000f450) => M21477 (0x7d640005edf0) => M21441 11:56:19 11:56:19 Mutex M21477 acquired here while holding mutex M21441 in main thread: 11:56:19 #0 pthread_rwlock_rdlock <null> (engine_testapp+0x000000462260) 11:56:19 #1 cb_rw_reader_enter <null> (libplatform.so.0.1.0+0x000000004800) 11:56:19 #2 RWLock::readerLock() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/rwlock.h:38 (ep.so+0x000000132360) 11:56:19 #3 ReaderLockHolder::ReaderLockHolder(RWLock&) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/locks.h:167 (ep.so+0x0000000f8087) 11:56:19 #4 EventuallyPersistentStore::getAndUpdateTtl(std::string const&, unsigned short, void const*, long) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep.cc:1970 (ep.so+0x0000000e6b45) 11:56:19 #5 EventuallyPersistentEngine::touch(void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep_engine.cc:4619 (ep.so+0x000000155fe8) 11:56:19 #6 processUnknownCommand(EventuallyPersistentEngine*, void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep_engine.cc:1126 (ep.so+0x000000163a9b) 11:56:19 #7 EvpUnknownCommand(engine_interface*, void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep_engine.cc:1312 (ep.so+0x000000137365) 11:56:19 #8 mock_unknown_command /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/memcached/programs/engine_testapp/engine_testapp.c (engine_testapp+0x0000004c8f1a) 11:56:19 #9 gat(engine_interface*, engine_interface_v1*, char const*, unsigned short, unsigned int, bool) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/tests/ep_test_apis.cc:348 (ep_testsuite.so+0x0000000e2d6b) 11:56:19 #10 test_expired_item_with_item_eviction(engine_interface*, engine_interface_v1*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/tests/ep_testsuite.cc:11401 (ep_testsuite.so+0x0000000acbd4) 11:56:19 #11 execute_test /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/memcached/programs/engine_testapp/engine_testapp.c (engine_testapp+0x0000004c4e9f) 11:56:19 #12 main crtstuff.c (engine_testapp+0x0000004c2e01) 11:56:19 11:56:19 Mutex M21441 acquired here while holding mutex M21477 in thread T8: 11:56:19 #0 pthread_mutex_lock <null> (engine_testapp+0x00000047e9e0) 11:56:19 #1 cb_mutex_enter <null> (libplatform.so.0.1.0+0x0000000039c0) 11:56:19 #2 Mutex::acquire() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/mutex.cc:31 (ep.so+0x0000001e241e) 11:56:19 #3 LockHolder::lock() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/locks.h:71 (ep.so+0x000000080bc3) 11:56:19 #4 LockHolder::LockHolder(Mutex&, bool) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/locks.h:48 (ep.so+0x000000080832) 11:56:19 #5 HashTable::getLockedBucket(int, int*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/stored-value.h:1266 (ep.so+0x00000008280a) 11:56:19 #6 HashTable::getLockedBucket(std::string const&, int*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/stored-value.h:1295 (ep.so+0x00000007c61b) 11:56:19 #7 EventuallyPersistentStore::deleteExpiredItem(unsigned short, std::string&, long, unsigned long) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep.cc:481 (ep.so+0x0000000d4d80) 11:56:19 #8 ExpiredItemsCallback::callback(compaction_ctx&) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep.cc:1258 (ep.so+0x000000123ecb) 11:56:19 #9 CouchKVStore::compactVBucket(unsigned short, compaction_ctx*, Callback<compaction_ctx>&, Callback<KVStatsCtx>&) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/couch-kvstore/couch-kvstore.cc:862 (ep.so+0x0000003159f7) 11:56:19 #10 EventuallyPersistentStore::compactVBucket(unsigned short, compaction_ctx*, void const*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep.cc:1326 (ep.so+0x0000000df2ec) 11:56:19 #11 CompactVBucketTask::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/tasks.cc:76 (ep.so+0x000000251ed1) 11:56:19 #12 ExecutorThread::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/executorthread.cc:109 (ep.so+0x0000001e38f1) 11:56:19 #13 launch_executor_thread(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/executorthread.cc:34 (ep.so+0x0000001e2f1a) 11:56:19 #14 platform_thread_wrap /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/platform/src/cb_pthreads.c (libplatform.so.0.1.0+0x00000000377c) Change-Id: I5d5ca33fdd3c17df2be9d2b2d6acc8c254f1cb2d Reviewed-on: http://review.couchbase.org/63418 Well-Formed: buildbot <[email protected]> Tested-by: buildbot <[email protected]> Reviewed-by: Will Gardner <[email protected]> Reviewed-by: Chiyoung Seo <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Jul 7, 2016
[Not a backport, this code was altered/removed in master] 11:56:19 Read of size 8 at 0x7d6400050df8 by main thread (mutexes: write M45364, write M44294): 11:56:19 #0 std::_Hashtable<std::string, std::pair<std::string const, std::list<VBucketBGFetchItem*, std::allocator<VBucketBGFetchItem*> > >, std::allocator<std::pair<std::string const, std::list<VBucketBGFetchItem*, std::allocator<VBucketBGFetchItem*> > > >, std::__detail::_Select1st, std::equal_to<std::string>, std::hash<std::string>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::size() const /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/hashtable.h:500 (ep.so+0x00000008d98e) 11:56:19 #1 std::unordered_map<std::string, std::list<VBucketBGFetchItem*, std::allocator<VBucketBGFetchItem*> >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::list<VBucketBGFetchItem*, std::allocator<VBucketBGFetchItem*> > > > >::size() const /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/unordered_map.h:264 (ep.so+0x000000085be0) 11:56:19 #2 VBucket::numPendingBGFetchItems() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/vbucket.h:333 (ep.so+0x000000101789) 11:56:19 #3 EventuallyPersistentStore::bgFetch(std::string const&, unsigned short, unsigned long, void const*, bool) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep.cc:1651 (ep.so+0x0000000d71f7) 11:56:19 #4 EventuallyPersistentStore::getInternal(std::string const&, unsigned short, void const*, bool, bool, vbucket_state_t, bool) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep.cc:1708 (ep.so+0x0000000e3e21) 11:56:19 #5 EventuallyPersistentStore::get(std::string const&, unsigned short, void const*, bool, bool, bool) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep.h:242 (ep.so+0x00000019eee9) 11:56:19 #6 EventuallyPersistentEngine::get(void const*, void**, void const*, int, unsigned short, bool) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep_engine.h:259 (ep.so+0x00000016fe75) 11:56:19 #7 EvpGet(engine_interface*, void const*, void**, void const*, int, unsigned short) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep_engine.cc:202 (ep.so+0x000000136911) 11:56:19 #8 mock_get /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/memcached/programs/engine_testapp/engine_testapp.c (engine_testapp+0x0000004c6e84) 11:56:19 #9 get_item_info(engine_interface*, engine_interface_v1*, item_info*, char const*, unsigned short) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/tests/ep_test_apis.cc:356 (ep_testsuite.so+0x0000000e2f2d) 11:56:19 #10 check_key_value(engine_interface*, engine_interface_v1*, char const*, char const*, unsigned long, unsigned short) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/tests/ep_testsuite.cc:155 (ep_testsuite.so+0x0000000b2590) 11:56:19 #11 test_duplicate_items_disk(engine_interface*, engine_interface_v1*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/tests/ep_testsuite.cc:7839 (ep_testsuite.so+0x00000005a4b6) 11:56:19 #12 execute_test /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/memcached/programs/engine_testapp/engine_testapp.c (engine_testapp+0x0000004c4e9f) 11:56:19 #13 main crtstuff.c (engine_testapp+0x0000004c2e01) 11:56:19 11:56:19 Previous write of size 8 at 0x7d6400050df8 by thread T1 (mutexes: write M44318): 11:56:19 #0 std::_Hashtable<std::string, std::pair<std::string const, std::list<VBucketBGFetchItem*, std::allocator<VBucketBGFetchItem*> > >, std::allocator<std::pair<std::string const, std::list<VBucketBGFetchItem*, std::allocator<VBucketBGFetchItem*> > > >, std::__detail::_Select1st, std::equal_to<std::string>, std::hash<std::string>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::clear() /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/hashtable.h:1943 (ep.so+0x000000087ba8) 11:56:19 #1 std::unordered_map<std::string, std::list<VBucketBGFetchItem*, std::allocator<VBucketBGFetchItem*> >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::list<VBucketBGFetchItem*, std::allocator<VBucketBGFetchItem*> > > > >::clear() /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/unordered_map.h:528 (ep.so+0x000000086f40) 11:56:19 #2 VBucket::getBGFetchItems(std::unordered_map<std::string, std::list<VBucketBGFetchItem*, std::allocator<VBucketBGFetchItem*> >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::list<VBucketBGFetchItem*, std::allocator<VBucketBGFetchItem*> > > > >&) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/vbucket.cc:294 (ep.so+0x0000002b4026) 11:56:19 #3 BgFetcher::run(GlobalTask*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/bgfetcher.cc:155 (ep.so+0x00000008449f) 11:56:19 #4 BgFetcherTask::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/tasks.cc:89 (ep.so+0x00000025205e) 11:56:19 #5 ExecutorThread::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/executorthread.cc:109 (ep.so+0x0000001e38f1) 11:56:19 #6 launch_executor_thread(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/executorthread.cc:34 (ep.so+0x0000001e2f1a) 11:56:19 #7 platform_thread_wrap /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/platform/src/cb_pthreads.c (libplatform.so.0.1.0+0x00000000377c) Change-Id: I66e3c2af1f58448a68fbfedf3dfa030a657ed9a7 Reviewed-on: http://review.couchbase.org/63369 Well-Formed: buildbot <[email protected]> Tested-by: buildbot <[email protected]> Reviewed-by: Will Gardner <[email protected]> Reviewed-by: Chiyoung Seo <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Jul 7, 2016
WARNING: ThreadSanitizer: data race (pid=14344)
Read of size 8 at 0x7d780000fa58 by thread T6:
#0 void STATWRITER_NAMESPACE::add_casted_stat<long>(char const*, long const&, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/abhinav/couchbase/ep-engine/src/statwriter.h:45 (ep.so+0x000000037ff5)
#1 EventuallyPersistentEngine::doEngineStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:3557 (ep.so+0x0000000be990)
#2 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:4554 (ep.so+0x0000000c5c8c)
#3 EventuallyPersistentStore::snapshotStats() /home/abhinav/couchbase/ep-engine/src/ep.cc:1671 (ep.so+0x00000008f1fe)
#4 StatSnap::run() /home/abhinav/couchbase/ep-engine/src/tasks.cc:97 (ep.so+0x00000013cea6)
#5 ExecutorThread::run() /home/abhinav/couchbase/ep-engine/src/executorthread.cc:112 (ep.so+0x0000000f94e3)
#6 launch_executor_thread(void*) /home/abhinav/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f9065)
#7 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d31)
Previous write of size 8 at 0x7d780000fa58 by main thread:
#0 EventuallyPersistentEngine::initialize(char const*) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:2167 (ep.so+0x0000000b728a)
#1 EvpInitialize(engine_interface*, char const*) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:133 (ep.so+0x0000000b4aa8)
#2 init_engine_instance /home/abhinav/couchbase/memcached/utilities/engine_loader.c:157 (libmcd_util.so.1.0.0+0x0000000058bb)
#3 create_bucket(bool, char const*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:980 (engine_testapp+0x0000000b9e12)
#4 execute_test(test, char const*, char const*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1083 (engine_testapp+0x0000000b93db)
#5 __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287 (libc.so.6+0x000000021ec4)
(Reviewed-on: http://review.couchbase.org/55776)
Change-Id: Ibec6c267f9138aab626359c703fc067f91e1ee43
Reviewed-on: http://review.couchbase.org/63419
Well-Formed: buildbot <[email protected]>
Reviewed-by: Will Gardner <[email protected]>
Reviewed-by: Chiyoung Seo <[email protected]>
Tested-by: abhinav dangeti <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Jul 7, 2016
…t lock + [Not a backport, this code was altered/removed in master] + Address this lock inversion by moving the code that reads the vbucket snapshot range to outside the vbucket's state lock context. 15:30:43 WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=235352) 15:30:43 Cycle in lock order graph: M21536 (0x7d640002f720) => M21533 (0x7d640002f5f0) => M21536 15:30:43 15:30:43 Mutex M21533 acquired here while holding mutex M21536 in thread T17: 15:30:43 #0 pthread_rwlock_rdlock <null> (engine_testapp+0x000000462260) 15:30:43 #1 cb_rw_reader_enter <null> (libplatform.so.0.1.0+0x000000004800) 15:30:43 #2 RWLock::readerLock() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/rwlock.h:38 (ep.so+0x0000001327e0) 15:30:43 #3 ReaderLockHolder::ReaderLockHolder(RWLock&) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/locks.h:167 (ep.so+0x0000000f84c7) 15:30:43 #4 EventuallyPersistentStore::addTAPBackfillItem(Item const&, unsigned char, bool) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep.cc:851 (ep.so+0x0000000d9c67) 15:30:43 #5 PassiveStream::commitMutation(MutationResponse*, bool) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/dcp-stream.cc:1370 (ep.so+0x00000029e25c) 15:30:43 #6 PassiveStream::processMutation(MutationResponse*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/dcp-stream.cc:1346 (ep.so+0x00000029d0a0) 15:30:43 #7 PassiveStream::processBufferedMessages(unsigned int&) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/dcp-stream.cc:1286 (ep.so+0x00000029c9f2) 15:30:43 #8 DcpConsumer::processBufferedItems() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/dcp-consumer.cc:599 (ep.so+0x0000002632d4) 15:30:43 #9 Processer::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/dcp-consumer.cc:48 (ep.so+0x000000262ecf) 15:30:43 #10 ExecutorThread::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/executorthread.cc:109 (ep.so+0x0000001e3dc1) 15:30:43 #11 launch_executor_thread(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/executorthread.cc:34 (ep.so+0x0000001e33ea) 15:30:43 #12 platform_thread_wrap /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/platform/src/cb_pthreads.c (libplatform.so.0.1.0+0x00000000377c) 15:30:43 15:30:43 Mutex M21536 acquired here while holding mutex M21533 in main thread: 15:30:43 #0 pthread_mutex_lock <null> (engine_testapp+0x00000047e9e0) 15:30:43 #1 cb_mutex_enter <null> (libplatform.so.0.1.0+0x0000000039c0) 15:30:43 #2 Mutex::acquire() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/mutex.cc:31 (ep.so+0x0000001e28ee) 15:30:43 #3 LockHolder::lock() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/locks.h:71 (ep.so+0x000000080bc3) 15:30:43 #4 LockHolder::LockHolder(Mutex&, bool) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/locks.h:48 (ep.so+0x000000080832) 15:30:43 #5 VBucket::getCurrentSnapshot(unsigned long&, unsigned long&) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/vbucket.h:233 (ep.so+0x0000000fb245) 15:30:43 #6 ActiveStream::ActiveStream(EventuallyPersistentEngine*, SingleThreadedRCPtr<DcpProducer>, std::string const&, unsigned int, unsigned int, unsigned short, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/dcp-stream.cc:293 (ep.so+0x000000291276) 15:30:43 #7 DcpProducer::streamRequest(unsigned int, unsigned int, unsigned short, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long*, ENGINE_ERROR_CODE (*)(vbucket_failover_t*, unsigned long, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/dcp-producer.cc:259 (ep.so+0x00000027b9a3) 15:30:43 #8 EvpDcpStreamReq(engine_interface*, void const*, unsigned int, unsigned int, unsigned short, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long*, ENGINE_ERROR_CODE (*)(vbucket_failover_t*, unsigned long, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep_engine.cc:1471 (ep.so+0x0000001395e3) 15:30:43 #9 mock_dcp_stream_req /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/memcached/programs/engine_testapp/engine_testapp.c (engine_testapp+0x0000004caf81) 15:30:43 #10 dcp_stream(engine_interface*, engine_interface_v1*, char const*, void const*, unsigned short, unsigned int, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, int, int, int, int, int, bool, bool, unsigned long, bool) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/tests/ep_testsuite.cc:3427 (ep_testsuite.so+0x0000000b357e) 15:30:43 #11 test_dcp_replica_stream_backfill(engine_interface*, engine_interface_v1*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/tests/ep_testsuite.cc:5311 (ep_testsuite.so+0x00000008e78a) 15:30:43 #12 execute_test /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/memcached/programs/engine_testapp/engine_testapp.c (engine_testapp+0x0000004c4e9f) 15:30:43 #13 main crtstuff.c (engine_testapp+0x0000004c2e01) Change-Id: Idc09ce9af98669f74f28d1fd4b1cc15f7d8b1152 Reviewed-on: http://review.couchbase.org/63379 Well-Formed: buildbot <[email protected]> Tested-by: buildbot <[email protected]> Reviewed-by: Will Gardner <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Jul 7, 2016
WARNING: ThreadSanitizer: data race (pid=3212)
Write of size 8 at 0x7d5000016908 by thread T5 (mutexes: write M26478):
#0 PassiveStream::reconnectStream(RCPtr<VBucket>&, unsigned int, unsigned long) /home/abhinav/couchbase/ep-engine/src/dcp/stream.cc:1097 (ep.so+0x000000076c0f)
#1 DcpConsumer::doRollback(unsigned int, unsigned short, unsigned long) /home/abhinav/couchbase/ep-engine/src/dcp/consumer.cc:676 (ep.so+0x00000005db67)
#2 RollbackTask::run() /home/abhinav/couchbase/ep-engine/src/dcp/consumer.cc:574 (ep.so+0x00000005d9d4)
#3 ExecutorThread::run() /home/abhinav/couchbase/ep-engine/src/executorthread.cc:112 (ep.so+0x0000000f8916)
#4 launch_executor_thread(void*) /home/abhinav/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f84b5)
#5 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d31)
Previous read of size 8 at 0x7d5000016908 by main thread (mutexes: write M1367):
#0 PassiveStream::setDead_UNLOCKED(end_stream_status_t, LockHolder*) /home/abhinav/couchbase/ep-engine/src/dcp/stream.cc:1046 (ep.so+0x0000000759ca)
#1 PassiveStream::setDead(end_stream_status_t) /home/abhinav/couchbase/ep-engine/src/dcp/stream.cc:1056 (ep.so+0x0000000766d7)
#2 DcpConsumer::closeAllStreams() /home/abhinav/couchbase/ep-engine/src/dcp/consumer.cc:860 (ep.so+0x00000005a006)
#3 DcpConnMap::disconnect_UNLOCKED(void const*) /home/abhinav/couchbase/ep-engine/src/connmap.cc:1137 (ep.so+0x000000049972)
#4 DcpConnMap::disconnect(void const*) /home/abhinav/couchbase/ep-engine/src/connmap.cc:1111 (ep.so+0x00000004969b)
#5 EventuallyPersistentEngine::handleDisconnect(void const*) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:6224 (ep.so+0x0000000d3bea)
#6 EvpHandleDisconnect(void const*, ENGINE_EVENT_TYPE, void const*, void const*) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:1783 (ep.so+0x0000000b7046)
#7 mock_perform_callbacks /home/abhinav/couchbase/memcached/programs/engine_testapp/mock_server.c:296 (engine_testapp+0x0000000bd420)
#8 test_rollback_to_zero(engine_interface*, engine_interface_v1*) /home/abhinav/couchbase/ep-engine/tests/ep_testsuite.cc:5434 (ep_testsuite.so+0x00000007f45f)
#9 execute_test(test, char const*, char const*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1090 (engine_testapp+0x0000000b946c)
#10 __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287 (libc.so.6+0x000000021ec4)
(Reviewed-on: http://review.couchbase.org/55785)
Change-Id: I287bd95f8b03cb207419d0a0e57ca71be6058b19
Reviewed-on: http://review.couchbase.org/63446
Well-Formed: buildbot <[email protected]>
Reviewed-by: Dave Rigby <[email protected]>
Tested-by: buildbot <[email protected]>
Reviewed-by: Chiyoung Seo <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Jul 7, 2016
…stats request
WARNING: ThreadSanitizer: data race (pid=82258)
Read of size 1 at 0x7d4c0000a208 by thread T11 (mutexes: write M2483, write M19044):
#0 DcpProducer::addStats(void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/dcp/producer.cc:601 (ep.so+0x000000063e2d)
#1 ConnStatBuilder::operator()(SingleThreadedRCPtr<ConnHandler>&) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep_engine.cc:3903 (ep.so+0x0000000d6931)
#2 EventuallyPersistentEngine::doDcpStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep_engine.cc:4160 (ep.so+0x0000000b904a)
#3 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep_engine.cc:4580 (ep.so+0x0000000bcba4)
#4 EventuallyPersistentStore::snapshotStats() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep.cc:1700 (ep.so+0x000000088386)
#5 StatSnap::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/tasks.cc:98 (ep.so+0x00000012ba26)
#6 ExecutorThread::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorthread.cc:115 (ep.so+0x0000000eaeed)
#7 launch_executor_thread(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000eaab5)
#8 platform_thread_wrap(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/platform/src/cb_pthreads.cc:53 (libplatform.so.0.1.0+0x0000000048bb)
Previous write of size 1 at 0x7d4c0000a208 by main thread:
#0 DcpProducer::handleResponse(protocol_binary_response_header*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/dcp/producer.cc:547 (ep.so+0x000000063231)
#1 EvpDcpResponseHandler(engine_interface*, void const*, protocol_binary_response_header*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep_engine.cc:1765 (ep.so+0x0000000ae08b)
#2 mock_dcp_response_handler(engine_interface*, void const*, protocol_binary_response_header*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/memcached/programs/engine_testapp/engine_testapp.cc:796 (engine_testapp+0x0000004c68e5)
#3 sendDcpAck(engine_interface*, engine_interface_v1*, void const*, protocol_binary_command, protocol_binary_response_status, unsigned int) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/tests/ep_test_apis.cc:983 (ep_testsuite.so+0x0000000a6a22)
#4 test_dcp_noop(engine_interface*, engine_interface_v1*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/tests/ep_testsuite.cc:3975 (ep_testsuite.so+0x000000068deb)
#5 execute_test(test, char const*, char const*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/memcached/programs/engine_testapp/engine_testapp.cc:1090 (engine_testapp+0x0000004c4192)
#6 __libc_start_main /build/buildd/eglibc-2.15/csu/libc-start.c:226 (libc.so.6+0x00000002176c)
(Reviewed-on: http://review.couchbase.org/56306)
Change-Id: Ice7236da5cc885d9e7612894ba3d37e357e13b4a
Reviewed-on: http://review.couchbase.org/63440
Well-Formed: buildbot <[email protected]>
Reviewed-by: Dave Rigby <[email protected]>
Tested-by: buildbot <[email protected]>
Reviewed-by: Chiyoung Seo <[email protected]>
sriganes
pushed a commit
to sriganes/ep-engine
that referenced
this pull request
Jul 7, 2016
Flusher::taskId is accessed from different threads without correct
synchronization - see ThreadSanitizer report:
WARNING: ThreadSanitizer: data race (pid=13527)
Write of size 8 at 0x7d4400009650 by thread T14 (mutexes: write M5523):
#0 Flusher::step(GlobalTask*) couchbase/ep-engine/src/flusher.cc:200 (ep.so+0x0000000e78ae)
couchbase#1 FlusherTask::run() couchbase/ep-engine/src/tasks.cc:61 (ep.so+0x0000001202e2)
couchbase#2 ExecutorThread::run() couchbase/ep-engine/src/executorthread.cc:124 (ep.so+0x0000000e2e05)
couchbase#3 launch_executor_thread(void*) couchbase/ep-engine/src/executorthread.cc:34 (ep.so+0x0000000e28b9)
couchbase#4 platform_thread_wrap .ccache/tmp/cb_pthread.tmp.7e5bc917ff0e.45579.i:0 (libplatform.so.0.1.0+0x000000003891)
Previous read of size 8 at 0x7d4400009650 by main thread:
#0 Flusher::wait() couchbase/ep-engine/src/flusher.cc:41 (ep.so+0x0000000e66cf)
couchbase#1 EventuallyPersistentStore::~EventuallyPersistentStore() couchbase/ep-engine/src/ep.cc:514 (ep.so+0x000000071386)
couchbase#2 EventuallyPersistentEngine::~EventuallyPersistentEngine() couchbase/ep-engine/src/ep_engine.cc:6201 (ep.so+0x0000000bfeea)
couchbase#3 EvpDestroy(engine_interface*, bool) couchbase/ep-engine/src/ep_engine.cc:141 (ep.so+0x0000000a0e9c)
couchbase#4 mock_destroy(engine_interface*, bool) couchbase/memcached/programs/engine_testapp/engine_testapp.cc:98 (engine_testapp+0x0000000c4b87)
couchbase#5 execute_test(test, char const*, char const*) couchbase/memcached/programs/engine_testapp/engine_testapp.cc:995 (engine_testapp+0x0000000c4076)
#6 __libc_start_main /build/buildd/eglibc-2.15/csu/libc-start.c:226 (libc.so.6+0x00000002176c)
Given that taskId is a simple primitive type (size_t) fix by removing
the mutex (which wasn't acquired for all accesses) and replace taskId
with an atomic type.
Change-Id: Idc75278ed2882abd173297b77bdb72834cbe4163
Reviewed-on: http://review.couchbase.org/53220
Tested-by: buildbot <[email protected]>
Reviewed-by: Chiyoung Seo <[email protected]>
sriganes
pushed a commit
to sriganes/ep-engine
that referenced
this pull request
Jul 7, 2016
WARNING: ThreadSanitizer: data race (pid=12312)
Read of size 2 at 0x7d400000fff8 by main thread (mutexes: write M12542):
#0 VBCBAdaptor::getDescription() /home/abhinav/couchbase/ep-engine/src/ep.h:128 (ep.so+0x0000000a7fe1)
couchbase#1 ExecutorPool::_stopTaskGroup(unsigned long, task_type_t, bool) /home/abhinav/couchbase/ep-engine/src/executorpool.cc:562 (ep.so+0x0000000f3c21)
couchbase#2 ExecutorPool::stopTaskGroup(unsigned long, task_type_t, bool) /home/abhinav/couchbase/ep-engine/src/executorpool.cc:585 (ep.so+0x0000000f3f5e)
couchbase#3 ~EventuallyPersistentStore /home/abhinav/couchbase/ep-engine/src/ep.cc:468 (ep.so+0x0000000836c6)
couchbase#4 ~EventuallyPersistentEngine /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:6326 (ep.so+0x0000000d4eda)
couchbase#5 EvpDestroy(engine_interface*, bool) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:141 (ep.so+0x0000000b4b8c)
#6 mock_destroy(engine_interface*, bool) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:98 (engine_testapp+0x0000000ba027)
#7 destroy_bucket(engine_interface*, engine_interface_v1*, bool) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:995 (engine_testapp+0x0000000b952e)
#8 __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287 (libc.so.6+0x000000021ec4)
Previous write of size 2 at 0x7d400000fff8 by thread T10:
#0 VBCBAdaptor::run() /home/abhinav/couchbase/ep-engine/src/ep.cc:3776 (ep.so+0x00000009d7e3)
couchbase#1 ExecutorThread::run() /home/abhinav/couchbase/ep-engine/src/executorthread.cc:112 (ep.so+0x0000000f94d3)
couchbase#2 launch_executor_thread(void*) /home/abhinav/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f9055)
couchbase#3 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d31)
Change-Id: I316fb1a845fbee09f634d39e64057c170fab4795
Reviewed-on: http://review.couchbase.org/55775
Tested-by: abhinav dangeti <[email protected]>
Reviewed-by: Chiyoung Seo <[email protected]>
sriganes
pushed a commit
to sriganes/ep-engine
that referenced
this pull request
Jul 7, 2016
WARNING: ThreadSanitizer: data race (pid=14344)
Read of size 8 at 0x7d780000fa58 by thread T6:
#0 void STATWRITER_NAMESPACE::add_casted_stat<long>(char const*, long const&, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/abhinav/couchbase/ep-engine/src/statwriter.h:45 (ep.so+0x000000037ff5)
couchbase#1 EventuallyPersistentEngine::doEngineStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:3557 (ep.so+0x0000000be990)
couchbase#2 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:4554 (ep.so+0x0000000c5c8c)
couchbase#3 EventuallyPersistentStore::snapshotStats() /home/abhinav/couchbase/ep-engine/src/ep.cc:1671 (ep.so+0x00000008f1fe)
couchbase#4 StatSnap::run() /home/abhinav/couchbase/ep-engine/src/tasks.cc:97 (ep.so+0x00000013cea6)
couchbase#5 ExecutorThread::run() /home/abhinav/couchbase/ep-engine/src/executorthread.cc:112 (ep.so+0x0000000f94e3)
#6 launch_executor_thread(void*) /home/abhinav/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f9065)
#7 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d31)
Previous write of size 8 at 0x7d780000fa58 by main thread:
#0 EventuallyPersistentEngine::initialize(char const*) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:2167 (ep.so+0x0000000b728a)
couchbase#1 EvpInitialize(engine_interface*, char const*) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:133 (ep.so+0x0000000b4aa8)
couchbase#2 init_engine_instance /home/abhinav/couchbase/memcached/utilities/engine_loader.c:157 (libmcd_util.so.1.0.0+0x0000000058bb)
couchbase#3 create_bucket(bool, char const*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:980 (engine_testapp+0x0000000b9e12)
couchbase#4 execute_test(test, char const*, char const*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1083 (engine_testapp+0x0000000b93db)
couchbase#5 __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287 (libc.so.6+0x000000021ec4)
Change-Id: Ibec6c267f9138aab626359c703fc067f91e1ee43
Reviewed-on: http://review.couchbase.org/55776
Tested-by: buildbot <[email protected]>
Reviewed-by: Chiyoung Seo <[email protected]>
sriganes
pushed a commit
to sriganes/ep-engine
that referenced
this pull request
Jul 7, 2016
WARNING: ThreadSanitizer: data race (pid=23450)
Write of size 8 at 0x7d680001f678 by thread T5 (mutexes: write M11600, write M15531):
#0 VBucket::notifyCheckpointPersisted(EventuallyPersistentEngine&, unsigned long, bool) /home/abhinav/couchbase/ep-engine/src/vbucket.cc:356 (ep.so+0x00000014b3e3)
couchbase#1 EventuallyPersistentStore::flushVBucket(unsigned short) /home/abhinav/couchbase/ep-engine/src/ep.cc:3334 (ep.so+0x000000099b9f)
couchbase#2 Flusher::flushVB() /home/abhinav/couchbase/ep-engine/src/flusher.cc:296 (ep.so+0x0000000ffe7f)
couchbase#3 Flusher::step(GlobalTask*) /home/abhinav/couchbase/ep-engine/src/flusher.cc:186 (ep.so+0x0000000fe375)
couchbase#4 FlusherTask::run() /home/abhinav/couchbase/ep-engine/src/tasks.cc:62 (ep.so+0x00000013c928)
couchbase#5 ExecutorThread::run() /home/abhinav/couchbase/ep-engine/src/executorthread.cc:112 (ep.so+0x0000000f9503)
#6 launch_executor_thread(void*) /home/abhinav/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f9085)
#7 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d31)
Previous read of size 8 at 0x7d680001f678 by main thread (mutexes: write M910146210457775232):
#0 VBucket::getHighPriorityChkSize() /home/abhinav/couchbase/ep-engine/src/vbucket.cc:401 (ep.so+0x00000014b7d9)
couchbase#1 VBucketCountVisitor::visitBucket(RCPtr<VBucket>&) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:3030 (ep.so+0x0000000ba940)
couchbase#2 VBucketCountAggregator::visitBucket(RCPtr<VBucket>&) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:3067 (ep.so+0x0000000baf43)
couchbase#3 EventuallyPersistentStore::visit(VBucketVisitor&) /home/abhinav/couchbase/ep-engine/src/ep.cc:3719 (ep.so+0x000000089917)
couchbase#4 EventuallyPersistentEngine::doEngineStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:3093 (ep.so+0x0000000bb5ab)
couchbase#5 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:4554 (ep.so+0x0000000c5cac)
#6 EvpGetStats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:213 (ep.so+0x0000000b4dee)
#7 mock_get_stats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:239 (engine_testapp+0x0000000ba9ad)
#8 get_int_stat(engine_interface*, engine_interface_v1*, char const*, char const*) /home/abhinav/couchbase/ep-engine/tests/ep_test_apis.cc:990 (ep_testsuite.so+0x0000000aebb1)
#9 test_access_scanner(engine_interface*, engine_interface_v1*) /home/abhinav/couchbase/ep-engine/tests/ep_testsuite.cc:8569 (ep_testsuite.so+0x00000002efd7)
#10 execute_test(test, char const*, char const*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1090 (engine_testapp+0x0000000b946c)
#11 __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287 (libc.so.6+0x000000021ec4)
Change-Id: I98021a535bd78d34e31d428e364192bb2ef33dcf
Reviewed-on: http://review.couchbase.org/55777
Reviewed-by: Chiyoung Seo <[email protected]>
Tested-by: Chiyoung Seo <[email protected]>
sriganes
pushed a commit
to sriganes/ep-engine
that referenced
this pull request
Jul 7, 2016
WARNING: ThreadSanitizer: data race (pid=24180)
Read of size 4 at 0x7d040000f608 by main thread (mutexes: write M1308043):
#0 WorkLoadPolicy::stringOfWorkLoadPattern() /home/abhinav/couchbase/ep-engine/src/workload.h:65 (ep.so+0x0000000bee15)
couchbase#1 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:4554 (ep.so+0x0000000c5cac)
couchbase#2 EvpGetStats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:213 (ep.so+0x0000000b4dee)
couchbase#3 mock_get_stats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:239 (engine_testapp+0x0000000ba9ad)
couchbase#4 get_int_stat(engine_interface*, engine_interface_v1*, char const*, char const*) /home/abhinav/couchbase/ep-engine/tests/ep_test_apis.cc:990 (ep_testsuite.so+0x0000000aebb1)
couchbase#5 test_access_scanner(engine_interface*, engine_interface_v1*) /home/abhinav/couchbase/ep-engine/tests/ep_testsuite.cc:8569 (ep_testsuite.so+0x00000002efd7)
#6 execute_test(test, char const*, char const*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1090 (engine_testapp+0x0000000b946c)
#7 __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287 (libc.so.6+0x000000021ec4)
Previous write of size 4 at 0x7d040000f608 by thread T10:
#0 WorkLoadPolicy::setWorkLoadPattern(workload_pattern_t) /home/abhinav/couchbase/ep-engine/src/workload.h:76 (ep.so+0x00000013d75b)
couchbase#1 ExecutorThread::run() /home/abhinav/couchbase/ep-engine/src/executorthread.cc:112 (ep.so+0x0000000f9503)
couchbase#2 launch_executor_thread(void*) /home/abhinav/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f9085)
couchbase#3 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d31)
Change-Id: If1dd4885a7beefc804e425d077ff18b117be8bdd
Reviewed-on: http://review.couchbase.org/55778
Tested-by: buildbot <[email protected]>
Reviewed-by: Chiyoung Seo <[email protected]>
sriganes
pushed a commit
to sriganes/ep-engine
that referenced
this pull request
Jul 7, 2016
WARNING: ThreadSanitizer: data race (pid=26185)
Write of size 4 at 0x7d4c0000a154 by main thread:
#0 ConnHandler::setLastWalkTime() /home/abhinav/couchbase/ep-engine/src/tapconnection.h:356 (ep.so+0x000000065641)
couchbase#1 EvpDcpStep(engine_interface*, void const*, dcp_message_producers*) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:1478 (ep.so+0x0000000b4d5b)
couchbase#2 mock_dcp_step(engine_interface*, void const*, dcp_message_producers*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:476 (engine_testapp+0x0000000baf95)
couchbase#3 dcp_stream(engine_interface*, engine_interface_v1*, char const*, void const*, unsigned short, unsigned int, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, int, int, int, int, bool, bool, unsigned char, bool, unsigned long*, bool) /home/abhinav/couchbase/ep-engine/tests/ep_testsuite.cc:4164 (ep_testsuite.so+0x0000000990df)
couchbase#4 test_dcp_producer_stream_req_dgm(engine_interface*, engine_interface_v1*) /home/abhinav/couchbase/ep-engine/tests/ep_testsuite.cc:4564 (ep_testsuite.so+0x000000077634)
couchbase#5 execute_test(test, char const*, char const*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1090 (engine_testapp+0x0000000b946c)
#6 __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287 (libc.so.6+0x000000021ec4)
Previous read of size 4 at 0x7d4c0000a154 by thread T10 (mutexes: write M1367):
#0 ConnHandler::getLastWalkTime() /home/abhinav/couchbase/ep-engine/src/tapconnection.h:360 (ep.so+0x000000049cbe)
couchbase#1 ConnManager::run() /home/abhinav/couchbase/ep-engine/src/connmap.cc:150 (ep.so+0x00000005031e)
couchbase#2 ExecutorThread::run() /home/abhinav/couchbase/ep-engine/src/executorthread.cc:112 (ep.so+0x0000000f8746)
couchbase#3 launch_executor_thread(void*) /home/abhinav/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f82e5)
couchbase#4 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d31)
Change-Id: I2c5024afde6cb749aad901572bfd68734f6d7d5d
Reviewed-on: http://review.couchbase.org/55780
Tested-by: buildbot <[email protected]>
Reviewed-by: Chiyoung Seo <[email protected]>
sriganes
pushed a commit
to sriganes/ep-engine
that referenced
this pull request
Jul 7, 2016
WARNING: ThreadSanitizer: data race (pid=27028)
Read of size 8 at 0x7d480000b1f8 by main thread (mutexes: write M32941632, write M1367, write M32940809):
#0 void STATWRITER_NAMESPACE::add_casted_stat<unsigned long>(char const*, unsigned long const&, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/abhinav/couchbase/ep-engine/src/statwriter.h:45 (ep.so+0x000000037825)
couchbase#1 ActiveStream::addStats(void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/abhinav/couchbase/ep-engine/src/dcp/stream.cc:477 (ep.so+0x000000071d16)
couchbase#2 DcpProducer::addStats(void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/abhinav/couchbase/ep-engine/src/dcp/producer.cc:602 (ep.so+0x000000068057)
couchbase#3 ConnStatBuilder::operator()(SingleThreadedRCPtr<ConnHandler>&) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:3887 (ep.so+0x0000000e13e1)
couchbase#4 EventuallyPersistentEngine::doDcpStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:4144 (ep.so+0x0000000c151a)
couchbase#5 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:4564 (ep.so+0x0000000c5405)
#6 EvpGetStats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:213 (ep.so+0x0000000b422e)
#7 mock_get_stats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:239 (engine_testapp+0x0000000ba9ad)
#8 get_int_stat(engine_interface*, engine_interface_v1*, char const*, char const*) /home/abhinav/couchbase/ep-engine/tests/ep_test_apis.cc:990 (ep_testsuite.so+0x0000000aeb81)
#9 dcp_stream(engine_interface*, engine_interface_v1*, char const*, void const*, unsigned short, unsigned int, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, int, int, int, int, bool, bool, unsigned char, bool, unsigned long*, bool) /home/abhinav/couchbase/ep-engine/tests/ep_testsuite.cc:4090 (ep_testsuite.so+0x00000009790c)
#10 test_dcp_producer_stream_req_dgm(engine_interface*, engine_interface_v1*) /home/abhinav/couchbase/ep-engine/tests/ep_testsuite.cc:4564 (ep_testsuite.so+0x000000077604)
#11 execute_test(test, char const*, char const*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1090 (engine_testapp+0x0000000b946c)
#12 __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287 (libc.so.6+0x000000021ec4)
Previous write of size 8 at 0x7d480000b1f8 by thread T9 (mutexes: write M32940880, write M32940855):
#0 ActiveStream::backfillReceived(Item*, backfill_source_t) /home/abhinav/couchbase/ep-engine/src/dcp/stream.cc:287 (ep.so+0x00000007054e)
couchbase#1 DiskCallback::callback(GetValue&) /home/abhinav/couchbase/ep-engine/src/dcp/backfill.cc:94 (ep.so+0x000000056067)
couchbase#2 CouchKVStore::recordDbDump(_db*, _docinfo*, void*) /home/abhinav/couchbase/ep-engine/src/couch-kvstore/couch-kvstore.cc:1757 (ep.so+0x00000018103f)
couchbase#3 recordDbDumpC(_db*, _docinfo*, void*) /home/abhinav/couchbase/ep-engine/src/couch-kvstore/couch-kvstore.cc:66 (ep.so+0x00000017fcc5)
couchbase#4 lookup_callback(couchfile_lookup_request*, _sized_buf const*, _sized_buf const*) /home/abhinav/couchbase/couchstore/src/couch_db.cc:767 (libcouchstore.so+0x00000000d7f5)
couchbase#5 btree_lookup_inner(couchfile_lookup_request*, unsigned long, int, int) /home/abhinav/couchbase/couchstore/src/btree_read.cc:99 (libcouchstore.so+0x00000000b5b2)
#6 btree_lookup_inner(couchfile_lookup_request*, unsigned long, int, int) /home/abhinav/couchbase/couchstore/src/btree_read.cc:69 (libcouchstore.so+0x00000000b370)
#7 btree_lookup_inner(couchfile_lookup_request*, unsigned long, int, int) /home/abhinav/couchbase/couchstore/src/btree_read.cc:69 (libcouchstore.so+0x00000000b370)
#8 btree_lookup /home/abhinav/couchbase/couchstore/src/btree_read.cc:131 (libcouchstore.so+0x00000000b00c)
#9 couchstore_changes_since /home/abhinav/couchbase/couchstore/src/couch_db.cc:812 (libcouchstore.so+0x00000000d601)
#10 CouchKVStore::scan(ScanContext*) /home/abhinav/couchbase/ep-engine/src/couch-kvstore/couch-kvstore.cc:1264 (ep.so+0x00000017f77e)
#11 DCPBackfill::scan() /home/abhinav/couchbase/ep-engine/src/dcp/backfill.cc:193 (ep.so+0x000000057672)
#12 DCPBackfill::run() /home/abhinav/couchbase/ep-engine/src/dcp/backfill.cc:118 (ep.so+0x000000056647)
#13 BackfillManager::backfill() /home/abhinav/couchbase/ep-engine/src/dcp/backfill-manager.cc:240 (ep.so+0x0000000508d5)
#14 BackfillManagerTask::run() /home/abhinav/couchbase/ep-engine/src/dcp/backfill-manager.cc:43 (ep.so+0x00000005052f)
#15 ExecutorThread::run() /home/abhinav/couchbase/ep-engine/src/executorthread.cc:112 (ep.so+0x0000000f8796)
#16 launch_executor_thread(void*) /home/abhinav/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f8335)
#17 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d31)
Change-Id: I166917524b5fcad285b3623ff160e875c316d983
Reviewed-on: http://review.couchbase.org/55781
Tested-by: buildbot <[email protected]>
Reviewed-by: Chiyoung Seo <[email protected]>
sriganes
pushed a commit
to sriganes/ep-engine
that referenced
this pull request
Jul 7, 2016
WARNING: ThreadSanitizer: data race (pid=2443)
Read of size 1 at 0x7d5000016a58 by thread T10:
#0 ConnHandler::doDisconnect() /home/abhinav/couchbase/ep-engine/src/tapconnection.h:375 (ep.so+0x000000058416)
couchbase#1 ExecutorThread::run() /home/abhinav/couchbase/ep-engine/src/executorthread.cc:112 (ep.so+0x0000000f87f6)
couchbase#2 launch_executor_thread(void*) /home/abhinav/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f8395)
couchbase#3 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d31)
Previous write of size 1 at 0x7d5000016a58 by main thread:
[failed to restore the stack]
Location is heap block of size 480 at 0x7d5000016a00 allocated by main thread:
#0 operator new(unsigned long) <null>:0 (engine_testapp+0x00000005084d)
couchbase#1 DcpConnMap::newConsumer(void const*, std::string const&) /home/abhinav/couchbase/ep-engine/src/connmap.cc:969 (ep.so+0x000000048384)
couchbase#2 EventuallyPersistentEngine::dcpOpen(void const*, unsigned int, unsigned int, unsigned int, void*, unsigned short) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:6189 (ep.so+0x0000000d3668)
couchbase#3 EvpDcpOpen(engine_interface*, void const*, unsigned int, unsigned int, unsigned int, void*, unsigned short) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:1494 (ep.so+0x0000000b4e5f)
couchbase#4 mock_dcp_open(engine_interface*, void const*, unsigned int, unsigned int, unsigned int, void*, unsigned short) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:488 (engine_testapp+0x0000000bb015)
couchbase#5 test_dcp_consumer_flow_control_aggressive(engine_interface*, engine_interface_v1*) /home/abhinav/couchbase/ep-engine/tests/ep_testsuite.cc:3826 (ep_testsuite.so+0x00000006ecfd)
#6 execute_test(test, char const*, char const*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1090 (engine_testapp+0x0000000b946c)
#7 __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287 (libc.so.6+0x000000021ec4)
Change-Id: Id5223e93c416e5e5287c137d561aea1e453cbd41
Reviewed-on: http://review.couchbase.org/55784
Tested-by: buildbot <[email protected]>
Reviewed-by: Sundararaman Sridharan <[email protected]>
Reviewed-by: Chiyoung Seo <[email protected]>
sriganes
pushed a commit
to sriganes/ep-engine
that referenced
this pull request
Jul 7, 2016
WARNING: ThreadSanitizer: data race (pid=3212)
Write of size 8 at 0x7d5000016908 by thread T5 (mutexes: write M26478):
#0 PassiveStream::reconnectStream(RCPtr<VBucket>&, unsigned int, unsigned long) /home/abhinav/couchbase/ep-engine/src/dcp/stream.cc:1097 (ep.so+0x000000076c0f)
couchbase#1 DcpConsumer::doRollback(unsigned int, unsigned short, unsigned long) /home/abhinav/couchbase/ep-engine/src/dcp/consumer.cc:676 (ep.so+0x00000005db67)
couchbase#2 RollbackTask::run() /home/abhinav/couchbase/ep-engine/src/dcp/consumer.cc:574 (ep.so+0x00000005d9d4)
couchbase#3 ExecutorThread::run() /home/abhinav/couchbase/ep-engine/src/executorthread.cc:112 (ep.so+0x0000000f8916)
couchbase#4 launch_executor_thread(void*) /home/abhinav/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f84b5)
couchbase#5 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d31)
Previous read of size 8 at 0x7d5000016908 by main thread (mutexes: write M1367):
#0 PassiveStream::setDead_UNLOCKED(end_stream_status_t, LockHolder*) /home/abhinav/couchbase/ep-engine/src/dcp/stream.cc:1046 (ep.so+0x0000000759ca)
couchbase#1 PassiveStream::setDead(end_stream_status_t) /home/abhinav/couchbase/ep-engine/src/dcp/stream.cc:1056 (ep.so+0x0000000766d7)
couchbase#2 DcpConsumer::closeAllStreams() /home/abhinav/couchbase/ep-engine/src/dcp/consumer.cc:860 (ep.so+0x00000005a006)
couchbase#3 DcpConnMap::disconnect_UNLOCKED(void const*) /home/abhinav/couchbase/ep-engine/src/connmap.cc:1137 (ep.so+0x000000049972)
couchbase#4 DcpConnMap::disconnect(void const*) /home/abhinav/couchbase/ep-engine/src/connmap.cc:1111 (ep.so+0x00000004969b)
couchbase#5 EventuallyPersistentEngine::handleDisconnect(void const*) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:6224 (ep.so+0x0000000d3bea)
#6 EvpHandleDisconnect(void const*, ENGINE_EVENT_TYPE, void const*, void const*) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:1783 (ep.so+0x0000000b7046)
#7 mock_perform_callbacks /home/abhinav/couchbase/memcached/programs/engine_testapp/mock_server.c:296 (engine_testapp+0x0000000bd420)
#8 test_rollback_to_zero(engine_interface*, engine_interface_v1*) /home/abhinav/couchbase/ep-engine/tests/ep_testsuite.cc:5434 (ep_testsuite.so+0x00000007f45f)
#9 execute_test(test, char const*, char const*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1090 (engine_testapp+0x0000000b946c)
#10 __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287 (libc.so.6+0x000000021ec4)
Change-Id: I287bd95f8b03cb207419d0a0e57ca71be6058b19
Reviewed-on: http://review.couchbase.org/55785
Tested-by: buildbot <[email protected]>
Reviewed-by: Chiyoung Seo <[email protected]>
sriganes
pushed a commit
to sriganes/ep-engine
that referenced
this pull request
Jul 7, 2016
WARNING: ThreadSanitizer: data race (pid=4590)
Write of size 8 at 0x7d540000fe88 by thread T8 (mutexes: write M11599):
#0 EventuallyPersistentStore::flushVBucket(unsigned short) /home/abhinav/couchbase/ep-engine/src/ep.cc:3307 (ep.so+0x00000009954f)
couchbase#1 Flusher::flushVB() /home/abhinav/couchbase/ep-engine/src/flusher.cc:296 (ep.so+0x0000000ff32f)
couchbase#2 Flusher::step(GlobalTask*) /home/abhinav/couchbase/ep-engine/src/flusher.cc:186 (ep.so+0x0000000fd825)
couchbase#3 FlusherTask::run() /home/abhinav/couchbase/ep-engine/src/tasks.cc:63 (ep.so+0x00000013bbb2)
couchbase#4 ExecutorThread::run() /home/abhinav/couchbase/ep-engine/src/executorthread.cc:112 (ep.so+0x0000000f89b6)
couchbase#5 launch_executor_thread(void*) /home/abhinav/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f8555)
#6 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d31)
Previous write of size 8 at 0x7d540000fe88 by thread T6 (mutexes: write M11602):
#0 EventuallyPersistentStore::flushVBucket(unsigned short) /home/abhinav/couchbase/ep-engine/src/ep.cc:3307 (ep.so+0x00000009954f)
couchbase#1 Flusher::flushVB() /home/abhinav/couchbase/ep-engine/src/flusher.cc:296 (ep.so+0x0000000ff32f)
couchbase#2 Flusher::step(GlobalTask*) /home/abhinav/couchbase/ep-engine/src/flusher.cc:186 (ep.so+0x0000000fd825)
couchbase#3 FlusherTask::run() /home/abhinav/couchbase/ep-engine/src/tasks.cc:63 (ep.so+0x00000013bbb2)
couchbase#4 ExecutorThread::run() /home/abhinav/couchbase/ep-engine/src/executorthread.cc:112 (ep.so+0x0000000f89b6)
couchbase#5 launch_executor_thread(void*) /home/abhinav/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f8555)
#6 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d31)
Change-Id: Iccf1b0eacba495a8147fe81922361d566cb1d6a0
Reviewed-on: http://review.couchbase.org/55786
Reviewed-by: Sundararaman Sridharan <[email protected]>
Tested-by: buildbot <[email protected]>
Reviewed-by: Chiyoung Seo <[email protected]>
sriganes
pushed a commit
to sriganes/ep-engine
that referenced
this pull request
Jul 7, 2016
WARNING: ThreadSanitizer: data race (pid=5892)
Atomic write of size 8 at 0x7d780000f788 by thread T20:
#0 __tsan_atomic64_store <null>:0 (engine_testapp+0x00000009b837)
couchbase#1 std::__atomic_base<unsigned long>::store(unsigned long, std::memory_order) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/bits/atomic_base.h:474 (ep.so+0x000000104462)
couchbase#2 ExecutorThread::run() /home/abhinav/couchbase/ep-engine/src/executorthread.cc:112 (ep.so+0x0000000f89a6)
couchbase#3 launch_executor_thread(void*) /home/abhinav/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f8545)
couchbase#4 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d31)
Previous read of size 8 at 0x7d780000f788 by main thread (mutexes: write M57136293536132864):
#0 gmtime_r <null>:0 (engine_testapp+0x00000006235e)
couchbase#1 cb_gmtime_r /home/abhinav/couchbase/platform/src/cb_time.c:92 (libplatform.so.0.1.0+0x0000000063de)
couchbase#2 EventuallyPersistentEngine::doEngineStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:3544 (ep.so+0x0000000bdf1d)
couchbase#3 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:4554 (ep.so+0x0000000c52fc)
couchbase#4 EvpGetStats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:213 (ep.so+0x0000000b441e)
couchbase#5 mock_get_stats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:239 (engine_testapp+0x0000000ba9ad)
#6 get_int_stat(engine_interface*, engine_interface_v1*, char const*, char const*) /home/abhinav/couchbase/ep-engine/tests/ep_test_apis.cc:990 (ep_testsuite.so+0x0000000aeb81)
#7 wait_for_stat_change(engine_interface*, engine_interface_v1*, char const*, int, char const*, long) /home/abhinav/couchbase/ep-engine/tests/ep_test_apis.cc:1108 (ep_testsuite.so+0x0000000b4f93)
#8 test_expiration_on_warmup(engine_interface*, engine_interface_v1*) /home/abhinav/couchbase/ep-engine/tests/ep_testsuite.cc:2402 (ep_testsuite.so+0x00000001d2cb)
#9 execute_test(test, char const*, char const*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1090 (engine_testapp+0x0000000b946c)
#10 __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287 (libc.so.6+0x000000021ec4)
Change-Id: Ibfdcc6df03fd968be6fdd694ca5ba979bb36068c
Reviewed-on: http://review.couchbase.org/55789
Tested-by: buildbot <[email protected]>
Reviewed-by: Chiyoung Seo <[email protected]>
sriganes
pushed a commit
to sriganes/ep-engine
that referenced
this pull request
Jul 7, 2016
WARNING: ThreadSanitizer: data race (pid=7679)
Read of size 8 at 0x7d780000f778 by main thread (mutexes: write M31817041):
#0 gmtime_r <null>:0 (engine_testapp+0x00000006235e)
couchbase#1 cb_gmtime_r /home/abhinav/couchbase/platform/src/cb_time.c:92 (libplatform.so.0.1.0+0x0000000063de)
couchbase#2 EventuallyPersistentEngine::doEngineStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:3528 (ep.so+0x0000000bde17)
couchbase#3 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:4555 (ep.so+0x0000000c531c)
couchbase#4 EvpGetStats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:213 (ep.so+0x0000000b441e)
couchbase#5 mock_get_stats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:239 (engine_testapp+0x0000000ba9ad)
#6 get_int_stat(engine_interface*, engine_interface_v1*, char const*, char const*) /home/abhinav/couchbase/ep-engine/tests/ep_test_apis.cc:990 (ep_testsuite.so+0x0000000aeb81)
#7 wait_for_stat_to_be(engine_interface*, engine_interface_v1*, char const*, int, char const*, long) /home/abhinav/couchbase/ep-engine/tests/ep_test_apis.cc:1120 (ep_testsuite.so+0x0000000b5093)
#8 test_access_scanner(engine_interface*, engine_interface_v1*) /home/abhinav/couchbase/ep-engine/tests/ep_testsuite.cc:8591 (ep_testsuite.so+0x00000002f4f0)
#9 execute_test(test, char const*, char const*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1090 (engine_testapp+0x0000000b946c)
#10 __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287 (libc.so.6+0x000000021ec4)
Previous atomic write of size 8 at 0x7d780000f778 by thread T9:
#0 __tsan_atomic64_store <null>:0 (engine_testapp+0x00000009b837)
couchbase#1 std::__atomic_base<unsigned long>::store(unsigned long, std::memory_order) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/bits/atomic_base.h:474 (ep.so+0x0000000199a8)
couchbase#2 ExecutorThread::run() /home/abhinav/couchbase/ep-engine/src/executorthread.cc:112 (ep.so+0x0000000f89c6)
couchbase#3 launch_executor_thread(void*) /home/abhinav/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f8565)
couchbase#4 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d31)
Change-Id: I076daf1df26164da3038cdd4dfd7ad405104dfc7
Reviewed-on: http://review.couchbase.org/55790
Tested-by: buildbot <[email protected]>
Reviewed-by: Chiyoung Seo <[email protected]>
sriganes
pushed a commit
to sriganes/ep-engine
that referenced
this pull request
Jul 7, 2016
WARNING: ThreadSanitizer: data race (pid=19078)
Write of size 1 at 0x7d600000f078 by thread T10 (mutexes: write M21717):
#0 Notifiable::setSuspended(bool) /home/abhinav/couchbase/ep-engine/src/tapconnection.h:764 (ep.so+0x00000005fe6a)
couchbase#1 TapProducer::suspendedConnection_UNLOCKED(bool) /home/abhinav/couchbase/ep-engine/src/tapconnection.cc:717 (ep.so+0x00000013b24e)
couchbase#2 ExecutorThread::run() /home/abhinav/couchbase/ep-engine/src/executorthread.cc:112 (ep.so+0x0000000f89f6)
couchbase#3 launch_executor_thread(void*) /home/abhinav/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f8595)
couchbase#4 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d31)
Previous read of size 1 at 0x7d600000f078 by main thread (mutexes: write M21715):
#0 Notifiable::isSuspended() /home/abhinav/couchbase/ep-engine/src/tapconnection.h:768 (ep.so+0x0000000dfdf2)
couchbase#1 EventuallyPersistentEngine::walkTapQueue(void const*, void**, void**, unsigned short*, unsigned char*, unsigned short*, unsigned int*, unsigned short*) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:2531 (ep.so+0x0000000b7ecc)
couchbase#2 EvpTapIterator(engine_interface*, void const*, void**, void**, unsigned short*, unsigned char*, unsigned short*, unsigned int*, unsigned short*) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:1440 (ep.so+0x0000000d5ad7)
couchbase#3 mock_tap_iterator(engine_interface*, void const*, void**, void**, unsigned short*, unsigned char*, unsigned short*, unsigned int*, unsigned short*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:81 (engine_testapp+0x0000000bbda2)
couchbase#4 test_tap_ack_stream(engine_interface*, engine_interface_v1*) /home/abhinav/couchbase/ep-engine/tests/ep_testsuite.cc:7353 (ep_testsuite.so+0x0000000504a7)
couchbase#5 execute_test(test, char const*, char const*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1090 (engine_testapp+0x0000000b946c)
#6 __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287 (libc.so.6+0x000000021ec4)
Change-Id: I596c93c048767911b052861193822ca17270a5dd
Reviewed-on: http://review.couchbase.org/55792
Tested-by: buildbot <[email protected]>
Reviewed-by: Chiyoung Seo <[email protected]>
sriganes
pushed a commit
to sriganes/ep-engine
that referenced
this pull request
Jul 7, 2016
… size
As reported by ThreadSanitizer (see below) when running ep_testsuite
47 (get/delete with missing db file).
EPStore::bgFetch() reads the size of the vBuckets' pending BG fetch
queue without a lock / holding pendingBGFetchesLock. Given this is
only used for a debug message it's essentially benign, however fix it
by refactoring the code so VBucket::queueBGFetchItem() returns the
current queue size, removing all need for the accessor.
Also simplify the debug code so we don't need to create a temporary
string when the log message is disabled.
ThreadSanitizer report:
WARNING: ThreadSanitizer: data race (pid=11844)
Read of size 8 at 0x7d680001f5c0 by main thread (mutexes: write M28852, write M17967):
#0 EventuallyPersistentStore::bgFetch(std::string const&, unsigned short, void const*, bool) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/hashtable.h:500 (ep.so+0x00000007f4d6)
couchbase#1 EventuallyPersistentStore::getInternal(std::string const&, unsigned short, void const*, bool, bool, vbucket_state_t, bool) /home/couchbase/couchbase/ep-engine/src/ep.cc:2033 (ep.so+0x00000008ade3)
couchbase#2 EventuallyPersistentEngine::get(void const*, void**, void const*, int, unsigned short, bool) /home/couchbase/couchbase/ep-engine/src/ep.h:277 (ep.so+0x0000000d40e1)
couchbase#3 EvpGet(engine_interface*, void const*, void**, void const*, int, unsigned short) /home/couchbase/couchbase/ep-engine/src/ep_engine.cc:201 (ep.so+0x0000000abbae)
couchbase#4 mock_get(engine_interface*, void const*, void**, void const*, int, unsigned short) /home/couchbase/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:206 (engine_testapp+0x0000004c4f32)
couchbase#5 test_get_delete_missing_file(engine_interface*, engine_interface_v1*) /home/couchbase/couchbase/ep-engine/tests/ep_testsuite.cc:1884 (ep_testsuite.so+0x000000017f5c)
#6 execute_test(test, char const*, char const*) /home/couchbase/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1090 (engine_testapp+0x0000004c40b2)
#7 __libc_start_main /build/buildd/eglibc-2.15/csu/libc-start.c:226 (libc.so.6+0x00000002176c)
Previous write of size 8 at 0x7d680001f5c0 by thread T4 (mutexes: write M18022):
#0 memset <null> (engine_testapp+0x00000045dab0)
couchbase#1 VBucket::getBGFetchItems(std::unordered_map<std::string, vb_bgfetch_item_ctx_t, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, vb_bgfetch_item_ctx_t> > >&) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/hashtable.h:1944 (ep.so+0x000000137e54)
couchbase#2 BgFetcher::run(GlobalTask*) /home/couchbase/couchbase/ep-engine/src/bgfetcher.cc:132 (ep.so+0x00000002264a)
couchbase#3 BgFetcherTask::run() /home/couchbase/couchbase/ep-engine/src/tasks.cc:107 (ep.so+0x00000012abe2)
couchbase#4 ExecutorThread::run() /home/couchbase/couchbase/ep-engine/src/executorthread.cc:112 (ep.so+0x0000000ea34e)
couchbase#5 launch_executor_thread(void*) /home/couchbase/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000e9f05)
#6 platform_thread_wrap /home/couchbase/.ccache/tmp/cb_pthread.tmp.00b591814417.18511.i (libplatform.so.0.1.0+0x000000003d91)
Change-Id: I8152e33b9689e4009d13a695d29d26d895f58ccd
Reviewed-on: http://review.couchbase.org/55810
Reviewed-by: Chiyoung Seo <[email protected]>
Tested-by: Chiyoung Seo <[email protected]>
sriganes
pushed a commit
to sriganes/ep-engine
that referenced
this pull request
Jul 7, 2016
WARNING: ThreadSanitizer: data race (pid=16718)
Write of size 8 at 0x7f6a616aad60 by main thread:
#0 create_instance /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:1809:9 (ep.so+0x0000000b2c9c)
couchbase#1 create_engine_instance /home/abhinav/couchbase/memcached/utilities/engine_loader.c:121:31 (libmcd_util.so.1.0.0+0x0000000056fd)
couchbase#2 create_bucket(bool, char const*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:924:9 (engine_testapp+0x0000000b99ed)
couchbase#3 create_buckets(char const*, int, std::vector<BucketHolder, std::allocator<BucketHolder> >&) /home/abhinav/couchbase/ep-engine/tests/ep_testsuite_common.cc:316:36 (ep_testsuite.so+0x0000000ab215)
couchbase#4 test_multi_bucket_set_get(test*) /home/abhinav/couchbase/ep-engine/tests/ep_testsuite.cc:14174:9 (ep_testsuite.so+0x000000093011)
couchbase#5 execute_test(test, char const*, char const*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1103:19 (engine_testapp+0x0000000b937d)
#6 main /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1439 (engine_testapp+0x0000000b937d)
Previous read of size 8 at 0x7f6a616aad60 by thread T2 (mutexes: write M11526, write M11540):
#0 LOG(EXTENSION_LOG_LEVEL, char const*, ...) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:1916:9 (ep.so+0x0000000b2eb4)
couchbase#1 TaskQueue::_schedule(SingleThreadedRCPtr<GlobalTask>&) /home/abhinav/couchbase/ep-engine/src/taskqueue.cc:225:5 (ep.so+0x000000141957)
couchbase#2 TaskQueue::schedule(SingleThreadedRCPtr<GlobalTask>&) /home/abhinav/couchbase/ep-engine/src/taskqueue.cc:239:5 (ep.so+0x000000141bde)
couchbase#3 ExecutorPool::_schedule(SingleThreadedRCPtr<GlobalTask>, task_type_t) /home/abhinav/couchbase/ep-engine/src/executorpool.cc:430:5 (ep.so+0x0000000ef2f5)
couchbase#4 ExecutorPool::schedule(SingleThreadedRCPtr<GlobalTask>, task_type_t) /home/abhinav/couchbase/ep-engine/src/executorpool.cc:437:17 (ep.so+0x0000000ef44f)
couchbase#5 Warmup::scheduleEstimateDatabaseItemCount() /home/abhinav/couchbase/ep-engine/src/warmup.cc:538:9 (ep.so+0x000000158b7a)
#6 Warmup::step() /home/abhinav/couchbase/ep-engine/src/warmup.cc:883:13 (ep.so+0x000000155510)
#7 Warmup::transition(int, bool) /home/abhinav/couchbase/ep-engine/src/warmup.cc:920:9 (ep.so+0x00000015856e)
#8 Warmup::createVBuckets(unsigned short) /home/abhinav/couchbase/ep-engine/src/warmup.cc:525 (ep.so+0x00000015856e)
#9 WarmupCreateVBuckets::run() /home/abhinav/couchbase/ep-engine/src/warmup.h:249:9 (ep.so+0x000000165236)
#10 ExecutorThread::run() /home/abhinav/couchbase/ep-engine/src/executorthread.cc:112:26 (ep.so+0x0000000f8296)
#11 launch_executor_thread(void*) /home/abhinav/couchbase/ep-engine/src/executorthread.cc:33:9 (ep.so+0x0000000f7e35)
#12 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23:5 (libplatform.so.0.1.0+0x000000003d31)
Location is global 'loggerApi' of size 8 at 0x7f6a616aad60 (ep.so+0x00000042ad60)
Change-Id: I3ec917106a9684dfc8ce26664a6926135b6cb299
Reviewed-on: http://review.couchbase.org/55840
Reviewed-by: Chiyoung Seo <[email protected]>
Tested-by: buildbot <[email protected]>
sriganes
pushed a commit
to sriganes/ep-engine
that referenced
this pull request
Jul 7, 2016
WARNING: ThreadSanitizer: data race (pid=18824)
Write of size 4 at 0x7fcc31350244 by thread T12 (mutexes: write M44413):
#0 add_response /home/abhinav/couchbase/ep-engine/tests/ep_test_apis.cc:75 (ep_testsuite.so+0x0000000acbcc)
couchbase#1 sendResponse(bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*), void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:92 (ep.so+0x0000000d004c)
couchbase#2 processUnknownCommand(EventuallyPersistentEngine*, void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:1266 (ep.so+0x0000000d603c)
couchbase#3 EvpUnknownCommand(engine_interface*, void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:1387 (ep.so+0x0000000b3f58)
couchbase#4 mock_unknown_command(engine_interface*, void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:380 (engine_testapp+0x0000000bab29)
couchbase#5 del_with_meta(engine_interface*, engine_interface_v1*, char const*, unsigned long, unsigned int, ItemMetaData*, unsigned long, bool, bool, long, unsigned char, void const*) /home/abhinav/couchbase/ep-engine/tests/ep_test_apis.cc:360 (ep_testsuite.so+0x0000000ae69f)
#6 multi_del_with_meta(void*) /home/abhinav/couchbase/ep-engine/tests/ep_testsuite.cc:13299 (ep_testsuite.so+0x00000009572e)
#7 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d31)
Previous write of size 4 at 0x7fcc31350244 by thread T11 (mutexes: write M1638603450185076640):
#0 add_response /home/abhinav/couchbase/ep-engine/tests/ep_test_apis.cc:75 (ep_testsuite.so+0x0000000acbcc)
couchbase#1 sendResponse(bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*), void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:92 (ep.so+0x0000000ced72)
couchbase#2 processUnknownCommand(EventuallyPersistentEngine*, void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:1258 (ep.so+0x0000000d5718)
couchbase#3 EvpUnknownCommand(engine_interface*, void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:1387 (ep.so+0x0000000b3f58)
couchbase#4 mock_unknown_command(engine_interface*, void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:380 (engine_testapp+0x0000000bab29)
couchbase#5 set_with_meta(engine_interface*, engine_interface_v1*, char const*, unsigned long, char const*, unsigned long, unsigned int, ItemMetaData*, unsigned long, bool, unsigned char, bool, long, unsigned char, void const*) /home/abhinav/couchbase/ep-engine/tests/ep_test_apis.cc:702 (ep_testsuite.so+0x0000000b24db)
#6 multi_set_with_meta(void*) /home/abhinav/couchbase/ep-engine/tests/ep_testsuite.cc:13279 (ep_testsuite.so+0x000000094e33)
#7 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d31)
Change-Id: Ic53d401fb674dbe161aa73381e2c08c5995f262a
Reviewed-on: http://review.couchbase.org/55849
Reviewed-by: Chiyoung Seo <[email protected]>
Tested-by: buildbot <[email protected]>
sriganes
pushed a commit
to sriganes/ep-engine
that referenced
this pull request
Jul 7, 2016
WARNING: ThreadSanitizer: data race (pid=20056)
Write of size 8 at 0x7d600000f038 by main thread (mutexes: write M1412):
#0 ConnHandler::setCookie(void const*) /home/abhinav/couchbase/ep-engine/src/tapconnection.h:344 (ep.so+0x000000042367)
couchbase#1 EventuallyPersistentEngine::createTapQueue(void const*, std::string&, unsigned int, void const*, unsigned long) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:2655 (ep.so+0x0000000b86da)
couchbase#2 EvpGetTapIterator(engine_interface*, void const*, void const*, unsigned long, unsigned int, void const*, unsigned long) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:1462 (ep.so+0x0000000b46a3)
couchbase#3 mock_get_tap_iterator(engine_interface*, void const*, void const*, unsigned long, unsigned int, void const*, unsigned long) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:467 (engine_testapp+0x0000000bae3e)
couchbase#4 test_tap_ack_stream(engine_interface*, engine_interface_v1*) /home/abhinav/couchbase/ep-engine/tests/ep_testsuite.cc:7341 (ep_testsuite.so+0x000000050416)
couchbase#5 execute_test(test, char const*, char const*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1090 (engine_testapp+0x0000000b946c)
#6 __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287 (libc.so.6+0x000000021ec4)
Previous read of size 8 at 0x7d600000f038 by thread T9 (mutexes: write M1411):
#0 ConnHandler::getCookie() const /home/abhinav/couchbase/ep-engine/src/tapconnection.h:340 (ep.so+0x00000004067c)
couchbase#1 bool TapConnMap::performOp<Item*>(std::string const&, TapOperation<Item*>&, Item*) /home/abhinav/couchbase/ep-engine/src/connmap.h:389 (ep.so+0x00000001fa08)
couchbase#2 ItemResidentCallback::callback(CacheLookup&) /home/abhinav/couchbase/ep-engine/src/backfill.cc:63 (ep.so+0x00000001d9ca)
couchbase#3 CouchKVStore::recordDbDump(_db*, _docinfo*, void*) /home/abhinav/couchbase/ep-engine/src/couch-kvstore/couch-kvstore.cc:1654 (ep.so+0x000000180ca0)
couchbase#4 recordDbDumpC(_db*, _docinfo*, void*) /home/abhinav/couchbase/ep-engine/src/couch-kvstore/couch-kvstore.cc:66 (ep.so+0x00000017fe95)
couchbase#5 lookup_callback(couchfile_lookup_request*, _sized_buf const*, _sized_buf const*) /home/abhinav/couchbase/couchstore/src/couch_db.cc:767 (libcouchstore.so+0x00000000d7e5)
#6 btree_lookup_inner(couchfile_lookup_request*, unsigned long, int, int) /home/abhinav/couchbase/couchstore/src/btree_read.cc:99 (libcouchstore.so+0x00000000b5a2)
#7 btree_lookup /home/abhinav/couchbase/couchstore/src/btree_read.cc:131 (libcouchstore.so+0x00000000affc)
#8 couchstore_changes_since /home/abhinav/couchbase/couchstore/src/couch_db.cc:812 (libcouchstore.so+0x00000000d5f1)
#9 CouchKVStore::scan(ScanContext*) /home/abhinav/couchbase/ep-engine/src/couch-kvstore/couch-kvstore.cc:1264 (ep.so+0x00000017f94e)
#10 BackfillDiskLoad::run() /home/abhinav/couchbase/ep-engine/src/backfill.cc:131 (ep.so+0x00000001e449)
#11 ExecutorThread::run() /home/abhinav/couchbase/ep-engine/src/executorthread.cc:112 (ep.so+0x0000000f8956)
#12 launch_executor_thread(void*) /home/abhinav/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f84f5)
#13 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d31)
Change-Id: I8a668f17013c95abc9786d853ed2c6462cae5320
Reviewed-on: http://review.couchbase.org/55794
Tested-by: buildbot <[email protected]>
Reviewed-by: Chiyoung Seo <[email protected]>
sriganes
pushed a commit
to sriganes/ep-engine
that referenced
this pull request
Jul 7, 2016
As reported by ThreadSanitizer (see below), there is a race between
setting the current task associated with a ExecutorThread and reading
the name of that thread.
Unfortunately there doesn't seem to be a straightforward way to solve
this without adding a new mutex; currentTask (the variable the race is
on) is a SingleThreadedRCPtr, which is non-trivial to make thread-safe
(i.e. atomic). I did consider changing currenTask (and all other
ExTask variables) to be a std::shared_ptr as in C++11 this has support
for updating atomically, however the support in mainstream compilers
apparently isn't quite there yet.
Therefore I've just added a mutex to guard currentTask.
ThreadSanitizer report:
WARNING: ThreadSanitizer: data race (pid=27332)
Read of size 8 at 0x7d340000c8f0 by main thread (mutexes: write M19366):
#0 ExecutorThread::getTaskableName() const /home/couchbase/couchbase/ep-engine/src/atomic.h:309 (ep.so+0x0000000e6178)
couchbase#1 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/couchbase/ep-engine/src/ep_engine.cc:4346 (ep.so+0x0000000bc4dd)
couchbase#2 EvpGetStats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/couchbase/ep-engine/src/ep_engine.cc:213 (ep.so+0x0000000ab49e)
couchbase#3 mock_get_stats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:239 (engine_testapp+0x0000004c54ad)
couchbase#4 test_worker_stats(engine_interface*, engine_interface_v1*) /home/couchbase/couchbase/ep-engine/tests/ep_testsuite.cc:8901 (ep_testsuite.so+0x000000039768)
couchbase#5 execute_test(test, char const*, char const*) /home/couchbase/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1090 (engine_testapp+0x0000004c40b2)
#6 __libc_start_main /build/buildd/eglibc-2.15/csu/libc-start.c:226 (libc.so.6+0x00000002176c)
Previous write of size 8 at 0x7d340000c8f0 by thread T5:
#0 ExecutorThread::run() /home/couchbase/couchbase/ep-engine/src/atomic.h:322 (ep.so+0x0000000e9906)
couchbase#1 launch_executor_thread(void*) /home/couchbase/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000e9795)
couchbase#2 platform_thread_wrap /home/couchbase/.ccache/tmp/cb_pthread.tmp.00b591814417.18511.i (libplatform.so.0.1.0+0x000000003d91)
Change-Id: Id02f7a98b40b952a415cf9027a8f2243af38fc4d
Reviewed-on: http://review.couchbase.org/55829
Tested-by: buildbot <[email protected]>
Reviewed-by: Chiyoung Seo <[email protected]>
sriganes
pushed a commit
to sriganes/ep-engine
that referenced
this pull request
Jul 7, 2016
As reported by ThreadSanitizer (see below), we have a lock inversion
creating a potential deadlock in Connmap, related to how we shutdown
connections:
There exists a cycle in lock order graph:
M2176 (Connmap::releaseLock in ConnMap::notifyPausedConnection() connmap.cc:235) =>
M128093 (mock_server::conn_struct::mutex) =>
M2177 (Connmap::connsLock in TapConnMap::shutdownAllConnections(), connmap.cc:770) =>
M2176 (Connmap::releaseLock in TapConnMap::shutdownAllConnections(), connmap.cc:777) DEADLOCK!
The problem appears to be that in TapConnMap::shutdownAllConnections()
we first acquire {connsLock}, then acquire {releaseLock}; all while
holding the cookie lock.
Fix is to drop releaseLock in shutdownAllConnections() once we've
released any references, *then* acquire the connLock to actually clear
out the connection map.
ThreadSanitizer report follows (irrelevent parts removed):
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=1087)
Cycle in lock order graph: M2176 (0x7d840001b810) => M128093 (0x7d280000efa0) => M2177 (0x7d840001b850) => M2176
Mutex M128093 acquired here while holding mutex M2176 in thread T10:
<cut>
Mutex M2176 previously acquired by the same thread here:
#0 pthread_mutex_lock <null> ()
...
couchbase#5 ConnMap::notifyPausedConnection() ep-engine/src/connmap.cc:235 ()
<cut>
Mutex M2177 acquired here while holding mutex M128093 in main thread:
#0 pthread_mutex_lock <null> ()
...
couchbase#5 TapConnMap::newProducer() ep-engine/src/connmap.cc:378 ()
#6 EventuallyPersistentEngine::createTapQueue() ep-engine/src/ep_engine.cc:2663:23 ()
<cut>
Mutex M128093 previously acquired by the same thread here:
#0 pthread_mutex_lock <null> ()
couchbase#1 cb_mutex_enter platform/src/cb_pthreads.c:115:14 ()
couchbase#2 lock_mock_cookie memcached/programs/engine_testapp/mock_server.c:436:4 ()
couchbase#3 test_tap_stream() ep-engine/tests/ep_testsuite.cc:6751:5 ()
couchbase#4 execute_test() memcached/programs/engine_testapp/engine_testapp.cc:1090:19 ()
couchbase#5 main memcached/programs/engine_testapp/engine_testapp.cc:1439 ()
Mutex M2176 acquired here while holding mutex M2177 in main thread:
#0 pthread_mutex_lock <null> ()
...
couchbase#5 TapConnMap::shutdownAllConnections() ep-engine/src/connmap.cc:777 ()
#6 EventuallyPersistentEngine::destroy() ep-engine/src/ep_engine.cc:2190:9 ()
<cut>
Mutex M2177 previously acquired by the same thread here:
#0 pthread_mutex_lock <null> ()
...
couchbase#5 TapConnMap::shutdownAllConnections() ep-engine/src/connmap.cc:770 ()
#6 EventuallyPersistentEngine::destroy() ep-engine/src/ep_engine.cc:2190:9 ()
<cut>
Change-Id: Ic9a4f028b202277729df025333ce630be056903d
Reviewed-on: http://review.couchbase.org/55863
Tested-by: buildbot <[email protected]>
Reviewed-by: Chiyoung Seo <[email protected]>
sriganes
pushed a commit
to sriganes/ep-engine
that referenced
this pull request
Jul 7, 2016
WARNING: ThreadSanitizer: data race (pid=27652)
Write of size 8 at 0x7d08000443c0 by main thread (mutexes: write M57876):
#0 operator delete(void*) <null>:0 (engine_testapp+0x000000050e7b)
couchbase#1 __gnu_cxx::new_allocator<std::_List_node<unsigned short> >::deallocate(std::_List_node<unsigned short>*, unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/ext/new_allocator.h:110 (ep.so+0x00000005d69a)
couchbase#2 DcpConsumer::step(dcp_message_producers*) /home/abhinav/couchbase/ep-engine/src/dcp/consumer.cc:516 (ep.so+0x00000005c5cc)
couchbase#3 EvpDcpStep(engine_interface*, void const*, dcp_message_producers*) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:1479 (ep.so+0x0000000b480b)
couchbase#4 mock_dcp_step(engine_interface*, void const*, dcp_message_producers*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:476 (engine_testapp+0x0000000bb055)
couchbase#5 dcp_step(engine_interface*, engine_interface_v1*, void const*) /home/abhinav/couchbase/ep-engine/tests/ep_test_apis.cc:1219 (ep_testsuite.so+0x0000000b61bd)
#6 test_chk_manager_rollback(engine_interface*, engine_interface_v1*) /home/abhinav/couchbase/ep-engine/tests/ep_testsuite.cc:5526 (ep_testsuite.so+0x0000000809b4)
#7 execute_test(test, char const*, char const*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1090 (engine_testapp+0x0000000b952c)
#8 __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287 (libc.so.6+0x000000021ec4)
Previous write of size 8 at 0x7d08000443c0 by thread T16:
#0 operator new(unsigned long) <null>:0 (engine_testapp+0x00000005090d)
couchbase#1 __gnu_cxx::new_allocator<std::_List_node<unsigned short> >::allocate(unsigned long, void const*) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/ext/new_allocator.h:104 (ep.so+0x00000005f265)
couchbase#2 PassiveStream::reconnectStream(RCPtr<VBucket>&, unsigned int, unsigned long) /home/abhinav/couchbase/ep-engine/src/dcp/stream.cc:1104 (ep.so+0x000000076f5f)
couchbase#3 DcpConsumer::doRollback(unsigned int, unsigned short, unsigned long) /home/abhinav/couchbase/ep-engine/src/dcp/consumer.cc:676 (ep.so+0x00000005db67)
couchbase#4 RollbackTask::run() /home/abhinav/couchbase/ep-engine/src/dcp/consumer.cc:574 (ep.so+0x00000005d9d4)
couchbase#5 ExecutorThread::run() /home/abhinav/couchbase/ep-engine/src/executorthread.cc:115 (ep.so+0x0000000f834c)
#6 launch_executor_thread(void*) /home/abhinav/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f7eb5)
#7 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d71)
Change-Id: I196a78e54bf8014967a51cdb081126597153f77b
Reviewed-on: http://review.couchbase.org/55881
Tested-by: buildbot <[email protected]>
Reviewed-by: Chiyoung Seo <[email protected]>
sriganes
pushed a commit
to sriganes/ep-engine
that referenced
this pull request
Jul 7, 2016
As reported by ThreadSanitizer. CouchKVStore maintains three maps of
vBucketID to counter - dbFileRevMap, cachedDocCount &
cachedDeleteCount. These are read by some of the stats functions
(e.g. doDcpVbTakeoverStats) without a lock and hence there is a
potential race.
Solve this by changing the type of these counters to RelaxedAtomic.
WARNING: ThreadSanitizer: data race (pid=14070)
Write of size 8 at 0x7d9000002000 by thread T7 (mutexes: write M12364):
#0 CouchKVStore::saveDocs(unsigned short, unsigned long, _doc**, _docinfo**, unsigned long, KVStatsCtx&) /home/couchbase/couchbase/ep-engine/src/couch-kvstore/couch-kvstore.cc:1932:9 (ep.so+0x000000146628)
couchbase#1 CouchKVStore::commit2couchstore(Callback<KVStatsCtx>*) /home/couchbase/couchbase/ep-engine/src/couch-kvstore/couch-kvstore.cc:1808:34 (ep.so+0x00000013fcb7)
couchbase#2 CouchKVStore::commit(Callback<KVStatsCtx>*) /home/couchbase/couchbase/ep-engine/src/couch-kvstore/couch-kvstore.cc:1095:13 (ep.so+0x00000013f941)
couchbase#3 EventuallyPersistentStore::commit(unsigned short) /home/couchbase/couchbase/ep-engine/src/ep.cc:3351:13 (ep.so+0x00000008a0f6)
couchbase#4 EventuallyPersistentStore::flushVBucket(unsigned short) /home/couchbase/couchbase/ep-engine/src/ep.cc:3298:17 (ep.so+0x0000000891b0)
couchbase#5 Flusher::flushVB() /home/couchbase/couchbase/ep-engine/src/flusher.cc:296:13 (ep.so+0x0000000ddd05)
#6 Flusher::step(GlobalTask*) /home/couchbase/couchbase/ep-engine/src/flusher.cc:186:9 (ep.so+0x0000000dc6e5)
#7 FlusherTask::run() /home/couchbase/couchbase/ep-engine/src/tasks.cc:63:12 (ep.so+0x000000112222)
#8 ExecutorThread::run() /home/couchbase/couchbase/ep-engine/src/executorthread.cc:115:26 (ep.so+0x0000000d86ee)
#9 launch_executor_thread(void*) /home/couchbase/couchbase/ep-engine/src/executorthread.cc:33:9 (ep.so+0x0000000d8265)
#10 platform_thread_wrap /home/couchbase/couchbase/platform/src/cb_pthreads.c:23:5 (libplatform.so.0.1.0+0x000000003c31)
Previous read of size 8 at 0x7d9000002000 by main thread (mutexes: write M18926):
#0 CouchKVStore::getNumPersistedDeletes(unsigned short) /home/couchbase/couchbase/ep-engine/src/couch-kvstore/couch-kvstore.cc:2239:23 (ep.so+0x000000147e0f)
couchbase#1 EventuallyPersistentEngine::doDcpVbTakeoverStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), std::string&, unsigned short) /home/couchbase/couchbase/ep-engine/src/ep_engine.cc:5721:28 (ep.so+0x0000000b196e)
couchbase#2 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/couchbase/ep-engine/src/ep_engine.cc:4672:14 (ep.so+0x0000000b069f)
couchbase#3 EvpGetStats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/couchbase/ep-engine/src/ep_engine.cc:214:38 (ep.so+0x00000009f26e)
couchbase#4 mock_get_stats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:239:19 (engine_testapp+0x0000004c553d)
couchbase#5 get_int_stat(engine_interface*, engine_interface_v1*, char const*, char const*) /home/couchbase/couchbase/ep-engine/tests/ep_test_apis.cc:990:24 (ep_testsuite.so+0x000000083c04)
#6 test_dcp_vbtakeover_no_stream(engine_interface*, engine_interface_v1*) /home/couchbase/couchbase/ep-engine/tests/ep_testsuite.cc:3565:15 (ep_testsuite.so+0x000000055d43)
#7 execute_test(test, char const*, char const*) /home/couchbase/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1090:19 (engine_testapp+0x0000004c4142)
#8 main /home/couchbase/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1439 (engine_testapp+0x0000004c4142)
Change-Id: I593b31417bfe47d8ed50dd986f51763604e54645
Reviewed-on: http://review.couchbase.org/55873
Reviewed-by: abhinav dangeti <[email protected]>
Reviewed-by: Chiyoung Seo <[email protected]>
Tested-by: Chiyoung Seo <[email protected]>
sriganes
pushed a commit
to sriganes/ep-engine
that referenced
this pull request
Jul 7, 2016
- As part of the cancelling the processer task in ExecutorPool's stopTaskGroup, the consumer's destructor will be invoked as the reference has been released. - The consumer's destructor will try to cancel the processer task, which tries to acquire ExecutorPool::tMutex, but the same lock was already acquired by this very thread as part of stopTaskGroup. #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 couchbase#1 0x00007fea5d5f5657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0 couchbase#2 0x00007fea5d5f5480 in __GI___pthread_mutex_lock (mutex=0x7d5000018120) at ../nptl/pthread_mutex_lock.c:79 couchbase#3 0x00007fea5e0c1cd5 in pthread_mutex_lock () couchbase#4 0x00007fea5da10ea9 in cb_mutex_enter (mutex=0x7d5000018120) at /home/abhinav/couchbase/platform/src/cb_pthreads.c:115 couchbase#5 0x00007fea588bce2a in Mutex::acquire (this=0x7d5000018118) at /home/abhinav/couchbase/ep-engine/src/mutex.cc:31 #6 0x00007fea58892277 in lock (this=<optimized out>) at /home/abhinav/couchbase/ep-engine/src/locks.h:70 #7 LockHolder (m=..., tryLock=false, this=<optimized out>) at /home/abhinav/couchbase/ep-engine/src/locks.h:47 #8 ExecutorPool::_cancel (this=0x7d5000018000, taskId=41, eraseTask=<optimized out>) at /home/abhinav/couchbase/ep-engine/src/executorpool.cc:295 #9 0x00007fea58892a38 in ExecutorPool::cancel (this=0x7d5000018120, taskId=128, eraseTask=false) at /home/abhinav/couchbase/ep-engine/src/executorpool.cc:328 #10 0x00007fea587fe9c7 in cancelTask (this=0x7d5000016800) at /home/abhinav/couchbase/ep-engine/src/dcp/consumer.cc:114 #11 DcpConsumer::~DcpConsumer (this=0x7d5000016800) at /home/abhinav/couchbase/ep-engine/src/dcp/consumer.cc:104 #12 0x00007fea587ff126 in DcpConsumer::~DcpConsumer (this=0x7d5000016800) at /home/abhinav/couchbase/ep-engine/src/dcp/consumer.cc:103 #13 0x00007fea58804e14 in ~SingleThreadedRCPtr (this=<optimized out>) at /home/abhinav/couchbase/ep-engine/src/atomic.h:271 #14 ~Processer (this=0x7d140000e600) at /home/abhinav/couchbase/ep-engine/src/dcp/consumer.cc:35 #15 Processer::~Processer (this=0x7d140000e600) at /home/abhinav/couchbase/ep-engine/src/dcp/consumer.cc:35 #16 0x00007fea58897596 in ~SingleThreadedRCPtr (this=0x7d140000e600) at /home/abhinav/couchbase/ep-engine/src/atomic.h:271 #17 ExecutorPool::_stopTaskGroup (this=<optimized out>, taskGID=<optimized out>, taskType=<optimized out>, force=<optimized out>) at /home/abhinav/couchbase/ep-engine/src/executorpool.cc:576 #18 0x00007fea58897d7f in ExecutorPool::stopTaskGroup (this=0x7d5000018120, taskGID=128, taskType=WRITER_TASK_IDX, force=false) at /home/abhinav/couchbase/ep-engine/src/executorpool.cc:585 #19 0x00007fea588283a7 in EventuallyPersistentStore::~EventuallyPersistentStore (this=0x7d540000fc80) at /home/abhinav/couchbase/ep-engine/src/ep.cc:468 #20 0x00007fea58878e4b in EventuallyPersistentEngine::~EventuallyPersistentEngine (this=0x7d780000f000) at /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:6336 #21 0x00007fea58858aad in EvpDestroy (handle=0x7d780000f000, force=<optimized out>) at /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:142 #22 0x00007fea5e1070e8 in mock_destroy (handle=<optimized out>, force=false) at /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:98 #23 0x00007fea5e1065ef in destroy_bucket (handle=0x7d480000b880, handle_v1=0x7d480000b880, force=false) at /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:995 #24 execute_test (engine=<optimized out>, default_cfg=<optimized out>, test=...) at /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1112 #25 main (argc=<optimized out>, argv=<optimized out>) at /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1439 Change-Id: Iaebb3d0ed8d0cfbcef13dafd689e7c08f94069d8 Reviewed-on: http://review.couchbase.org/55884 Reviewed-by: Chiyoung Seo <[email protected]> Tested-by: buildbot <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Jul 28, 2016
This has small improvements in memory and cpu usage.
Also fixes several ThreadSanitizer races from unit tests - for example:
WARNING: ThreadSanitizer: data race (pid=21672)
Write of size 8 at 0x7d140000e7b8 by main thread (mutexes: write M14972, write M14985):
#0 memcpy <null> (engine_testapp+0x000000453040)
#1 TaskQueue::_wake(SingleThreadedRCPtr<GlobalTask>&) /home/daver/repos/couchbase/server/ep-engine/src/taskqueue.cc:255 (ep.so+0x0000002577f6)
#2 TaskQueue::wake(SingleThreadedRCPtr<GlobalTask>&) /home/daver/repos/couchbase/server/ep-engine/src/taskqueue.cc:282 (ep.so+0x000000257c73)
#3 ExecutorPool::_wake(unsigned long) /home/daver/repos/couchbase/server/ep-engine/src/executorpool.cc:320 (ep.so+0x0000001acc76)
#4 ExecutorPool::wake(unsigned long) /home/daver/repos/couchbase/server/ep-engine/src/executorpool.cc:328 (ep.so+0x0000001ace13)
#5 Flusher::wait() /home/daver/repos/couchbase/server/ep-engine/src/flusher.cc:41 (ep.so+0x0000001cc4ff)
#6 EventuallyPersistentStore::stopFlusher() /home/daver/repos/couchbase/server/ep-engine/src/ep.cc:402 (ep.so+0x0000000d54d5)
#7 ~EventuallyPersistentStore /home/daver/repos/couchbase/server/ep-engine/src/ep.cc:364 (ep.so+0x0000000d49cb)
#8 ~EventuallyPersistentEngine /home/daver/repos/couchbase/server/ep-engine/src/ep_engine.cc:5778 (ep.so+0x000000161043)
#9 EvpDestroy(engine_interface*, bool) /home/daver/repos/couchbase/server/ep-engine/src/ep_engine.cc:143 (ep.so+0x000000135efa)
#10 mock_destroy /home/daver/repos/couchbase/server/memcached/programs/engine_testapp/engine_testapp.c:61 (engine_testapp+0x0000004bb9d6)
#11 destroy_engine /home/daver/repos/couchbase/server/memcached/programs/engine_testapp/engine_testapp.c:998 (engine_testapp+0x0000004bb646)
#12 execute_test /home/daver/repos/couchbase/server/memcached/programs/engine_testapp/engine_testapp.c:1048 (engine_testapp+0x0000004baa11)
#13 main /home/daver/repos/couchbase/server/memcached/programs/engine_testapp/engine_testapp.c:1296 (engine_testapp+0x0000004b8861)
Previous read of size 8 at 0x7d140000e7b8 by thread T14:
#0 ExecutorThread::run() /home/daver/repos/couchbase/server/ep-engine/src/executorthread.cc:106 (ep.so+0x0000001e3488)
#1 launch_executor_thread(void*) /home/daver/repos/couchbase/server/ep-engine/src/executorthread.cc:34 (ep.so+0x0000001e2a5a)
#2 platform_thread_wrap /home/daver/repos/couchbase/server/platform/src/cb_pthreads.c:19 (libplatform.so.0.1.0+0x0000000035dc)
Change-Id: I78fdddb832251fc062058c04f75f8d22c4c2f68d
Reviewed-on: http://review.couchbase.org/62912
Well-Formed: buildbot <[email protected]>
Reviewed-by: Will Gardner <[email protected]>
Tested-by: buildbot <[email protected]>
Reviewed-by: Chiyoung Seo <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Jul 28, 2016
Fix race identified by ThreadSanitizer:
WARNING: ThreadSanitizer: data race (pid=10664)
Read of size 4 at 0x7d4400009888 by main thread (mutexes: write M18321):
#0 Flusher::stateName() const /home/vagrant/couchbase-server/ep-engine/src/flusher.cc:119 (ep.so+0x0000001fc22e)
#1 EventuallyPersistentEngine::doEngineStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:3025 (ep.so+0x000000180649)
#2 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:4374 (ep.so+0x00000018d960)
#3 EvpGetStats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:214 (ep.so+0x000000174062)
#4 mock_get_stats /home/vagrant/couchbase-server/memcached/programs/engine_testapp/engine_testapp.c:196 (exe+0x0000000a741d)
#5 get_int_stat(engine_interface*, engine_interface_v1*, char const*, char const*) /home/vagrant/couchbase-server/ep-engine/tests/ep_test_apis.cc:799 (ep_testsuite.so+0x0000000dae46)
#6 verify_curr_items(engine_interface*, engine_interface_v1*, int, char const*) /home/vagrant/couchbase-server/ep-engine/tests/ep_test_apis.cc:834 (ep_testsuite.so+0x0000000e1d31)
#7 test_dcp_producer_stream_req_disk(engine_interface*, engine_interface_v1*) /home/vagrant/couchbase-server/ep-engine/tests/ep_testsuite.cc:3589 (ep_testsuite.so+0x000000094c9a)
#8 execute_test /home/vagrant/couchbase-server/memcached/programs/engine_testapp/engine_testapp.c:1055 (exe+0x0000000a3e83)
#9 main /home/vagrant/couchbase-server/memcached/programs/engine_testapp/engine_testapp.c:1313 (exe+0x0000000a1d84)
Previous write of size 4 at 0x7d4400009888 by thread T5:
#0 Flusher::transition_state(flusher_state) /home/vagrant/couchbase-server/ep-engine/src/flusher.cc:114 (ep.so+0x0000001fb7df)
#1 Flusher::step(GlobalTask*) /home/vagrant/couchbase-server/ep-engine/src/flusher.cc:167 (ep.so+0x0000001fc9d1)
#2 FlusherTask::run() /home/vagrant/couchbase-server/ep-engine/src/tasks.cc:44 (ep.so+0x00000027870e)
#3 ExecutorThread::run() /home/vagrant/couchbase-server/ep-engine/src/executorthread.cc:110 (ep.so+0x0000002160ff)
#4 launch_executor_thread(void*) /home/vagrant/couchbase-server/ep-engine/src/executorthread.cc:34 (ep.so+0x00000021560a)
#5 platform_thread_wrap /home/vagrant/couchbase-server/platform/src/cb_pthreads.c:19 (libplatform.so.0.1.0+0x0000000033f4)
Change-Id: Iaeb60efdc1032de7ba344e04cce454cc1d876d40
Reviewed-on: http://review.couchbase.org/62914
Well-Formed: buildbot <[email protected]>
Reviewed-by: Will Gardner <[email protected]>
Tested-by: buildbot <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Jul 28, 2016
Flusher::taskId is accessed from different threads without correct
synchronization - see ThreadSanitizer report:
WARNING: ThreadSanitizer: data race (pid=13527)
Write of size 8 at 0x7d4400009650 by thread T14 (mutexes: write M5523):
#0 Flusher::step(GlobalTask*) couchbase/ep-engine/src/flusher.cc:200 (ep.so+0x0000000e78ae)
#1 FlusherTask::run() couchbase/ep-engine/src/tasks.cc:61 (ep.so+0x0000001202e2)
#2 ExecutorThread::run() couchbase/ep-engine/src/executorthread.cc:124 (ep.so+0x0000000e2e05)
#3 launch_executor_thread(void*) couchbase/ep-engine/src/executorthread.cc:34 (ep.so+0x0000000e28b9)
#4 platform_thread_wrap .ccache/tmp/cb_pthread.tmp.7e5bc917ff0e.45579.i:0 (libplatform.so.0.1.0+0x000000003891)
Previous read of size 8 at 0x7d4400009650 by main thread:
#0 Flusher::wait() couchbase/ep-engine/src/flusher.cc:41 (ep.so+0x0000000e66cf)
#1 EventuallyPersistentStore::~EventuallyPersistentStore() couchbase/ep-engine/src/ep.cc:514 (ep.so+0x000000071386)
#2 EventuallyPersistentEngine::~EventuallyPersistentEngine() couchbase/ep-engine/src/ep_engine.cc:6201 (ep.so+0x0000000bfeea)
#3 EvpDestroy(engine_interface*, bool) couchbase/ep-engine/src/ep_engine.cc:141 (ep.so+0x0000000a0e9c)
#4 mock_destroy(engine_interface*, bool) couchbase/memcached/programs/engine_testapp/engine_testapp.cc:98 (engine_testapp+0x0000000c4b87)
#5 execute_test(test, char const*, char const*) couchbase/memcached/programs/engine_testapp/engine_testapp.cc:995 (engine_testapp+0x0000000c4076)
#6 __libc_start_main /build/buildd/eglibc-2.15/csu/libc-start.c:226 (libc.so.6+0x00000002176c)
Given that taskId is a simple primitive type (size_t) fix by removing
the mutex (which wasn't acquired for all accesses) and replace taskId
with an atomic type.
Change-Id: Idc75278ed2882abd173297b77bdb72834cbe4163
Reviewed-on: http://review.couchbase.org/62915
Well-Formed: buildbot <[email protected]>
Reviewed-by: Will Gardner <[email protected]>
Tested-by: buildbot <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Jul 28, 2016
Context: warmupState, estimatedWarmupCount,
warmup, metadata
As reported by ThreadSanitizer:
WARNING: ThreadSanitizer: data race (pid=7023)
Write of size 8 at 0x7d240000d5e0 by thread T7:
#0 Warmup::checkForAccessLog() /home/daver/repos/couchbase/server/ep-engine/src/warmup.cc:590 (ep.so+0x0000002d1ffc)
#1 WarmupCheckforAccessLog::run() /home/daver/repos/couchbase/server/ep-engine/src/warmup.h:303 (ep.so+0x0000002e2bfb)
#2 ExecutorThread::run() /home/daver/repos/couchbase/server/ep-engine/src/executorthread.cc:106 (ep.so+0x0000001e34f9)
#3 launch_executor_thread(void*) /home/daver/repos/couchbase/server/ep-engine/src/executorthread.cc:34 (ep.so+0x0000001e2b7a)
#4 platform_thread_wrap /home/daver/repos/couchbase/server/platform/src/cb_pthreads.c:19 (libplatform.so.0.1.0+0x0000000035dc)
Previous read of size 8 at 0x7d240000d5e0 by main thread (mutexes: write M699180732592992152):
#0 Warmup::addStats(void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) const /home/daver/repos/couchbase/server/ep-engine/src/warmup.cc:893 (ep.so+0x0000002d6086)
#1 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/daver/repos/couchbase/server/ep-engine/src/ep_engine.cc:4422 (ep.so+0x000000151813)
#2 EvpGetStats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/daver/repos/couchbase/server/ep-engine/src/ep_engine.cc:214 (ep.so+0x0000001367b2)
#3 mock_get_stats /home/daver/repos/couchbase/server/memcached/programs/engine_testapp/engine_testapp.c:194 (engine_testapp+0x0000004bde63)
#4 wait_for_warmup_complete(engine_interface*, engine_interface_v1*) /home/daver/repos/couchbase/server/ep-engine/tests/ep_test_apis.cc:898 (ep_testsuite.so+0x0000000ead95)
#5 test_setup(engine_interface*, engine_interface_v1*) /home/daver/repos/couchbase/server/ep-engine/tests/ep_testsuite.cc:168 (ep_testsuite.so+0x0000000237d3)
#6 execute_test /home/daver/repos/couchbase/server/memcached/programs/engine_testapp/engine_testapp.c:1037 (engine_testapp+0x0000004ba82b)
#7 main /home/daver/repos/couchbase/server/memcached/programs/engine_testapp/engine_testapp.c:1296 (engine_testapp+0x0000004b8861)
Change-Id: If96933b3b8b0aa1ed75073a0d8d629f138da081f
Reviewed-on: http://review.couchbase.org/62916
Well-Formed: buildbot <[email protected]>
Reviewed-by: Will Gardner <[email protected]>
Tested-by: buildbot <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Jul 28, 2016
Race identified by ThreadSanitizer:
Read of size 8 at 0x7d08000c1430 by thread T10:
#0 ConnNotifier::notifyConnections() /home/vagrant/couchbase-server/ep-engine/src/connmap.cc:127 (ep.so+0x0000002537e8)
#1 ConnNotifierCallback::run() /home/vagrant/couchbase-server/ep-engine/src/connmap.cc:80 (ep.so+0x000000277e8b)
#2 ExecutorThread::run() /home/vagrant/couchbase-server/ep-engine/src/executorthread.cc:110 (ep.so+0x0000002163af)
#3 launch_executor_thread(void*) /home/vagrant/couchbase-server/ep-engine/src/executorthread.cc:34 (ep.so+0x0000002158ba)
#4 platform_thread_wrap /home/vagrant/couchbase-server/platform/src/cb_pthreads.c:19 (libplatform.so.0.1.0+0x0000000033f4)
Previous write of size 8 at 0x7d08000c1430 by main thread:
#0 ConnNotifier::start() /home/vagrant/couchbase-server/ep-engine/src/connmap.cc:99 (ep.so+0x00000025336f)
#1 ConnMap::initialize(conn_notifier_type) /home/vagrant/couchbase-server/ep-engine/src/connmap.cc:194 (ep.so+0x0000002544a6)
#2 EventuallyPersistentEngine::initialize(char const*) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:2060 (ep.so+0x000000179b4a)
#3 EvpInitialize(engine_interface*, char const*) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:135 (ep.so+0x000000173675)
#4 init_engine /home/vagrant/couchbase-server/memcached/utilities/engine_loader.c:116 (libmcd_util.so.1.0.0+0x000000003fac)
#5 start_your_engines /home/vagrant/couchbase-server/memcached/programs/engine_testapp/engine_testapp.c:913 (exe+0x0000000a2fb5)
#6 execute_test /home/vagrant/couchbase-server/memcached/programs/engine_testapp/engine_testapp.c:1048 (exe+0x0000000a3d29)
#7 main /home/vagrant/couchbase-server/memcached/programs/engine_testapp/engine_testapp.c:1313 (exe+0x0000000a1d84)
Change-Id: I16cfdff1ea363bbb07a62a92f09f829483276b3d
Reviewed-on: http://review.couchbase.org/62917
Well-Formed: buildbot <[email protected]>
Reviewed-by: Will Gardner <[email protected]>
Tested-by: buildbot <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Jul 28, 2016
Address possible data races in ActiveStream context when gathering
stats.
WARNING: ThreadSanitizer: data race (pid=27028)
Read of size 8 at 0x7d480000b1f8 by main thread (mutexes: write M32941632, write M1367, write M32940809):
#0 void STATWRITER_NAMESPACE::add_casted_stat<unsigned long>(char const*, unsigned long const&, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/abhinav/couchbase/ep-engine/src/statwriter.h:45 (ep.so+0x000000037825)
#1 ActiveStream::addStats(void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/abhinav/couchbase/ep-engine/src/dcp/stream.cc:477 (ep.so+0x000000071d16)
#2 DcpProducer::addStats(void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/abhinav/couchbase/ep-engine/src/dcp/producer.cc:602 (ep.so+0x000000068057)
#3 ConnStatBuilder::operator()(SingleThreadedRCPtr<ConnHandler>&) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:3887 (ep.so+0x0000000e13e1)
#4 EventuallyPersistentEngine::doDcpStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:4144 (ep.so+0x0000000c151a)
#5 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:4564 (ep.so+0x0000000c5405)
#6 EvpGetStats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:213 (ep.so+0x0000000b422e)
#7 mock_get_stats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:239 (engine_testapp+0x0000000ba9ad)
#8 get_int_stat(engine_interface*, engine_interface_v1*, char const*, char const*) /home/abhinav/couchbase/ep-engine/tests/ep_test_apis.cc:990 (ep_testsuite.so+0x0000000aeb81)
#9 dcp_stream(engine_interface*, engine_interface_v1*, char const*, void const*, unsigned short, unsigned int, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, int, int, int, int, bool, bool, unsigned char, bool, unsigned long*, bool) /home/abhinav/couchbase/ep-engine/tests/ep_testsuite.cc:4090 (ep_testsuite.so+0x00000009790c)
#10 test_dcp_producer_stream_req_dgm(engine_interface*, engine_interface_v1*) /home/abhinav/couchbase/ep-engine/tests/ep_testsuite.cc:4564 (ep_testsuite.so+0x000000077604)
#11 execute_test(test, char const*, char const*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1090 (engine_testapp+0x0000000b946c)
#12 __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287 (libc.so.6+0x000000021ec4)
Previous write of size 8 at 0x7d480000b1f8 by thread T9 (mutexes: write M32940880, write M32940855):
#0 ActiveStream::backfillReceived(Item*, backfill_source_t) /home/abhinav/couchbase/ep-engine/src/dcp/stream.cc:287 (ep.so+0x00000007054e)
#1 DiskCallback::callback(GetValue&) /home/abhinav/couchbase/ep-engine/src/dcp/backfill.cc:94 (ep.so+0x000000056067)
#2 CouchKVStore::recordDbDump(_db*, _docinfo*, void*) /home/abhinav/couchbase/ep-engine/src/couch-kvstore/couch-kvstore.cc:1757 (ep.so+0x00000018103f)
#3 recordDbDumpC(_db*, _docinfo*, void*) /home/abhinav/couchbase/ep-engine/src/couch-kvstore/couch-kvstore.cc:66 (ep.so+0x00000017fcc5)
#4 lookup_callback(couchfile_lookup_request*, _sized_buf const*, _sized_buf const*) /home/abhinav/couchbase/couchstore/src/couch_db.cc:767 (libcouchstore.so+0x00000000d7f5)
#5 btree_lookup_inner(couchfile_lookup_request*, unsigned long, int, int) /home/abhinav/couchbase/couchstore/src/btree_read.cc:99 (libcouchstore.so+0x00000000b5b2)
#6 btree_lookup_inner(couchfile_lookup_request*, unsigned long, int, int) /home/abhinav/couchbase/couchstore/src/btree_read.cc:69 (libcouchstore.so+0x00000000b370)
#7 btree_lookup_inner(couchfile_lookup_request*, unsigned long, int, int) /home/abhinav/couchbase/couchstore/src/btree_read.cc:69 (libcouchstore.so+0x00000000b370)
#8 btree_lookup /home/abhinav/couchbase/couchstore/src/btree_read.cc:131 (libcouchstore.so+0x00000000b00c)
#9 couchstore_changes_since /home/abhinav/couchbase/couchstore/src/couch_db.cc:812 (libcouchstore.so+0x00000000d601)
#10 CouchKVStore::scan(ScanContext*) /home/abhinav/couchbase/ep-engine/src/couch-kvstore/couch-kvstore.cc:1264 (ep.so+0x00000017f77e)
#11 DCPBackfill::scan() /home/abhinav/couchbase/ep-engine/src/dcp/backfill.cc:193 (ep.so+0x000000057672)
#12 DCPBackfill::run() /home/abhinav/couchbase/ep-engine/src/dcp/backfill.cc:118 (ep.so+0x000000056647)
#13 BackfillManager::backfill() /home/abhinav/couchbase/ep-engine/src/dcp/backfill-manager.cc:240 (ep.so+0x0000000508d5)
#14 BackfillManagerTask::run() /home/abhinav/couchbase/ep-engine/src/dcp/backfill-manager.cc:43 (ep.so+0x00000005052f)
#15 ExecutorThread::run() /home/abhinav/couchbase/ep-engine/src/executorthread.cc:112 (ep.so+0x0000000f8796)
#16 launch_executor_thread(void*) /home/abhinav/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f8335)
#17 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d31)
Change-Id: I166917524b5fcad285b3623ff160e875c316d983
Reviewed-on: http://review.couchbase.org/62918
Well-Formed: buildbot <[email protected]>
Reviewed-by: Will Gardner <[email protected]>
Tested-by: buildbot <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Jul 28, 2016
WARNING: ThreadSanitizer: data race (pid=23450)
Write of size 8 at 0x7d680001f678 by thread T5 (mutexes: write M11600, write M15531):
#0 VBucket::notifyCheckpointPersisted(EventuallyPersistentEngine&, unsigned long, bool) /home/abhinav/couchbase/ep-engine/src/vbucket.cc:356 (ep.so+0x00000014b3e3)
#1 EventuallyPersistentStore::flushVBucket(unsigned short) /home/abhinav/couchbase/ep-engine/src/ep.cc:3334 (ep.so+0x000000099b9f)
#2 Flusher::flushVB() /home/abhinav/couchbase/ep-engine/src/flusher.cc:296 (ep.so+0x0000000ffe7f)
#3 Flusher::step(GlobalTask*) /home/abhinav/couchbase/ep-engine/src/flusher.cc:186 (ep.so+0x0000000fe375)
#4 FlusherTask::run() /home/abhinav/couchbase/ep-engine/src/tasks.cc:62 (ep.so+0x00000013c928)
#5 ExecutorThread::run() /home/abhinav/couchbase/ep-engine/src/executorthread.cc:112 (ep.so+0x0000000f9503)
#6 launch_executor_thread(void*) /home/abhinav/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f9085)
#7 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d31)
Previous read of size 8 at 0x7d680001f678 by main thread (mutexes: write M910146210457775232):
#0 VBucket::getHighPriorityChkSize() /home/abhinav/couchbase/ep-engine/src/vbucket.cc:401 (ep.so+0x00000014b7d9)
#1 VBucketCountVisitor::visitBucket(RCPtr<VBucket>&) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:3030 (ep.so+0x0000000ba940)
#2 VBucketCountAggregator::visitBucket(RCPtr<VBucket>&) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:3067 (ep.so+0x0000000baf43)
#3 EventuallyPersistentStore::visit(VBucketVisitor&) /home/abhinav/couchbase/ep-engine/src/ep.cc:3719 (ep.so+0x000000089917)
#4 EventuallyPersistentEngine::doEngineStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:3093 (ep.so+0x0000000bb5ab)
#5 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:4554 (ep.so+0x0000000c5cac)
#6 EvpGetStats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:213 (ep.so+0x0000000b4dee)
#7 mock_get_stats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:239 (engine_testapp+0x0000000ba9ad)
#8 get_int_stat(engine_interface*, engine_interface_v1*, char const*, char const*) /home/abhinav/couchbase/ep-engine/tests/ep_test_apis.cc:990 (ep_testsuite.so+0x0000000aebb1)
#9 test_access_scanner(engine_interface*, engine_interface_v1*) /home/abhinav/couchbase/ep-engine/tests/ep_testsuite.cc:8569 (ep_testsuite.so+0x00000002efd7)
#10 execute_test(test, char const*, char const*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1090 (engine_testapp+0x0000000b946c)
#11 __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287 (libc.so.6+0x000000021ec4)
Change-Id: I98021a535bd78d34e31d428e364192bb2ef33dcf
Reviewed-on: http://review.couchbase.org/62966
Well-Formed: buildbot <[email protected]>
Reviewed-by: Will Gardner <[email protected]>
Tested-by: buildbot <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Jul 28, 2016
There is a data race in updating EPStore::lastTransTimePerItem. This
could result in an incorrect value for the persist_time field in an
OBSERVE response.
WARNING: ThreadSanitizer: data race (pid=4590)
Write of size 8 at 0x7d540000fe88 by thread T8 (mutexes: write M11599):
#0 EventuallyPersistentStore::flushVBucket(unsigned short) /home/abhinav/couchbase/ep-engine/src/ep.cc:3307 (ep.so+0x00000009954f)
#1 Flusher::flushVB() /home/abhinav/couchbase/ep-engine/src/flusher.cc:296 (ep.so+0x0000000ff32f)
#2 Flusher::step(GlobalTask*) /home/abhinav/couchbase/ep-engine/src/flusher.cc:186 (ep.so+0x0000000fd825)
#3 FlusherTask::run() /home/abhinav/couchbase/ep-engine/src/tasks.cc:63 (ep.so+0x00000013bbb2)
#4 ExecutorThread::run() /home/abhinav/couchbase/ep-engine/src/executorthread.cc:112 (ep.so+0x0000000f89b6)
#5 launch_executor_thread(void*) /home/abhinav/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f8555)
#6 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d31)
Previous write of size 8 at 0x7d540000fe88 by thread T6 (mutexes: write M11602):
#0 EventuallyPersistentStore::flushVBucket(unsigned short) /home/abhinav/couchbase/ep-engine/src/ep.cc:3307 (ep.so+0x00000009954f)
#1 Flusher::flushVB() /home/abhinav/couchbase/ep-engine/src/flusher.cc:296 (ep.so+0x0000000ff32f)
#2 Flusher::step(GlobalTask*) /home/abhinav/couchbase/ep-engine/src/flusher.cc:186 (ep.so+0x0000000fd825)
#3 FlusherTask::run() /home/abhinav/couchbase/ep-engine/src/tasks.cc:63 (ep.so+0x00000013bbb2)
#4 ExecutorThread::run() /home/abhinav/couchbase/ep-engine/src/executorthread.cc:112 (ep.so+0x0000000f89b6)
#5 launch_executor_thread(void*) /home/abhinav/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f8555)
#6 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d31)
Change-Id: Iccf1b0eacba495a8147fe81922361d566cb1d6a0
Reviewed-on: http://review.couchbase.org/62967
Well-Formed: buildbot <[email protected]>
Reviewed-by: Will Gardner <[email protected]>
Tested-by: buildbot <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Jul 28, 2016
This variable is reported in the ThreadSanitizer output as being read
by a stats function, which could result in incorrect stats printed to
the user. However, from code inspection the compactor
(EventuallyPersistentStore::scheduleCompaction) also reads this
variable, and hence could result in incorrect task scheduling.
WARNING: ThreadSanitizer: data race (pid=24180)
Read of size 4 at 0x7d040000f608 by main thread (mutexes: write M1308043):
#0 WorkLoadPolicy::stringOfWorkLoadPattern() ep-engine/src/workload.h:65 (ep.so+0x0000000bee15)
#1 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) ep-engine/src/ep_engine.cc:4554 (ep.so+0x0000000c5cac)
#2 EvpGetStats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) ep-engine/src/ep_engine.cc:213 (ep.so+0x0000000b4dee)
#3 mock_get_stats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) memcached/programs/engine_testapp/engine_testapp.cc:239 (engine_testapp+0x0000000ba9ad)
#4 get_int_stat(engine_interface*, engine_interface_v1*, char const*, char const*) ep-engine/tests/ep_test_apis.cc:990 (ep_testsuite.so+0x0000000aebb1)
#5 test_access_scanner(engine_interface*, engine_interface_v1*) ep-engine/tests/ep_testsuite.cc:8569 (ep_testsuite.so+0x00000002efd7)
#6 execute_test(test, char const*, char const*) memcached/programs/engine_testapp/engine_testapp.cc:1090 (engine_testapp+0x0000000b946c)
#7 __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287 (libc.so.6+0x000000021ec4)
Previous write of size 4 at 0x7d040000f608 by thread T10:
#0 WorkLoadPolicy::setWorkLoadPattern(workload_pattern_t) ep-engine/src/workload.h:76 (ep.so+0x00000013d75b)
#1 ExecutorThread::run() ep-engine/src/executorthread.cc:112 (ep.so+0x0000000f9503)
#2 launch_executor_thread(void*) ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f9085)
#3 platform_thread_wrap platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d31)
Change-Id: If1dd4885a7beefc804e425d077ff18b117be8bdd
Reviewed-on: http://review.couchbase.org/62968
Well-Formed: buildbot <[email protected]>
Reviewed-by: Will Gardner <[email protected]>
Tested-by: buildbot <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Jul 28, 2016
Fix race as identified by ThreadSanitizer:
WARNING: ThreadSanitizer: data race (pid=4243)
Read of size 8 at 0x7d04000fde60 by main thread (mutexes: write M1367):
#0 std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> >::operator++() /usr/include/c++/4.8/bits/stl_list.h:235 (ep.so+0x00000013a129)
#1 std::iterator_traits<std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> > >::difference_type std::__distance<std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> > >(std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> >, std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> >, std::input_iterator_tag) /usr/include/c++/4.8/bits/stl_iterator_base_funcs.h:82 (ep.so+0x000000138b67)
#2 std::iterator_traits<std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> > >::difference_type std::distance<std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> > >(std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> >, std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> >) /usr/include/c++/4.8/bits/stl_iterator_base_funcs.h:118 (ep.so+0x000000136b63)
#3 std::list<SingleThreadedRCPtr<GlobalTask>, std::allocator<SingleThreadedRCPtr<GlobalTask> > >::size() const /usr/include/c++/4.8/bits/stl_list.h:874 (ep.so+0x000000135538)
#4 ExecutorPool::doTaskQStat(EventuallyPersistentEngine*, void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) ep-engine/src/executorpool.cc:654 (ep.so+0x0000001331b6)
#5 EventuallyPersistentEngine::doWorkloadStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) ep-engine/src/ep_engine.cc:4198 (ep.so+0x000000112ed3)
#6 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) ep-engine/src/ep_engine.cc:4454 (ep.so+0x000000114cb4)
#7 EvpGetStats ep-engine/src/ep_engine.cc:217 (ep.so+0x000000102b14)
#8 mock_get_stats memcached/programs/engine_testapp/engine_testapp.c:195 (exe+0x0000000026de)
#9 test_workload_stats ep-engine/tests/ep_testsuite.cc:7094 (ep_testsuite.so+0x00000004e931)
#10 execute_test memcached/programs/engine_testapp/engine_testapp.c:1055 (exe+0x0000000059fc)
#11 main memcached/programs/engine_testapp/engine_testapp.c:1313 (exe+0x000000006606)
Previous write of size 8 at 0x7d04000fde60 by thread T5 (mutexes: write M45):
#0 RCValue::_rc_decref() const ep-engine/src/atomic.h:293 (ep.so+0x000000096797)
#1 __gnu_cxx::new_allocator<std::_List_node<SingleThreadedRCPtr<GlobalTask> > >::allocate(unsigned long, void const*) /usr/include/c++/4.8/ext/new_allocator.h:104 (ep.so+0x00000017b42b)
#2 std::_List_base<SingleThreadedRCPtr<GlobalTask>, std::allocator<SingleThreadedRCPtr<GlobalTask> > >::_M_get_node() /usr/include/c++/4.8/bits/stl_list.h:334 (ep.so+0x00000017b0dc)
#3 std::_List_node<SingleThreadedRCPtr<GlobalTask> >* std::list<SingleThreadedRCPtr<GlobalTask>, std::allocator<SingleThreadedRCPtr<GlobalTask> > >::_M_create_node<SingleThreadedRCPtr<GlobalTask> const&>(SingleThreadedRCPtr<GlobalTask> const&) /usr/include/c++/4.8/bits/stl_list.h:502 (ep.so+0x00000017a25a)
#4 void std::list<SingleThreadedRCPtr<GlobalTask>, std::allocator<SingleThreadedRCPtr<GlobalTask> > >::_M_insert<SingleThreadedRCPtr<GlobalTask> const&>(std::_List_iterator<SingleThreadedRCPtr<GlobalTask> >, SingleThreadedRCPtr<GlobalTask> const&) /usr/include/c++/4.8/bits/stl_list.h:1561 (ep.so+0x000000178e7b)
#5 std::list<SingleThreadedRCPtr<GlobalTask>, std::allocator<SingleThreadedRCPtr<GlobalTask> > >::push_back(SingleThreadedRCPtr<GlobalTask> const&) /usr/include/c++/4.8/bits/stl_list.h:1016 (ep.so+0x00000017817b)
#6 TaskQueue::_fetchNextTask(ExecutorThread&, bool) ep-engine/src/taskqueue.cc:126 (ep.so+0x000000176932)
#7 TaskQueue::fetchNextTask(ExecutorThread&, bool) ep-engine/src/taskqueue.cc:142 (ep.so+0x000000176acf)
#8 ExecutorPool::_nextTask(ExecutorThread&, unsigned char) ep-engine/src/executorpool.cc:214 (ep.so+0x000000130707)
#9 ExecutorPool::nextTask(ExecutorThread&, unsigned char) ep-engine/src/executorpool.cc:229 (ep.so+0x0000001307a5)
#10 ExecutorThread::run() ep-engine/src/executorthread.cc:78 (ep.so+0x000000149da0)
#11 launch_executor_thread ep-engine/src/executorthread.cc:34 (ep.so+0x00000014990a)
#12 platform_thread_wrap platform/src/cb_pthreads.c:19 (libplatform.so.0.1.0+0x000000002d8b)
#13 __tsan_write_range ??:0 (libtsan.so.0+0x00000001b1c9)
Fix by adding new helper methods which will return the size of the
various queues (while holding the TaskQueue's mutex).
Change-Id: If5e8d357e45803d78c4ba6ed1475e6e1a90e1c89
Reviewed-on: http://review.couchbase.org/62969
Well-Formed: buildbot <[email protected]>
Reviewed-by: Will Gardner <[email protected]>
Tested-by: buildbot <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Jul 28, 2016
WARNING: ThreadSanitizer: data race (pid=2443)
Read of size 1 at 0x7d5000016a58 by thread T10:
#0 ConnHandler::doDisconnect() /home/abhinav/couchbase/ep-engine/src/tapconnection.h:375 (ep.so+0x000000058416)
#1 ExecutorThread::run() /home/abhinav/couchbase/ep-engine/src/executorthread.cc:112 (ep.so+0x0000000f87f6)
#2 launch_executor_thread(void*) /home/abhinav/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f8395)
#3 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d31)
Previous write of size 1 at 0x7d5000016a58 by main thread:
[failed to restore the stack]
Location is heap block of size 480 at 0x7d5000016a00 allocated by main thread:
#0 operator new(unsigned long) <null>:0 (engine_testapp+0x00000005084d)
#1 DcpConnMap::newConsumer(void const*, std::string const&) /home/abhinav/couchbase/ep-engine/src/connmap.cc:969 (ep.so+0x000000048384)
#2 EventuallyPersistentEngine::dcpOpen(void const*, unsigned int, unsigned int, unsigned int, void*, unsigned short) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:6189 (ep.so+0x0000000d3668)
#3 EvpDcpOpen(engine_interface*, void const*, unsigned int, unsigned int, unsigned int, void*, unsigned short) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:1494 (ep.so+0x0000000b4e5f)
#4 mock_dcp_open(engine_interface*, void const*, unsigned int, unsigned int, unsigned int, void*, unsigned short) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:488 (engine_testapp+0x0000000bb015)
#5 test_dcp_consumer_flow_control_aggressive(engine_interface*, engine_interface_v1*) /home/abhinav/couchbase/ep-engine/tests/ep_testsuite.cc:3826 (ep_testsuite.so+0x00000006ecfd)
#6 execute_test(test, char const*, char const*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1090 (engine_testapp+0x0000000b946c)
#7 __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287 (libc.so.6+0x000000021ec4)
Change-Id: Id5223e93c416e5e5287c137d561aea1e453cbd41
Reviewed-on: http://review.couchbase.org/62970
Well-Formed: buildbot <[email protected]>
Reviewed-by: Will Gardner <[email protected]>
Tested-by: buildbot <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Jul 28, 2016
These variables are used in the calculation of ep_db_data_size and
ep_db_file_size stats, and crucially those stats are used by ns_server
when determining if a bucket should be compacted.
It is possible that due to this issue, compaction may not be triggered
when expected, or triggered when it shouldn't.
As identified by ThreadSantizer:
WARNING: ThreadSanitizer: data race (pid=4009)
Write of size 8 at 0x7d440000c5b0 by thread T6 (mutexes: write M55):
#0 KVStatsCallback::callback(KVStatsCtx&) /repos/couchbase/server/source/ep-engine/src/ep.cc:933 (ep.so+0x0000000f0a22)
#1 CouchKVStore::commit2couchstore(Callback<KVStatsCtx>*, unsigned long, unsigned long) /repos/couchbase/server/source/ep-engine/src/couch-kvstore/couch-kvstore.cc:1697 (ep.so+0x0000001aa8c6)
#2 CouchKVStore::commit(Callback<KVStatsCtx>*, unsigned long, unsigned long) /repos/couchbase/server/source/ep-engine/src/couch-kvstore/couch-kvstore.cc:1040 (ep.so+0x0000001a6483)
#3 EventuallyPersistentStore::flushVBucket(unsigned short) /repos/couchbase/server/source/ep-engine/src/ep.cc:2909 (ep.so+0x0000000e780b)
#4 Flusher::flushVB() /repos/couchbase/server/source/ep-engine/src/flusher.cc:283 (ep.so+0x00000013f363)
#5 Flusher::step(GlobalTask*) /repos/couchbase/server/source/ep-engine/src/flusher.cc:174 (ep.so+0x00000013e9c8)
#6 FlusherTask::run() /repos/couchbase/server/source/ep-engine/src/tasks.cc:44 (ep.so+0x000000174a85)
#7 ExecutorThread::run() /repos/couchbase/server/source/ep-engine/src/executorthread.cc:110 (ep.so+0x00000014a0c1)
#8 launch_executor_thread /repos/couchbase/server/source/ep-engine/src/executorthread.cc:34 (ep.so+0x00000014990a)
#9 platform_thread_wrap /repos/couchbase/server/source/platform/src/cb_pthreads.c:19 (libplatform.so.0.1.0+0x000000002d8b)
#10 __tsan_write_range ??:0 (libtsan.so.0+0x00000001b1c9)
Previous read of size 8 at 0x7d440000c5b0 by main thread (mutexes: write M193510842443017784):
#0 VBucketCountVisitor::visitBucket(RCPtr<VBucket>&) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:2889 (ep.so+0x00000010c631)
#1 VBucketCountAggregator::visitBucket(RCPtr<VBucket>&) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:2926 (ep.so+0x000000121392)
#2 EventuallyPersistentStore::visit(VBucketVisitor&) /repos/couchbase/server/source/ep-engine/src/ep.cc:3278 (ep.so+0x0000000e99d9)
#3 EventuallyPersistentEngine::doEngineStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:2955 (ep.so+0x00000010cb46)
#4 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:4344 (ep.so+0x000000113c0f)
#5 EvpGetStats /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:217 (ep.so+0x000000102b14)
#6 mock_get_stats /repos/couchbase/server/source/memcached/programs/engine_testapp/engine_testapp.c:195 (exe+0x0000000026de)
#7 get_int_stat(engine_interface*, engine_interface_v1*, char const*, char const*) /repos/couchbase/server/source/ep-engine/tests/ep_test_apis.cc:799 (ep_testsuite.so+0x0000000832d8)
#8 wait_for_flusher_to_settle(engine_interface*, engine_interface_v1*) /repos/couchbase/server/source/ep-engine/tests/ep_test_apis.cc:900 (ep_testsuite.so+0x000000083cd4)
#9 wait_for_persisted_value(engine_interface*, engine_interface_v1*, char const*, char const*, unsigned short) /repos/couchbase/server/source/ep-engine/tests/ep_test_apis.cc:917 (ep_testsuite.so+0x000000083de5)
#10 test_io_stats /repos/couchbase/server/source/ep-engine/tests/ep_testsuite.cc:6279 (ep_testsuite.so+0x0000000468b3)
#11 execute_test /repos/couchbase/server/source/memcached/programs/engine_testapp/engine_testapp.c:1055 (exe+0x0000000059fc)
#12 main /repos/couchbase/server/source/memcached/programs/engine_testapp/engine_testapp.c:1313 (exe+0x000000006606)
Change to AtomicValues to fix this.
Change-Id: Ie7f9a403f809e751ff1802cceb5d2a77a483a586
Reviewed-on: http://review.couchbase.org/62971
Well-Formed: buildbot <[email protected]>
Reviewed-by: Jim Walker <[email protected]>
Reviewed-by: Will Gardner <[email protected]>
Tested-by: buildbot <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Jul 28, 2016
As reported by ThreadSanitizer (see below), there is a race between
setting the current task associated with a ExecutorThread and reading
the name of that thread.
Unfortunately there doesn't seem to be a straightforward way to solve
this without adding a new mutex; currentTask (the variable the race is
on) is a SingleThreadedRCPtr, which is non-trivial to make thread-safe
(i.e. atomic). I did consider changing currenTask (and all other
ExTask variables) to be a std::shared_ptr as in C++11 this has support
for updating atomically, however the support in mainstream compilers
apparently isn't quite there yet.
Therefore I've just added a mutex to guard currentTask.
ThreadSanitizer report:
WARNING: ThreadSanitizer: data race (pid=27332)
Read of size 8 at 0x7d340000c8f0 by main thread (mutexes: write M19366):
#0 ExecutorThread::getTaskableName() const /home/couchbase/couchbase/ep-engine/src/atomic.h:309 (ep.so+0x0000000e6178)
#1 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/couchbase/ep-engine/src/ep_engine.cc:4346 (ep.so+0x0000000bc4dd)
#2 EvpGetStats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/couchbase/ep-engine/src/ep_engine.cc:213 (ep.so+0x0000000ab49e)
#3 mock_get_stats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:239 (engine_testapp+0x0000004c54ad)
#4 test_worker_stats(engine_interface*, engine_interface_v1*) /home/couchbase/couchbase/ep-engine/tests/ep_testsuite.cc:8901 (ep_testsuite.so+0x000000039768)
#5 execute_test(test, char const*, char const*) /home/couchbase/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1090 (engine_testapp+0x0000004c40b2)
#6 __libc_start_main /build/buildd/eglibc-2.15/csu/libc-start.c:226 (libc.so.6+0x00000002176c)
Previous write of size 8 at 0x7d340000c8f0 by thread T5:
#0 ExecutorThread::run() /home/couchbase/couchbase/ep-engine/src/atomic.h:322 (ep.so+0x0000000e9906)
#1 launch_executor_thread(void*) /home/couchbase/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000e9795)
#2 platform_thread_wrap /home/couchbase/.ccache/tmp/cb_pthread.tmp.00b591814417.18511.i (libplatform.so.0.1.0+0x000000003d91)
Change-Id: Id02f7a98b40b952a415cf9027a8f2243af38fc4d
Reviewed-on: http://review.couchbase.org/62973
Well-Formed: buildbot <[email protected]>
Reviewed-by: Will Gardner <[email protected]>
Tested-by: buildbot <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Jul 28, 2016
WARNING: ThreadSanitizer: data race (pid=12312)
Read of size 2 at 0x7d400000fff8 by main thread (mutexes: write M12542):
#0 VBCBAdaptor::getDescription() /home/abhinav/couchbase/ep-engine/src/ep.h:128 (ep.so+0x0000000a7fe1)
#1 ExecutorPool::_stopTaskGroup(unsigned long, task_type_t, bool) /home/abhinav/couchbase/ep-engine/src/executorpool.cc:562 (ep.so+0x0000000f3c21)
#2 ExecutorPool::stopTaskGroup(unsigned long, task_type_t, bool) /home/abhinav/couchbase/ep-engine/src/executorpool.cc:585 (ep.so+0x0000000f3f5e)
#3 ~EventuallyPersistentStore /home/abhinav/couchbase/ep-engine/src/ep.cc:468 (ep.so+0x0000000836c6)
#4 ~EventuallyPersistentEngine /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:6326 (ep.so+0x0000000d4eda)
#5 EvpDestroy(engine_interface*, bool) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:141 (ep.so+0x0000000b4b8c)
#6 mock_destroy(engine_interface*, bool) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:98 (engine_testapp+0x0000000ba027)
#7 destroy_bucket(engine_interface*, engine_interface_v1*, bool) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:995 (engine_testapp+0x0000000b952e)
#8 __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287 (libc.so.6+0x000000021ec4)
Previous write of size 2 at 0x7d400000fff8 by thread T10:
#0 VBCBAdaptor::run() /home/abhinav/couchbase/ep-engine/src/ep.cc:3776 (ep.so+0x00000009d7e3)
#1 ExecutorThread::run() /home/abhinav/couchbase/ep-engine/src/executorthread.cc:112 (ep.so+0x0000000f94d3)
#2 launch_executor_thread(void*) /home/abhinav/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f9055)
#3 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d31)
Change-Id: I316fb1a845fbee09f634d39e64057c170fab4795
Reviewed-on: http://review.couchbase.org/62974
Well-Formed: buildbot <[email protected]>
Reviewed-by: Will Gardner <[email protected]>
Tested-by: buildbot <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Jul 28, 2016
This value is accessed without a lock from addWorkerStats. Change to
atomic to fix race.
As reported by ThreadSanitizer:
WARNING: ThreadSanitizer: data race (pid=18761)
Read of size 8 at 0x7d4400007fa8 by main thread (mutexes: write M19371):
#0 ExecutorThread::getCurTime() /home/couchbase/server/ep-engine/src/executorthread.h:129:46 (ep.so+0x0000000e67a0)
#1 addWorkerStats(char const*, ExecutorThread*, void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/server/ep-engine/src/executorpool.cc:748 (ep.so+0x0000000e67a0)
#2 ExecutorPool::doWorkerStat(EventuallyPersistentEngine*, void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/server/ep-engine/src/executorpool.cc:760 (ep.so+0x0000000e67a0)
#3 EventuallyPersistentEngine::doDispatcherStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/server/ep-engine/src/ep_engine.cc:4352:5 (ep.so+0x0000000bc72d)
#4 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/server/ep-engine/src/ep_engine.cc:4588 (ep.so+0x0000000bc72d)
#5 EvpGetStats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/server/ep-engine/src/ep_engine.cc:214:38 (ep.so+0x0000000ab70e)
#6 mock_get_stats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/server/memcached/programs/engine_testapp/engine_testapp.cc:239:19 (engine_testapp+0x0000004c553d)
#7 test_worker_stats(engine_interface*, engine_interface_v1*) /home/couchbase/server/ep-engine/tests/ep_testsuite.cc:8901:24 (ep_testsuite.so+0x000000039908)
#8 execute_test(test, char const*, char const*) /home/couchbase/server/memcached/programs/engine_testapp/engine_testapp.cc:1090:19 (engine_testapp+0x0000004c4142)
#9 main /home/couchbase/server/memcached/programs/engine_testapp/engine_testapp.cc:1439 (engine_testapp+0x0000004c4142)
Previous write of size 8 at 0x7d4400007fa8 by thread T7 (mutexes: write M12645):
#0 TaskQueue::_doSleep(ExecutorThread&) /home/couchbase/server/ep-engine/src/taskqueue.cc:78:5 (ep.so+0x00000012eb21)
#1 TaskQueue::_fetchNextTask(ExecutorThread&, bool) /home/couchbase/server/ep-engine/src/taskqueue.cc:117:21 (ep.so+0x00000012ed66)
#2 TaskQueue::fetchNextTask(ExecutorThread&, bool) /home/couchbase/server/ep-engine/src/taskqueue.cc:160:17 (ep.so+0x00000012f907)
#3 ExecutorPool::_nextTask(ExecutorThread&, unsigned char) /home/couchbase/server/ep-engine/src/executorpool.cc:226:17 (ep.so+0x0000000dfa6f)
#4 ExecutorPool::nextTask(ExecutorThread&, unsigned char) /home/couchbase/server/ep-engine/src/executorpool.cc:241:21 (ep.so+0x0000000dfac6)
#5 ExecutorThread::run() /home/couchbase/server/ep-engine/src/executorthread.cc:81:28 (ep.so+0x0000000e9cfe)
#6 launch_executor_thread(void*) /home/couchbase/server/ep-engine/src/executorthread.cc:33:9 (ep.so+0x0000000e9b05)
#7 platform_thread_wrap /home/couchbase/server/platform/src/cb_pthreads.c:23:5 (libplatform.so.0.1.0+0x000000003dc1)
Change-Id: Idcaea9a157293dbf95ca236354673556a2f3c4ac
Reviewed-on: http://review.couchbase.org/62975
Well-Formed: buildbot <[email protected]>
Reviewed-by: Jim Walker <[email protected]>
Reviewed-by: Will Gardner <[email protected]>
Tested-by: buildbot <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Jul 28, 2016
The detected usage is just in stats, however there is an indirect
usage of this variable (persistenceQueueSmallEnough, via
stats.replicationThrottleWriteQueueCap) which /might/ result in an
incorrect queue size being used.
WARNING: ThreadSanitizer: data race (pid=31345)
Read of size 8 at 0x7d08000066c0 by thread T12:
#0 ReplicationThrottle::adjustWriteQueueCap(unsigned long) ep-engine/src/replicationthrottle.cc:49 (ep.so+0x000000114c7f)
#1 EventuallyPersistentEngine::doEngineStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) ep-engine/src/ep_engine.cc:3100 (ep.so+0x0000000bb424)
#2 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) ep-engine/src/ep_engine.cc:4597 (ep.so+0x0000000c54d5)
#3 EventuallyPersistentStore::snapshotStats() ep-engine/src/ep.cc:1744 (ep.so+0x000000090f2e)
#4 StatSnap::run() ep-engine/src/tasks.cc:100 (ep.so+0x000000136dd6)
#5 ExecutorThread::run() ep-engine/src/executorthread.cc:115 (ep.so+0x0000000f6966)
#6 launch_executor_thread(void*) ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f6515)
#7 platform_thread_wrap(void*) platform/src/cb_pthreads.cc:54 (libplatform.so.0.1.0+0x00000000551b)
12:23:35
Previous write of size 8 at 0x7d08000066c0 by main thread (mutexes: write M2287262534014660504):
#0 EPStoreValueChangeListener::sizeValueChanged(std::string const&, unsigned long) ep-engine/src/replicationthrottle.h:42 (ep.so+0x0000000b35ee)
#1 Configuration::setParameter(std::string const&, long) ep-engine/src/configuration.cc:225 (ep.so+0x000000197ac5)
#2 Configuration::setReplicationThrottleQueueCap(long const&) build/ep-engine/src/generated_configuration.cc:459 (ep.so+0x0000001a6fa8)
#3 setTapParam(EventuallyPersistentEngine*, char const*, char const*, std::string&) ep-engine/src/ep_engine.cc:323 (ep.so+0x0000000d7f3f)
#4 EvpUnknownCommand(engine_interface*, void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) ep-engine/src/ep_engine.cc:1365 (ep.so+0x0000000b4f68)
#5 mock_unknown_command(engine_interface*, void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) memcached/programs/engine_testapp/engine_testapp.cc:382 (engine_testapp+0x0000004ce149)
#6 set_param(engine_interface*, engine_interface_v1*, protocol_binary_engine_param_t, char const*, char const*) ep-engine/tests/ep_test_apis.cc:597 (ep_testsuite_dcp.so+0x000000038e17)
#7 test_consumer_backoff_stat(engine_interface*, engine_interface_v1*) ep-engine/tests/ep_testsuite_dcp.cc:2184 (ep_testsuite_dcp.so+0x000000017411)
#8 execute_test(test, char const*, char const*) memcached/programs/engine_testapp/engine_testapp.cc:1131 (engine_testapp+0x0000004cc600)
#9 __libc_start_main /build/buildd/eglibc-2.15/csu/libc-start.c:226 (libc.so.6+0x00000002176c)
Change-Id: Ie4ff039603f2ddfc5b44d5d7f217544307655d31
Reviewed-on: http://review.couchbase.org/62976
Well-Formed: buildbot <[email protected]>
Reviewed-by: Will Gardner <[email protected]>
Tested-by: buildbot <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Jul 28, 2016
WARNING: ThreadSanitizer: data race (pid=20056)
Write of size 8 at 0x7d600000f038 by main thread (mutexes: write M1412):
#0 ConnHandler::setCookie(void const*) /home/abhinav/couchbase/ep-engine/src/tapconnection.h:344 (ep.so+0x000000042367)
#1 EventuallyPersistentEngine::createTapQueue(void const*, std::string&, unsigned int, void const*, unsigned long) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:2655 (ep.so+0x0000000b86da)
#2 EvpGetTapIterator(engine_interface*, void const*, void const*, unsigned long, unsigned int, void const*, unsigned long) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:1462 (ep.so+0x0000000b46a3)
#3 mock_get_tap_iterator(engine_interface*, void const*, void const*, unsigned long, unsigned int, void const*, unsigned long) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:467 (engine_testapp+0x0000000bae3e)
#4 test_tap_ack_stream(engine_interface*, engine_interface_v1*) /home/abhinav/couchbase/ep-engine/tests/ep_testsuite.cc:7341 (ep_testsuite.so+0x000000050416)
#5 execute_test(test, char const*, char const*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1090 (engine_testapp+0x0000000b946c)
#6 __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287 (libc.so.6+0x000000021ec4)
Previous read of size 8 at 0x7d600000f038 by thread T9 (mutexes: write M1411):
#0 ConnHandler::getCookie() const /home/abhinav/couchbase/ep-engine/src/tapconnection.h:340 (ep.so+0x00000004067c)
#1 bool TapConnMap::performOp<Item*>(std::string const&, TapOperation<Item*>&, Item*) /home/abhinav/couchbase/ep-engine/src/connmap.h:389 (ep.so+0x00000001fa08)
#2 ItemResidentCallback::callback(CacheLookup&) /home/abhinav/couchbase/ep-engine/src/backfill.cc:63 (ep.so+0x00000001d9ca)
#3 CouchKVStore::recordDbDump(_db*, _docinfo*, void*) /home/abhinav/couchbase/ep-engine/src/couch-kvstore/couch-kvstore.cc:1654 (ep.so+0x000000180ca0)
#4 recordDbDumpC(_db*, _docinfo*, void*) /home/abhinav/couchbase/ep-engine/src/couch-kvstore/couch-kvstore.cc:66 (ep.so+0x00000017fe95)
#5 lookup_callback(couchfile_lookup_request*, _sized_buf const*, _sized_buf const*) /home/abhinav/couchbase/couchstore/src/couch_db.cc:767 (libcouchstore.so+0x00000000d7e5)
#6 btree_lookup_inner(couchfile_lookup_request*, unsigned long, int, int) /home/abhinav/couchbase/couchstore/src/btree_read.cc:99 (libcouchstore.so+0x00000000b5a2)
#7 btree_lookup /home/abhinav/couchbase/couchstore/src/btree_read.cc:131 (libcouchstore.so+0x00000000affc)
#8 couchstore_changes_since /home/abhinav/couchbase/couchstore/src/couch_db.cc:812 (libcouchstore.so+0x00000000d5f1)
#9 CouchKVStore::scan(ScanContext*) /home/abhinav/couchbase/ep-engine/src/couch-kvstore/couch-kvstore.cc:1264 (ep.so+0x00000017f94e)
#10 BackfillDiskLoad::run() /home/abhinav/couchbase/ep-engine/src/backfill.cc:131 (ep.so+0x00000001e449)
#11 ExecutorThread::run() /home/abhinav/couchbase/ep-engine/src/executorthread.cc:112 (ep.so+0x0000000f8956)
#12 launch_executor_thread(void*) /home/abhinav/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f84f5)
#13 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d31)
Change-Id: I8a668f17013c95abc9786d853ed2c6462cae5320
Reviewed-on: http://review.couchbase.org/62978
Well-Formed: buildbot <[email protected]>
Reviewed-by: Will Gardner <[email protected]>
Tested-by: buildbot <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Jul 28, 2016
WARNING: ThreadSanitizer: data race (pid=139161)
Read of size 4 at 0x7d480000b150 by thread T31 (mutexes: write M51120):
#0 DCPBackfill::scan() ep-engine/src/dcp/stream.h:126 (ep.so+0x000000053391)
#1 DCPBackfill::run() ep-engine/src/dcp/backfill.cc:118 (ep.so+0x000000052737)
#2 BackfillManager::backfill() ep-engine/src/dcp/backfill-manager.cc:240 (ep.so+0x00000004cf65)
#3 BackfillManagerTask::run() ep-engine/src/dcp/backfill-manager.cc:43 (ep.so+0x00000004cb8f)
#4 ExecutorThread::run() ep-engine/src/executorthread.cc:115 (ep.so+0x0000000eb94d)
#5 launch_executor_thread(void*) ep-engine/src/executorthread.cc:33 (ep.so+0x0000000eb515)
#6 platform_thread_wrap(void*) platform/src/cb_pthreads.cc:53 (libplatform.so.0.1.0+0x0000000048ab)
Previous write of size 4 at 0x7d480000b150 by main thread (mutexes: write M1241, write M32448, write M51071, write M51087):
#0 ActiveStream::transitionState(stream_state_t) ep-engine/src/dcp/stream.cc:829 (ep.so+0x00000006accb)
#1 ActiveStream::endStream(end_stream_status_t) ep-engine/src/dcp/stream.cc:688 (ep.so+0x00000006a8c2)
#2 ActiveStream::setDead(end_stream_status_t) ep-engine/src/dcp/stream.cc:654 (ep.so+0x00000006f27b)
#3 DcpProducer::setDisconnect(bool) ep-engine/src/dcp/producer.cc:835 (ep.so+0x000000065605)
#4 DcpConnMap::disconnect_UNLOCKED(void const*) ep-engine/src/connmap.cc:1116 (ep.so+0x000000045d6c)
#5 DcpConnMap::disconnect(void const*) ep-engine/src/connmap.cc:1109 (ep.so+0x000000045c8b)
#6 EventuallyPersistentEngine::handleDisconnect(void const*) ep-engine/src/ep_engine.cc:6265 (ep.so+0x0000000ca38a)
#7 EvpHandleDisconnect(void const*, ENGINE_EVENT_TYPE, void const*, void const*) ep-engine/src/ep_engine.cc:1802 (ep.so+0x0000000af976)
#8 destroy_mock_cookie memcached/programs/engine_testapp/mock_server.cc:325 (engine_testapp+0x0000004f4082)
#9 dcp_stream_req(engine_interface*, engine_interface_v1*, unsigned int, unsigned short, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, ENGINE_ERROR_CODE) ep-engine/tests/ep_testsuite.cc:4331 (ep_testsuite.so+0x000000090b06)
#10 test_failover_log_dcp(engine_interface*, engine_interface_v1*) ep-engine/tests/ep_testsuite.cc:14127 (ep_testsuite.so+0x00000007ce7a)
#11 execute_test(test, char const*, char const*) memcached/programs/engine_testapp/engine_testapp.cc:1091 (engine_testapp+0x0000004cb315)
#12 __libc_start_main /build/buildd/eglibc-2.15/csu/libc-start.c:226 (libc.so.6+0x00000002176c)
Change-Id: Icfc82fa877999d128184c9cac8f8c0e1cafc4e67
Reviewed-on: http://review.couchbase.org/63027
Well-Formed: buildbot <[email protected]>
Reviewed-by: Will Gardner <[email protected]>
Tested-by: buildbot <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Jul 28, 2016
WARNING: ThreadSanitizer: data race (pid=236996)
Write of size 8 at 0x7d380000dbd8 by thread T5:
#0 ExecutorThread::run() ep-engine/src/executorthread.cc:105 (ep.so+0x0000000ee0cc)
#1 launch_executor_thread(void*) ep-engine/src/executorthread.cc:33 (ep.so+0x0000000edd75)
#2 platform_thread_wrap(void*) platform/src/cb_pthreads.cc:54 (libplatform.so.0.1.0+0x000000004e7b)
Previous read of size 8 at 0x7d380000dbd8 by main thread (mutexes: write M266835151185571736):
#0 addWorkerStats(char const*, ExecutorThread*, void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) ep-engine/src/executorthread.h:108 (ep.so+0x0000000ea8dc)
#1 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) ep-engine/src/ep_engine.cc:4363 (ep.so+0x0000000c05dd)
#2 EvpGetStats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) ep-engine/src/ep_engine.cc:219 (ep.so+0x0000000af40e)
#3 mock_get_stats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) memcached/programs/engine_testapp/engine_testapp.cc:240 (engine_testapp+0x0000004cc71d)
#4 test_worker_stats(engine_interface*, engine_interface_v1*) ep-engine/tests/ep_testsuite.cc:9464 (ep_testsuite.so+0x00000003c038)
#5 execute_test(test, char const*, char const*) memcached/programs/engine_testapp/engine_testapp.cc:1091 (engine_testapp+0x0000004cb315)
#6 __libc_start_main /build/buildd/eglibc-2.15/csu/libc-start.c:226 (libc.so.6+0x00000002176c)
Change-Id: If01aba3cf6b3591f19c5bb62119e40998f12c8ff
Reviewed-on: http://review.couchbase.org/63028
Well-Formed: buildbot <[email protected]>
Reviewed-by: Jim Walker <[email protected]>
Reviewed-by: Will Gardner <[email protected]>
Tested-by: buildbot <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Jul 28, 2016
As identified by ThreadSanitizer:
WARNING: ThreadSanitizer: data race (pid=17259)
Write of size 8 at 0x7fec86e44de0 by main thread (mutexes: write M1161):
#0 gmtime ??:0 (libtsan.so.0+0x000000025135)
#1 EventuallyPersistentEngine::doEngineStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:3369 (ep.so+0x00000010f4be)
#2 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:4339 (ep.so+0x000000113c35)
#3 EvpGetStats /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:217 (ep.so+0x000000102b14)
#4 mock_get_stats /repos/couchbase/server/source/memcached/programs/engine_testapp/engine_testapp.c:195 (exe+0x0000000026de)
#5 get_int_stat(engine_interface*, engine_interface_v1*, char const*, char const*) /repos/couchbase/server/source/ep-engine/tests/ep_test_apis.cc:799 (ep_testsuite.so+0x0000000832d8)
#6 wait_for_stat_change(engine_interface*, engine_interface_v1*, char const*, int, char const*) /repos/couchbase/server/source/ep-engine/tests/ep_test_apis.cc:846 (ep_testsuite.so+0x0000000838d6)
#7 test_setup /repos/couchbase/server/source/ep-engine/tests/ep_testsuite.cc:178 (ep_testsuite.so+0x00000001ee69)
#8 execute_test /repos/couchbase/server/source/memcached/programs/engine_testapp/engine_testapp.c:1050 (exe+0x000000005970)
#9 main /repos/couchbase/server/source/memcached/programs/engine_testapp/engine_testapp.c:1313 (exe+0x000000006606)
Previous write of size 8 at 0x7fec86e44de0 by thread T7:
#0 gmtime ??:0 (libtsan.so.0+0x000000025135)
#1 EventuallyPersistentEngine::doEngineStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:3369 (ep.so+0x00000010f4be)
#2 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:4339 (ep.so+0x000000113c35)
#3 EventuallyPersistentStore::snapshotStats() /repos/couchbase/server/source/ep-engine/src/ep.cc:1465 (ep.so+0x0000000e150e)
#4 StatSnap::run() /repos/couchbase/server/source/ep-engine/src/tasks.cc:79 (ep.so+0x000000174db2)
#5 ExecutorThread::run() /repos/couchbase/server/source/ep-engine/src/executorthread.cc:110 (ep.so+0x00000014a0e7)
#6 launch_executor_thread /repos/couchbase/server/source/ep-engine/src/executorthread.cc:34 (ep.so+0x000000149930)
#7 platform_thread_wrap /repos/couchbase/server/source/platform/src/cb_pthreads.c:19 (libplatform.so.0.1.0+0x000000002d8b)
#8 __tsan_write_range ??:0 (libtsan.so.0+0x00000001b1c9)
Switch to gmtime_r which is thread-safe.
Change-Id: Id0773df65f4fc569c0a173b6185b1ef8bd91862d
Reviewed-on: http://review.couchbase.org/63080
Well-Formed: buildbot <[email protected]>
Reviewed-by: Jim Walker <[email protected]>
Reviewed-by: Will Gardner <[email protected]>
Tested-by: buildbot <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Jul 28, 2016
…t lock
+ [Not a backport, this code was altered/removed in master]
+ This scenario should not occur in real operation.
+ Most DCP unit tests would however flag this as an issue because
of how we do things in the tests --> [setting vbucket's state to
replica at the very beginning (by the main thread)].
+ Suppressing this lock inversion, by moving the function call to
update the vbucket's snapshot range to outside the state lock
context in setState(), as it isn't necessary to acquire the state
lock to update the snapshot range.
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=39750)
Cycle in lock order graph: M43306 (0x7d640000fcf0) => M43309 (0x7d640000fe18) => M43306
Mutex M43309 acquired here while holding mutex M43306 in main thread:
#0 pthread_mutex_lock <null> (engine_testapp+0x000000474420)
#1 cb_mutex_enter /home/daver/repos/couchbase/server/platform/src/cb_pthreads.c:85 (libplatform.so.0.1.0+0x0000000034a0)
#2 Mutex::acquire() /home/daver/repos/couchbase/server/ep-engine/src/mutex.cc:31 (ep.so+0x0000001c611e)
#3 LockHolder::lock() /home/daver/repos/couchbase/server/ep-engine/src/locks.h:71 (ep.so+0x00000006a4e3)
#4 LockHolder /home/daver/repos/couchbase/server/ep-engine/src/locks.h:48 (ep.so+0x00000006a172)
#5 VBucket::setCurrentSnapshot(unsigned long, unsigned long) /home/daver/repos/couchbase/server/ep-engine/src/vbucket.h:217 (ep.so+0x0000000e5ee5)
#6 VBucket::setState(vbucket_state_t, server_handle_v1_t*) /home/daver/repos/couchbase/server/ep-engine/src/vbucket.cc:196 (ep.so+0x0000002932e9)
#7 EventuallyPersistentStore::setVBucketState(unsigned short, vbucket_state_t, bool, bool) /home/daver/repos/couchbase/server/ep-engine/src/ep.cc:1060 (ep.so+0x0000000c0b61)
#8 EventuallyPersistentEngine::setVBucketState(unsigned short, vbucket_state_t, bool) /home/daver/repos/couchbase/server/ep-engine/src/ep_engine.h:628 (ep.so+0x000000188a12)
#9 setVBucket(EventuallyPersistentEngine*, void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/daver/repos/couchbase/server/ep-engine/src/ep_engine.cc:824 (ep.so+0x00000014aaaa)
#10 processUnknownCommand(EventuallyPersistentEngine*, void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/daver/repos/couchbase/server/ep-engine/src/ep_engine.cc:1118 (ep.so+0x000000147707)
#11 EvpUnknownCommand(engine_interface*, void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/daver/repos/couchbase/server/ep-engine/src/ep_engine.cc:1312 (ep.so+0x00000011a055)
#12 mock_unknown_command /home/daver/repos/couchbase/server/memcached/programs/engine_testapp/engine_testapp.c:335 (engine_testapp+0x0000004be97a)
#13 set_vbucket_state(engine_interface*, engine_interface_v1*, unsigned short, vbucket_state_t) /home/daver/repos/couchbase/server/ep-engine/tests/ep_test_apis.cc:484 (ep_testsuite.so+0x0000000e1562)
#14 test_dcp_replica_stream_backfill(engine_interface*, engine_interface_v1*) /home/daver/repos/couchbase/server/ep-engine/tests/ep_testsuite.cc:5278 (ep_testsuite.so+0x00000008c84a)
#15 execute_test /home/daver/repos/couchbase/server/memcached/programs/engine_testapp/engine_testapp.c:1042 (engine_testapp+0x0000004ba8ff)
#16 main /home/daver/repos/couchbase/server/memcached/programs/engine_testapp/engine_testapp.c:1296 (engine_testapp+0x0000004b8861)
Mutex M43306 acquired here while holding mutex M43309 in thread T17:
#0 pthread_rwlock_rdlock <null> (engine_testapp+0x000000457ca0)
#1 cb_rw_reader_enter /home/daver/repos/couchbase/server/platform/src/cb_pthreads.c:264 (libplatform.so.0.1.0+0x0000000042e0)
#2 RWLock::readerLock() /home/daver/repos/couchbase/server/ep-engine/src/rwlock.h:38 (ep.so+0x000000115cf0)
#3 ReaderLockHolder /home/daver/repos/couchbase/server/ep-engine/src/locks.h:167 (ep.so+0x0000000dbbe7)
#4 EventuallyPersistentStore::addTAPBackfillItem(Item const&, unsigned char, bool) /home/daver/repos/couchbase/server/ep-engine/src/ep.cc:851 (ep.so+0x0000000be35d)
#5 PassiveStream::commitMutation(MutationResponse*, bool) /home/daver/repos/couchbase/server/ep-engine/src/dcp-stream.cc:1370 (ep.so+0x00000027e8cc)
#6 PassiveStream::processMutation(MutationResponse*) /home/daver/repos/couchbase/server/ep-engine/src/dcp-stream.cc:1346 (ep.so+0x00000027d680)
#7 PassiveStream::processBufferedMessages(unsigned int&) /home/daver/repos/couchbase/server/ep-engine/src/dcp-stream.cc:1286 (ep.so+0x00000027cfbc)
#8 DcpConsumer::processBufferedItems() /home/daver/repos/couchbase/server/ep-engine/src/dcp-consumer.cc:599 (ep.so+0x0000002454cc)
#9 Processer::run() /home/daver/repos/couchbase/server/ep-engine/src/dcp-consumer.cc:48 (ep.so+0x0000002450ef)
#10 ExecutorThread::run() /home/daver/repos/couchbase/server/ep-engine/src/executorthread.cc:109 (ep.so+0x0000001c76d9)
#11 launch_executor_thread(void*) /home/daver/repos/couchbase/server/ep-engine/src/executorthread.cc:34 (ep.so+0x0000001c6caa)
#12 platform_thread_wrap /home/daver/repos/couchbase/server/platform/src/cb_pthreads.c:19 (libplatform.so.0.1.0+0x00000000325c)
Change-Id: I2f3cf88e6aebbf6078f533fb1ed87bd9fe618616
Reviewed-on: http://review.couchbase.org/63319
Well-Formed: buildbot <[email protected]>
Reviewed-by: Dave Rigby <[email protected]>
Tested-by: buildbot <[email protected]>
Reviewed-by: Will Gardner <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Jul 28, 2016
…t lock
+ [Not a backport, this code was altered/removed in master]
+ Address this lock inversion by moving reading the vbucket snapshot
range to outside the vbucket's state lock context.
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=245522)
Cycle in lock order graph: M21518 (0x7d640003e220) => M21515 (0x7d640003e0f0) => M21518
Mutex M21515 acquired here while holding mutex M21518 in thread T17:
#0 pthread_rwlock_rdlock <null> (engine_testapp+0x000000462260)
#1 cb_rw_reader_enter <null> (libplatform.so.0.1.0+0x000000004800)
#2 RWLock::readerLock() ep-engine/src/rwlock.h:38 (ep.so+0x000000132360)
#3 ReaderLockHolder::ReaderLockHolder(RWLock&) ep-engine/src/locks.h:167 (ep.so+0x0000000f8087)
#4 EventuallyPersistentStore::addTAPBackfillItem(Item const&, unsigned char, bool) ep-engine/src/ep.cc:851 (ep.so+0x0000000d9ba7)
#5 PassiveStream::commitMutation(MutationResponse*, bool) ep-engine/src/dcp-stream.cc:1370 (ep.so+0x00000029dd8c)
#6 PassiveStream::processMutation(MutationResponse*) ep-engine/src/dcp-stream.cc:1346 (ep.so+0x00000029cbd0)
#7 PassiveStream::processBufferedMessages(unsigned int&) ep-engine/src/dcp-stream.cc:1286 (ep.so+0x00000029c522)
#8 DccpConsumer::processBufferedItems() ep-engine/src/dcp-consumer.cc:599 (ep.so+0x000000262e04)
#9 Processer::run() ep-engine/src/dcp-consumer.cc:48 (ep.so+0x0000002629ff)
#10 ExecutorThread::run() ep-engine/src/executorthread.cc:109 (ep.so+0x0000001e38f1)
#11 launch_executor_thread(void*) ep-engine/src/executorthread.cc:34 (ep.so+0x0000001e2f1a)
#12 platform_thread_wrap platform/src/cb_pthreads.c (libplatform.so.0.1.0+0x00000000377c)
Mutex M21518 acquired here while holding mutex M21515 in main thread:
#0 pthread_mutex_lock <null> (engine_testapp+0x00000047e9e0)
#1 cb_mutex_enter <null> (libplatform.so.0.1.0+0x0000000039c0)
#2 Mutex::acquire() ep-engine/src/mutex.cc:31 (ep.so+0x0000001e241e)
#3 LockHolder::lock() ep-engine/src/locks.h:71 (ep.so+0x000000080bc3)
#4 LockHolder::LockHolder(Mutex&, bool) ep-engine/src/locks.h:48 (ep.so+0x000000080832)
#5 VBucket::getCurrentSnapshot(unsigned long&, unsigned long&) ep-engine/src/vbucket.h:233 (ep.so+0x0000000fae05)
#6 EventuallyPersistentEngine::doSeqnoStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), char const*, int) ep-engine/src/ep_engine.cc:4255 (ep.so+0x00000014f202)
#7 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) ep-engine/src/ep_engine.cc:4372 (ep.so+0x000000150bcb)
#8 EvpGetStats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) ep-engine/src/ep_engine.cc:214 (ep.so+0x000000136a72)
#9 mock_get_stats memcached/programs/engine_testapp/engine_testapp.c (engine_testapp+0x0000004c8403)
#10 get_int_stat(engine_interface*, engine_interface_v1*, char const*, char const*) ep-engine/tests/ep_test_apis.cc:771 (ep_testsuite.so+0x0000000e21ea)
#11 wait_for_stat_to_be(engine_interface*, engine_interface_v1*, char const*, int, char const*) ep-engine/tests/ep_test_apis.cc:860 (ep_testsuite.so+0x0000000e8d2b)
#12 test_dcp_replica_stream_backfill(engine_interface*, engine_interface_v1*) ep-engine/tests/ep_testsuite.cc:5306 (ep_testsuite.so+0x00000008e601)
#13 execute_test memcached/programs/engine_testapp/engine_testapp.c (engine_testapp+0x0000004c4e9f)
#14 main crtstuff.c (engine_testapp+0x0000004c2e01)
Change-Id: Ia4dd34ab152d1cc1d1658ebe957da7c3b8d32c06
Reviewed-on: http://review.couchbase.org/63366
Well-Formed: buildbot <[email protected]>
Tested-by: buildbot <[email protected]>
Reviewed-by: Will Gardner <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Aug 2, 2016
As identified by TSan. Seen whilst testing sherlock->watson merge,
analysed the code and it seems this is a latent issue and hard to
re-produce.
The issue is that when the executor thread does a reset on the current
task, the VBCBAdapator is the last one holding the ref-counted
vbucket, so destruction occurs and ~VBucket calls the destructor of
the checkpoint manager, which is the reverse locks ordering of a
previous code path.
Number of threads in play here, but main ones of interest:
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=170834)
Cycle in lock order graph: M_checkpoint (0x7d640002e9a8) => M_exepool (0x7d4c00008288) => M_taskqueue (0x7d4400008e80) => M_exethread (0x7d380000df60) => M_checkpoint
Mutex M_exepool acquired here while holding mutex M_checkpoint in thread T35:
#0 pthread_mutex_lock <null> (engine_testapp+0x000000486760)
#1 std::mutex::lock() /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/x86_64-linux-gnu/c++/4.9/bits/gthr-default.h:748 (ep.so+0x0000000f47b0)
#2 ExecutorPool::wake(unsigned long) ep-engine/src/executorpool.cc:355 (ep.so+0x0000000f48f1)
#3 Flusher::wake() ep-engine/src/flusher.cc:155 (ep.so+0x000000101ee6)
#4 NotifyFlusherCB::callback(unsigned short&) ep-engine/src/flusher.h:88 (ep.so+0x00000010d194)
#5 Checkpoint::queueDirty(SingleThreadedRCPtr<Item> const&, CheckpointManager*) ep-engine/src/checkpoint.h:675 (ep.so+0x0000000271b0)
#6 CheckpointManager::closeOpenCheckpoint_UNLOCKED() ep-engine/src/checkpoint.cc:454 (ep.so+0x000000028dcb)
#7 CheckpointManager::addNewCheckpoint_UNLOCKED(unsigned long, unsigned long, unsigned long) ep-engine/src/checkpoint.cc:371 (ep.so+0x00000002881f)
#8 CheckpointManager::checkOpenCheckpoint_UNLOCKED(bool, bool) ep-engine/src/checkpoint.cc:361 (ep.so+0x00000002bd71)
#9 CheckpointVisitor::visitBucket(RCPtr<VBucket>&) ep-engine/src/checkpoint_remover.cc:43 (ep.so+0x00000003c3bd)
#10 VBCBAdaptor::run() ep-engine/src/ep.cc:3924 (ep.so+0x0000000a6174)
#11 ExecutorThread::run() ep-engine/src/executorthread.cc:115 (ep.so+0x0000000fe1b6)
#12 launch_executor_thread(void*) ep-engine/src/executorthread.cc:33 (ep.so+0x0000000fdd15)
#13 platform_thread_wrap(void*) platform/src/cb_pthreads.cc:54 (libplatform.so.0.1.0+0x0000000057fb)
...
Mutex M_checkpoint acquired here while holding mutex M_exethread in thread T36:
#0 pthread_mutex_lock <null> (engine_testapp+0x000000486760)
#1 CheckpointManager::~CheckpointManager() /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/x86_64-linux-gnu/c++/4.9/bits/gthr-default.h:748 (ep.so+0x000000027fdd)
#2 VBucket::~VBucket() ep-engine/src/vbucket.cc:152 (ep.so+0x00000014a018)
#3 PagingVisitor::~PagingVisitor() ep-engine/src/atomic.h:190 (ep.so+0x00000010a5e6)
#4 PagingVisitor::~PagingVisitor() ep-engine/src/item_pager.cc:43 (ep.so+0x00000010a645)
#5 std::_Sp_counted_ptr<PagingVisitor*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/shared_ptr_base.h:373 (ep.so+0x00000010a2b0)
#6 VBCBAdaptor::~VBCBAdaptor() /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/shared_ptr_base.h:149 (ep.so+0x0000000aea7e)
#7 ExecutorThread::run() ep-engine/src/atomic.h:325 (ep.so+0x0000000fdee4)
#8 launch_executor_thread(void*) ep-engine/src/executorthread.cc:33 (ep.so+0x0000000fdd15)
#9 platform_thread_wrap(void*) platform/src/cb_pthreads.cc:54 (libplatform.so.0.1.0+0x0000000057fb)
Change-Id: I0a966b3d112963243e17647184123fd8b3200656
Reviewed-on: http://review.couchbase.org/66357
Well-Formed: buildbot <[email protected]>
Tested-by: buildbot <[email protected]>
Reviewed-by: Will Gardner <[email protected]>
Reviewed-by: Jim Walker <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Aug 2, 2016
…ker threads Fix the thread sanitizer warning http://cv.jenkins.couchbase.com/job/ep-engine-threadsanitizer-3.0.x/258/console WARNING: ThreadSanitizer: data race (pid=102290) Read of size 4 at 0x7d580000f71c by thread T14 (mutexes: write M969): #0 void ConnHandler::addStat<unsigned int>(char const*, unsigned int const&, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) const /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/tapconnection.h:294 (ep.so+0x00000020e9f3) #1 DcpProducer::addStats(void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/dcp-producer.cc:557 (ep.so+0x00000027fbe5) #2 ConnStatBuilder::operator()(SingleThreadedRCPtr<ConnHandler>&) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep_engine.cc:3701 (ep.so+0x000000183c44) #3 ConnStatBuilder std::for_each<std::_List_iterator<SingleThreadedRCPtr<ConnHandler> >, ConnStatBuilder>(std::_List_iterator<SingleThreadedRCPtr<ConnHandler> >, std::_List_iterator<SingleThreadedRCPtr<ConnHandler> >, ConnStatBuilder) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/stl_algo.h:3755 (ep.so+0x0000001838a5) #4 void ConnMap::each_UNLOCKED<ConnStatBuilder>(ConnStatBuilder) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/connmap.h:148 (ep.so+0x000000183808) #5 void ConnMap::each<ConnStatBuilder>(ConnStatBuilder) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/connmap.h:140 (ep.so+0x00000017732e) #6 EventuallyPersistentEngine::doDcpStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep_engine.cc:3954 (ep.so+0x00000014c5ed) Previous write of size 4 at 0x7d580000f71c by main thread: #0 DcpProducer::maybeSendNoop(dcp_message_producers*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/dcp-producer.cc:740 (ep.so+0x00000027d8ce) #1 DcpProducer::step(dcp_message_producers*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/dcp-producer.cc:323 (ep.so+0x00000027c920) #2 EvpDcpStep(engine_interface*, void const*, dcp_message_producers*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep_engine.cc:1404 (ep.so+0x000000138baa) Change-Id: I2a2b0b0f01b10ecb31701bfc2330881bbafc6b74 Reviewed-on: http://review.couchbase.org/64959 Well-Formed: buildbot <[email protected]> Tested-by: buildbot <[email protected]> Reviewed-by: Dave Rigby <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Oct 17, 2016
Address race condition identified by ThreadSanitizer on
BloomFilter::keyCounter.
WARNING: ThreadSanitizer: data race (pid=341)
Write of size 8 at 0x7d140000df80 by thread T11 (mutexes: write M20582, write M20817, write M19369):
#0 BloomFilter::addKey() ep-engine/src/bloomfilter.cc:119 (ep.so+0x00000002453d)
#1 VBucket::addToFilter() ep-engine/src/vbucket.cc:482 (ep.so+0x000000149704)
#2 PersistenceCallback::callback() ep-engine/src/ep.cc:3211 (ep.so+0x0000000b27c1)
#3 non-virtual thunk to PersistenceCallback::callback(int&) ep-engine/src/ep.cc:3167 (ep.so+0x0000000b36c2)
#4 CouchKVStore::commitCallback() ep-engine/src/couch-kvstore/couch-kvstore.cc:1898 (ep.so+0x00000017aa51)
#5 _ZN12CouchKVStore17commit2couchstoreEP8CallbackIJ10KVStatsCtxEE ep-engine/src/couch-kvstore/couch-kvstore.cc:1712 (ep.so+0x00000017208b)
#6 _ZN12CouchKVStore6commitEP8CallbackIJ10KVStatsCtxEE ep-engine/src/couch-kvstore/couch-kvstore.cc:954 (ep.so+0x000000171c92)
#7 EventuallyPersistentStore::commit(unsigned short) ep-engine/src/ep.cc:3451 (ep.so+0x0000000a29e6)
#8 EventuallyPersistentStore::flushVBucket(unsigned short) ep-engine/src/ep.cc:3399 (ep.so+0x0000000a1935)
#9 Flusher::flushVB() ep-engine/src/flusher.cc:293 (ep.so+0x0000001038ae)
#10 Flusher::step(GlobalTask*) ep-engine/src/flusher.cc:183 (ep.so+0x000000101f7e)
#11 FlusherTask::run() ep-engine/src/tasks.cc:138 (ep.so+0x00000013ac42)
#12 ExecutorThread::run() ep-engine/src/executorthread.cc:115 (ep.so+0x0000000fd53f)
#13 launch_executor_thread(void*) ep-engine/src/executorthread.cc:33 (ep.so+0x0000000fd095)
#14 platform_thread_wrap(void*) platform/src/cb_pthreads.cc:55 (libplatform_so.so.0.1.0+0x000000005deb)
Previous read of size 8 at 0x7d140000df80 by main thread (mutexes: write M15538, write M1828880534441356728):
#0 BloomFilter::getNumOfKeysInFilter() ep-engine/src/bloomfilter.cc:142 (ep.so+0x0000000246a0)
#1 VBucket::addStats() ep-engine/src/vbucket.cc:597 (ep.so+0x00000014a527)
#2 EventuallyPersistentEngine::doVBucketStats() ep-engine/src/ep_engine.cc:3837 (ep.so+0x0000000c69f6)
#3 EventuallyPersistentEngine::doVBucketStats() ep-engine/src/ep_engine.cc:3861 (ep.so+0x0000000c664a)
#4 EventuallyPersistentEngine::getStats() ep-engine/src/ep_engine.cc:4774 (ep.so+0x0000000cbbb3)
#5 EvpGetStats() ep-engine/src/ep_engine.cc:231 (ep.so+0x0000000baafe)
#6 mock_get_stats() memcached/programs/engine_testapp/engine_testapp.cc:215 (engine_testapp+0x0000004ce40d)
#7 std::string get_stat<std::string>() ep-engine/tests/ep_test_apis.cc:1140 (ep_testsuite_xdcr.so+0x000000020c34)
#8 unsigned long get_stat<unsigned long>() ep-engine/tests/ep_test_apis.cc:1179 (ep_testsuite_xdcr.so+0x00000002089b)
#9 get_ull_stat() ep-engine/tests/ep_test_apis.cc:1174 (ep_testsuite_xdcr.so+0x00000001e73e)
#10 test_del_with_meta_and_check_drift_stats() ep-engine/tests/ep_testsuite_xdcr.cc:1711 (ep_testsuite_xdcr.so+0x0000000143f1)
#11 execute_test() memcached/programs/engine_testapp/engine_testapp.cc:1118 (engine_testapp+0x0000004cc997)
#12 __libc_start_main /build/buildd/eglibc-2.15/csu/libc-start.c:226 (libc.so.6+0x00000002176c)
Change-Id: I4e9a589e9286e72646df515db891c57b143d1fdd
Reviewed-on: http://review.couchbase.org/68803
Tested-by: buildbot <[email protected]>
Reviewed-by: Daniel Owen <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Oct 20, 2016
When developing additional stats test for ep-engine_perfsuite, the following data race was uncovered. The patch ensures that accesses to vals are protected with a mutex. Also get_stat and get_histo_stat can only be called one at a time as they use the three global variables (requested_stat_name, actual_stat_value and histogram_stat_int_value). Therefore a secondary mutex is required to enforce this. It has been confirmed that with this patch the ThreadSanitizer issue no longer occurs. 30: WARNING: ThreadSanitizer: data race (pid=42065) 30: Read of size 8 at 0x7d1800060a38 by thread T18 (mutexes: write M1704): 30: #0 std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_S_right(std::_Rb_tree_node_base*) <null> (engine_testapp+0x00000049c503) 30: #1 std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_M_lower_bound(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*, std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (engine_testapp+0x0000004a30db) 30: #2 std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::lower_bound(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (engine_testapp+0x00000049c5f6) 30: #3 std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::lower_bound(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (engine_testapp+0x000000499047) 30: #4 std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::operator[](std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (engine_testapp+0x00000048e3ba) 30: #5 add_stats /home/owend/master/ep-engine/tests/ep_test_apis.cc:236 (ep_perfsuite.so+0x00000008f5b9) 30: #6 STATWRITER_NAMESPACE::add_casted_stat(char const*, char const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/owend/master/ep-engine/src/statwriter.h:39 (ep.so+0x0000000e5673) 30: #7 StatCheckpointVisitor::addCheckpointStat(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), EventuallyPersistentStore*, RCPtr<VBucket>&) <null> (ep.so+0x0000001c120c) 30: #8 StatCheckpointVisitor::visitBucket(RCPtr<VBucket>&) <null> (ep.so+0x0000001c10e5) 30: #9 EventuallyPersistentStore::visit(VBucketVisitor&) /home/owend/master/ep-engine/src/ep.cc:3796 (ep.so+0x000000174559) 30: #10 StatCheckpointTask::run() <null> (ep.so+0x0000001c155b) 30: #11 ExecutorThread::run() /home/owend/master/ep-engine/src/executorthread.cc:113 (ep.so+0x0000001ea96d) 30: #12 launch_executor_thread /home/owend/master/ep-engine/src/executorthread.cc:31 (ep.so+0x0000001ea0d0) 30: #13 CouchbaseThread::run() /home/owend/master/platform/src/cb_pthreads.cc:58 (libplatform_so.so.0.1.0+0x00000000c9c7) 30: #14 platform_thread_wrap /home/owend/master/platform/src/cb_pthreads.cc:71 (libplatform_so.so.0.1.0+0x00000000affe) 30: #15 <null> <null> (libtsan.so.0+0x0000000230d9) 30: 30: Previous write of size 8 at 0x7d1800060a38 by main thread (mutexes: write M1575): 30: #0 operator new(unsigned long) <null> (libtsan.so.0+0x000000025a33) 30: #1 __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::allocate(unsigned long, void const*) <null> (engine_testapp+0x0000004a6ca7) 30: #2 std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >::allocate(std::allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, unsigned long) <null> (engine_testapp+0x0000004a5168) 30: #3 std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_M_get_node() <null> (engine_testapp+0x0000004a316f) 30: #4 std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >* std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_M_create_node<std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>, std::tuple<> >(std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&, std::tuple<>&&) <null> (engine_testapp+0x00000049c6b4) 30: #5 std::_Rb_tree_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_M_emplace_hint_unique<std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>, std::tuple<> >(std::_Rb_tree_const_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&, std::tuple<>&&) <null> (engine_testapp+0x00000049928e) 30: #6 std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::operator[](std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (engine_testapp+0x00000048e488) 30: #7 add_stats /home/owend/master/ep-engine/tests/ep_test_apis.cc:236 (ep_perfsuite.so+0x00000008f5b9) 30: #8 STATWRITER_NAMESPACE::add_casted_stat(char const*, char const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/owend/master/ep-engine/src/statwriter.h:39 (ep.so+0x0000000e5673) 30: #9 void STATWRITER_NAMESPACE::add_casted_stat<unsigned long>(char const*, unsigned long const&, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/owend/master/ep-engine/src/statwriter.h:48 (ep.so+0x0000000e96ae) 30: #10 EventuallyPersistentEngine::addSeqnoVbStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), RCPtr<VBucket> const&) /home/owend/master/ep-engine/src/ep_engine.cc:4445 (ep.so+0x0000001ad56d) 30: #11 EventuallyPersistentEngine::doSeqnoStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), char const*, int) /home/owend/master/ep-engine/src/ep_engine.cc:4490 (ep.so+0x0000001ad95f) 30: #12 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/owend/master/ep-engine/src/ep_engine.cc:4595 (ep.so+0x0000001ae4de) 30: #13 EvpGetStats /home/owend/master/ep-engine/src/ep_engine.cc:232 (ep.so+0x00000019851e) 30: #14 ENGINE_ERROR_CODE std::_Bind<ENGINE_ERROR_CODE (*(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)))(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*))>::__call<ENGINE_ERROR_CODE, , 0ul, 1ul, 2ul, 3ul, 4ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul, 2ul, 3ul, 4ul>) <null> (engine_testapp+0x00000049f211) 30: #15 ENGINE_ERROR_CODE std::_Bind<ENGINE_ERROR_CODE (*(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)))(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*))>::operator()<, ENGINE_ERROR_CODE>() <null> (engine_testapp+0x00000049aa02) 30: #16 std::_Function_handler<ENGINE_ERROR_CODE (), std::_Bind<ENGINE_ERROR_CODE (*(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)))(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*))> >::_M_invoke(std::_Any_data const&) <null> (engine_testapp+0x00000049145b) 30: #17 std::function<ENGINE_ERROR_CODE ()>::operator()() const <null> (engine_testapp+0x000000484e28) 30: #18 call_engine_and_handle_EWOULDBLOCK /home/owend/master/memcached/programs/engine_testapp/engine_testapp.cc:127 (engine_testapp+0x00000047cf00) 30: #19 mock_get_stats /home/owend/master/memcached/programs/engine_testapp/engine_testapp.cc:220 (engine_testapp+0x00000047d896) 30: #20 perf_stat_latency_core /home/owend/master/ep-engine/tests/ep_perfsuite.cc:1238 (ep_perfsuite.so+0x000000063d3d) 30: #21 perf_stat_latency /home/owend/master/ep-engine/tests/ep_perfsuite.cc:1306 (ep_perfsuite.so+0x00000006427a) 30: #22 perf_slow_stat_latency_100vb /home/owend/master/ep-engine/tests/ep_perfsuite.cc:1365 (ep_perfsuite.so+0x00000006498c) 30: #23 execute_test /home/owend/master/memcached/programs/engine_testapp/engine_testapp.cc:962 (engine_testapp+0x000000481776) 30: #24 main /home/owend/master/memcached/programs/engine_testapp/engine_testapp.cc:1359 (engine_testapp+0x000000482ab1) Change-Id: I7bdc847c0913244409fa044e312d53b484dc2dab Reviewed-on: http://review.couchbase.org/68813 Reviewed-by: Dave Rigby <[email protected]> Tested-by: buildbot <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Nov 1, 2016
…kfills
The existing code was safe, however ThreadSanitizer doesn't know about
our own Spinlocks. Given this shouldn't be a hot path, switch to
normal std:mutex.
ThreadSanitizer report:
WARNING: ThreadSanitizer: data race (pid=23569)
Read of size 2 at 0x7d840000eca2 by thread T8 (mutexes: write M294, read M27095, write M66205, write M102676, write M95235):
#0 DcpConnMap::canAddBackfillToActiveQ() ep-engine/src/connmap.cc:1308 (ep.so+0x000000045ac5)
#1 BackfillManager::schedule() ep-engine/src/dcp/backfill-manager.cc:142 (ep.so+0x00000005b0eb)
#2 DcpProducer::scheduleBackfillManager() ep-engine/src/dcp/producer.cc:702 (ep.so+0x000000078fe3)
#3 ActiveStream::scheduleBackfill_UNLOCKED(bool) ep-engine/src/dcp/stream.cc:1016 (ep.so+0x00000008f280)
#4 ActiveStream::transitionState(stream_state_t) ep-engine/src/dcp/stream.cc:1145 (ep.so+0x000000090589)
#5 ActiveStream::setActive() ep-engine/src/dcp/stream.h:204 (ep.so+0x00000009958e)
#6 DcpProducer::streamRequest() ep-engine/src/dcp/producer.cc:327 (ep.so+0x00000007f85d)
#7 EvpDcpStreamReq ep-engine/src/ep_engine.cc:1573 (ep.so+0x0000000cea78)
#8 dcp_stream_req_executor memcached/daemon/mcbp_executors.cc:2272 (memcached+0x00000045925c)
#9 process_bin_packet memcached/daemon/mcbp_executors.cc:4650 (memcached+0x00000046481d)
#10 mcbp_complete_nread(McbpConnection*) memcached/daemon/mcbp_executors.cc:4759 (memcached+0x00000046481d)
#11 conn_nread(McbpConnection*) memcached/daemon/statemachine_mcbp.cc:314 (memcached+0x000000472678)
#12 McbpStateMachine::execute(McbpConnection&) memcached/daemon/statemachine_mcbp.h:43 (memcached+0x000000447054)
#13 McbpConnection::runStateMachinery() memcached/daemon/connection_mcbp.cc:1003 (memcached+0x000000447054)
#14 McbpConnection::runEventLoop(short) memcached/daemon/connection_mcbp.cc:1274 (memcached+0x0000004470dd)
#15 run_event_loop memcached/daemon/connections.cc:147 (memcached+0x00000044b9e9)
#16 event_handler(int, short, void*) memcached/daemon/memcached.cc:851 (memcached+0x00000041466c)
#17 event_persist_closure src/libevent/event.c:1319 (libevent_core-2.0.so.5+0x00000000b6b7)
#18 event_process_active_single_queue src/libevent/event.c:1363 (libevent_core-2.0.so.5+0x00000000b6b7)
#19 event_process_active src/libevent/event.c:1438 (libevent_core-2.0.so.5+0x00000000b6b7)
#20 event_base_loop src/libevent/event.c:1639 (libevent_core-2.0.so.5+0x00000000b6b7)
#21 CouchbaseThread::run() platform/src/cb_pthreads.cc:54 (libplatform.so.0.1.0+0x0000000057a5)
#22 platform_thread_wrap platform/src/cb_pthreads.cc:66 (libplatform.so.0.1.0+0x0000000057a5)
Previous write of size 2 at 0x7d840000eca2 by thread T55:
#0 DcpConnMap::decrNumActiveSnoozingBackfills() ep-engine/src/connmap.cc:1319 (ep.so+0x000000045b7b)
#1 BackfillManager::backfill() ep-engine/src/dcp/backfill-manager.cc:273 (ep.so+0x00000005a783)
#2 BackfillManagerTask::run() ep-engine/src/dcp/backfill-manager.cc:62 (ep.so+0x00000005ac1c)
#3 ExecutorThread::run() ep-engine/src/executorthread.cc:115 (ep.so+0x000000108d96)
#4 launch_executor_thread ep-engine/src/executorthread.cc:33 (ep.so+0x000000109675)
#5 CouchbaseThread::run() platform/src/cb_pthreads.cc:54 (libplatform.so.0.1.0+0x0000000057a5)
#6 platform_thread_wrap platform/src/cb_pthreads.cc:66 (libplatform.so.0.1.0+0x0000000057a5)
Change-Id: I88df57b268c1e615b7c5d2b7caf5f038a53692ba
Reviewed-on: http://review.couchbase.org/69235
Reviewed-by: Trond Norbye <[email protected]>
Tested-by: buildbot <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Nov 2, 2016
As identified by ThreadSantizer:
WARNING: ThreadSanitizer: data race (pid=23560)
Write of size 8 at 0x7ff76a2e7e50 by thread T52 (mutexes: write M64345, write M85935):
#0 VBucket::adjustCheckpointFlushTimeout(unsigned long) ep-engine/src/vbucket.cc:393 (ep.so+0x00000016c806)
#1 VBucket::notifyOnPersistence(EventuallyPersistentEngine&, unsigned long, bool) ep-engine/src/vbucket.cc:332 (ep.so+0x00000016c806)
#2 EventuallyPersistentStore::flushVBucket(unsigned short) ep-engine/src/ep.cc:3475 (ep.so+0x0000000be736)
#3 Flusher::flushVB() ep-engine/src/flusher.cc:284 (ep.so+0x0000001100d1)
#4 Flusher::step(GlobalTask*) ep-engine/src/flusher.cc:183 (ep.so+0x0000001110b7)
#5 FlusherTask::run() ep-engine/src/tasks.cc:138 (ep.so+0x000000156642)
#6 ExecutorThread::run() ep-engine/src/executorthread.cc:115 (ep.so+0x000000108d96)
#7 launch_executor_thread ep-engine/src/executorthread.cc:33 (ep.so+0x000000109675)
#8 CouchbaseThread::run() platform/src/cb_pthreads.cc:54 (libplatform.so.0.1.0+0x0000000057a5)
#9 platform_thread_wrap platform/src/cb_pthreads.cc:66 (libplatform.so.0.1.0+0x0000000057a5)
Previous read of size 8 at 0x7ff76a2e7e50 by thread T9 (mutexes: write M295):
#0 VBucket::getCheckpointFlushTimeout() ep-engine/src/vbucket.cc:406 (ep.so+0x0000001683e6)
#1 EventuallyPersistentEngine::doEngineStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) ep-engine/src/ep_engine.cc:3628 (ep.so+0x0000000e9678)
#2 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) ep-engine/src/ep_engine.cc:4680 (ep.so+0x0000000ea930)
#3 EvpGetStats ep-engine/src/ep_engine.cc:235 (ep.so+0x0000000eb9fe)
#4 stat_executor memcached/daemon/mcbp_executors.cc:3367 (memcached+0x00000046c94b)
#5 process_bin_packet memcached/daemon/mcbp_executors.cc:4650 (memcached+0x00000046481d)
#6 mcbp_complete_nread(McbpConnection*) memcached/daemon/mcbp_executors.cc:4759 (memcached+0x00000046481d)
#7 conn_nread(McbpConnection*) memcached/daemon/statemachine_mcbp.cc:314 (memcached+0x000000472678)
#8 McbpStateMachine::execute(McbpConnection&) memcached/daemon/statemachine_mcbp.h:43 (memcached+0x000000447054)
#9 McbpConnection::runStateMachinery() memcached/daemon/connection_mcbp.cc:1003 (memcached+0x000000447054)
#10 McbpConnection::runEventLoop(short) memcached/daemon/connection_mcbp.cc:1274 (memcached+0x0000004470dd)
#11 run_event_loop memcached/daemon/connections.cc:147 (memcached+0x00000044b9e9)
#12 event_handler(int, short, void*) memcached/daemon/memcached.cc:851 (memcached+0x00000041466c)
#13 event_persist_closure /home/couchbase/serverjenkins/workspace/cbdeps-platform-build/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1319 (libevent_core-2.0.so.5+0x00000000b6b7)
#14 event_process_active_single_queue /home/couchbase/serverjenkins/workspace/cbdeps-platform-build/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1363 (libevent_core-2.0.so.5+0x00000000b6b7)
#15 event_process_active /home/couchbase/serverjenkins/workspace/cbdeps-platform-build/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1438 (libevent_core-2.0.so.5+0x00000000b6b7)
#16 event_base_loop /home/couchbase/serverjenkins/workspace/cbdeps-platform-build/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1639 (libevent_core-2.0.so.5+0x00000000b6b7)
#17 CouchbaseThread::run() platform/src/cb_pthreads.cc:54 (libplatform.so.0.1.0+0x0000000057a5)
#18 platform_thread_wrap platform/src/cb_pthreads.cc:66 (libplatform.so.0.1.0+0x0000000057a5)
Location is global 'VBucket::chkFlushTimeout' of size 8 at 0x7ff76a2e7e50 (ep.so+0x000000431e50)
Change-Id: I2d6f928b8a5552cf08a91c6134ad5134810966c2
Reviewed-on: http://review.couchbase.org/69234
Reviewed-by: Trond Norbye <[email protected]>
Tested-by: buildbot <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Nov 23, 2016
tMutex must be acquired before accessing the threadQ data structure. 07:01:46 WARNING: ThreadSanitizer: data race (pid=32512) 07:01:46 Write of size 8 at 0x7d1c00016eb8 by thread T9 (mutexes: write M27337): 07:01:46 #0 operator delete(void*) <null> (engine_testapp+0x000000464cbb) 07:01:46 #1 <null> <null> (libstdc++.so.6+0x0000000c1ac7) 07:01:46 #2 ExecutorThread::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorthread.cc:127 (ep.so+0x0000000f7ff5) 07:01:46 #3 launch_executor_thread(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorthread.cc:32 (ep.so+0x0000000f78a5) 07:01:46 #4 CouchbaseThread::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/platform/src/cb_pthreads.cc:58 (libplatform_so.so.0.1.0+0x000000007b91) 07:01:46 #5 platform_thread_wrap(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/platform/src/cb_pthreads.cc:71 (libplatform_so.so.0.1.0+0x000000005e92) 07:01:46 07:01:46 Previous read of size 8 at 0x7d1c00016eb8 by main thread (mutexes: write M20977, write M26658): 07:01:46 #0 strlen <null> (engine_testapp+0x0000004652af) 07:01:46 #1 showJobLog(char const*, char const*, std::vector<TaskLogEntry, std::allocator<TaskLogEntry> > const&, void const*, void (char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/statwriter.h:39 (ep.so+0x0000000f346b) 07:01:46 #2 ExecutorPool::doWorkerStat(EventuallyPersistentEngine*, void const*, void (char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorpool.cc:801 (ep.so+0x0000000f2ce2) 07:01:46 #3 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep_engine.cc:4383 (ep.so+0x0000000c30bf) Change-Id: Iae7aa0fe9da15805671762516221f25da69fe5d8 Reviewed-on: http://review.couchbase.org/70262 Reviewed-by: Dave Rigby <[email protected]> Tested-by: buildbot <[email protected]>
ns-codereview
pushed a commit
that referenced
this pull request
Mar 2, 2017
HashTable will (must) be rather cleared in the destructor of VBucket object. Otherwise, race condition flagged by thread/address sanitizers when StoredValue is stored in more than one data structure. (like in both HashTable and Linked list in http://review.couchbase.org/#/c/74440) 16:55:17 ==27000==ERROR: AddressSanitizer: heap-use-after-free on address 0x60700001be60 at pc 0x2b5e33efd0ae bp 0x7ffd62e2d270 sp 0x7ffd62e2d268 16:55:17 READ of size 8 at 0x60700001be60 thread T0 16:55:17 #0 0x2b5e33efd0ad in boost::intrusive::list_node_traits<void*>::get_next(boost::intrusive::list_node<void*>* const&) /home/couchbase/jenkins/workspace/ep-engine-addresssanitizer-master/build/tlm/deps/boost.exploded/include/boost/intrusive/detail/list_node.hpp:63:13 16:55:17 #1 0x2b5e33efd0ad in boost::intrusive::list_iterator<boost::intrusive::mhtraits<StoredValue, boost::intrusive::list_member_hook<void, void, void>, &StoredValue::hook_>, true>::operator++() /home/couchbase/jenkins/workspace/ep-engine-addresssanitizer-master/build/tlm/deps/boost.exploded/include/boost/intrusive/detail/list_iterator.hpp:81 16:55:17 #2 0x2b5e33efd0ad in void boost::intrusive::list_impl<boost::intrusive::mhtraits<StoredValue, boost::intrusive::list_member_hook<void, void, void>, &StoredValue::hook_>, unsigned long, true, void>::clear_and_dispose<boost::intrusive::detail::null_disposer>(boost::intrusive::detail::null_disposer) /home/couchbase/jenkins/workspace/ep-engine-addresssanitizer-master/build/tlm/deps/boost.exploded/include/boost/intrusive/list.hpp:744 16:55:17 #3 0x2b5e33efd0ad in boost::intrusive::list_impl<boost::intrusive::mhtraits<StoredValue, boost::intrusive::list_member_hook<void, void, void>, &StoredValue::hook_>, unsigned long, true, void>::clear() /home/couchbase/jenkins/workspace/ep-engine-addresssanitizer-master/build/tlm/deps/boost.exploded/include/boost/intrusive/list.hpp:719 16:55:17 #4 0x2b5e33efd0ad in BasicLinkedList::~BasicLinkedList() /home/couchbase/jenkins/workspace/ep-engine-addresssanitizer-master/ep-engine/src/linkedlist.cc:30 16:55:17 #5 0x2b5e33efd0ad in BasicLinkedList::~BasicLinkedList() /home/couchbase/jenkins/workspace/ep-engine-addresssanitizer-master/ep-engine/src/linkedlist.cc:25 16:55:17 #6 0x2b5e33e5ece3 in std::default_delete<SequenceList>::operator()(SequenceList*) const /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/unique_ptr.h:76:2 16:55:17 #7 0x2b5e33e5ece3 in std::unique_ptr<SequenceList, std::default_delete<SequenceList> >::~unique_ptr() /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/unique_ptr.h:236 16:55:17 #8 0x2b5e33e5ece3 in EphemeralVBucket::~EphemeralVBucket() /home/couchbase/jenkins/workspace/ep-engine-addresssanitizer-master/ep-engine/src/ephemeral_vb.h:24 16:55:17 #9 0x2b5e33e5ece3 in EphemeralVBucket::~EphemeralVBucket() /home/couchbase/jenkins/workspace/ep-engine-addresssanitizer-master/ep-engine/src/ephemeral_vb.h:24 16:55:17 #10 0x2b5e33eaee1c in RCPtr<VBucket>::~RCPtr() /home/couchbase/jenkins/workspace/ep-engine-addresssanitizer-master/ep-engine/src/atomic.h:158:13 16:55:17 #11 0x2b5e33eaee1c in KVBucket::deleteVBucket(unsigned short, void const*) /home/couchbase/jenkins/workspace/ep-engine-addresssanitizer-master/ep-engine/src/kv_bucket.cc:902 16:55:17 #12 0x2b5e33e2cce5 in EventuallyPersistentEngine::deleteVBucket(unsigned short, void const*) /home/couchbase/jenkins/workspace/ep-engine-addresssanitizer-master/ep-engine/src/ep_engine.h:532:16 Change-Id: Ib46448cd396396dff183775a27c17c9f27d6a7e0 16:55:17 0x60700001be60 is located 0 bytes inside of 78-byte region [0x60700001be60,0x60700001beae) 16:55:17 freed by thread T12 (mc:nonIO_2) here: 16:55:17 #0 0x4ee992 in operator delete(void*) (/home/couchbase/jenkins/workspace/ep-engine-addresssanitizer-master/build/memcached/engine_testapp+0x4ee992) 16:55:17 #1 0x2b5e33e89a45 in StoredValue::operator delete(void*) /home/couchbase/jenkins/workspace/ep-engine-addresssanitizer-master/ep-engine/src/stored-value.h:41:9 16:55:17 #2 0x2b5e33e89a45 in HashTable::clear(bool) /home/couchbase/jenkins/workspace/ep-engine-addresssanitizer-master/ep-engine/src/hash_table.cc:98 16:55:17 #3 0x2b5e33f8aa49 in VBucketMemoryDeletionTask::run() /home/couchbase/jenkins/workspace/ep-engine-addresssanitizer-master/ep-engine/src/vbucketmemorydeletiontask.cc:45:5 Reviewed-on: http://review.couchbase.org/74500 Tested-by: Build Bot <[email protected]> Reviewed-by: Dave Rigby <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In the 3 modified files : cbcompact, cbepctl and cbstats the help is saying that the default bucket is "default".
It was ignored.