[ISSUE #7064] [RIP-66-1] Support KV(RocksDB) Storage for Metadata #7092

fujian-zfj · 2023-07-28T09:24:32Z

Which Issue(s) This PR Fixes

Fixes #7064

Brief Description

This proposal mainly optimizes and solves the existing perfermance problems in the million-topic scenario :

Metadata like topic, subscription, consumerOffset implement kv storage

codecov-commenter · 2023-07-30T03:01:00Z

Codecov Report

Attention: Patch coverage is 21.78344% with 614 lines in your changes missing coverage. Please review.

Project coverage is 42.54%. Comparing base (d429bd7) to head (26eec8f).
Report is 432 commits behind head on develop.

Files with missing lines	Patch %	Lines
...rocketmq/common/config/AbstractRocksDBStorage.java	0.00%	305 Missing ⚠️
...e/rocketmq/common/config/ConfigRocksDBStorage.java	0.00%	128 Missing ⚠️
...e/rocketmq/common/config/RocksDBConfigManager.java	0.00%	47 Missing ⚠️
...broker/offset/RocksDBLmqConsumerOffsetManager.java	0.00%	35 Missing ⚠️
.../subscription/RocksDBSubscriptionGroupManager.java	59.52%	15 Missing and 2 partials ⚠️
...tmq/broker/topic/RocksDBLmqTopicConfigManager.java	0.00%	13 Missing ⚠️
...ache/rocketmq/store/config/MessageStoreConfig.java	29.41%	12 Missing ⚠️
...bscription/RocksDBLmqSubscriptionGroupManager.java	0.00%	11 Missing ⚠️
...a/org/apache/rocketmq/broker/BrokerController.java	41.17%	3 Missing and 7 partials ⚠️
...ache/rocketmq/broker/topic/TopicConfigManager.java	78.72%	10 Missing ⚠️
... and 5 more

Additional details and impacted files

@@              Coverage Diff              @@
##             develop    #7092      +/-   ##
=============================================
- Coverage      42.73%   42.54%   -0.19%     
- Complexity      9295     9364      +69     
=============================================
  Files           1138     1150      +12     
  Lines          81254    81975     +721     
  Branches       10637    10688      +51     
=============================================
+ Hits           34721    34876     +155     
- Misses         42190    42752     +562     
- Partials        4343     4347       +4

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

RongtongJin · 2023-07-30T11:57:38Z

broker/src/main/java/org/apache/rocketmq/broker/offset/ConsumerOffsetManager.java

+    protected void removeConsumerOffset(String topicAtGroup) {
+
+    }


When was the method called？

This interface is mainly to let RocksDBConsumeOffsetManager to override and delete the data in rocksdb, you can see the overridden method in RocksDBConsumeOffsetManager.

RongtongJin · 2023-07-30T12:28:44Z

...t/src/main/java/org/apache/rocketmq/client/consumer/store/RocksDBOffsetSerializeWrapper.java

+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.client.consumer.store;


Putting this class in the client module is a bit strange

Because I found OffsetSerializeWrapper under the client moudle, so I put RocksDBOffsetSerializeWrapper in this directory as well.

OffsetSerializeWrapper is used in client module, RocksDBOffsetSerializeWrapper is used in broker module instead of client module, so I think this is not appropriate.

RongtongJin · 2023-07-30T12:30:31Z

example/src/main/java/org/apache/rocketmq/example/quickstart/Consumer.java

@@ -50,7 +50,7 @@ public static void main(String[] args) throws MQClientException {
         * </pre>
         */
        // Uncomment the following line while debugging, namesrvAddr should be set to your local address
-//        consumer.setNamesrvAddr(DEFAULT_NAMESRVADDR);


It would be better not to modify this

If you want to modify this, please also modify the comments above.

Signed-off-by: Li Zhanhui <[email protected]>

lizhanhui · 2023-07-31T04:07:10Z

@fujian-zfj Fixed bazel build scripts for you

fujian-zfj · 2023-07-31T04:12:31Z

@fujian-zfj Fixed bazel build scripts for you

OK，thank you

Signed-off-by: Li Zhanhui <[email protected]>

…etmq into develop_rocksdb_metadata

fujian-zfj and others added 28 commits November 30, 2021 20:19

typo int readme[ecosystem]

09c0c8c

Merge branch 'apache:develop' into develop

89713ba

Merge branch 'apache:develop' into develop

8d0c4fc

Merge branch 'apache:develop' into develop

2c9e961

Merge branch 'apache:develop' into develop

93d6669

Merge branch 'apache:develop' into develop

2dd16d6

Merge branch 'apache:develop' into develop

e02c7aa

Merge branch 'apache:develop' into develop

cb8c7b0

Merge branch 'apache:develop' into develop

03f707f

Merge branch 'apache:develop' into develop

6ef608f

Merge branch 'apache:develop' into develop

818919f

Merge branch 'apache:develop' into develop

b315381

Merge branch 'apache:develop' into develop

5d1b050

Merge branch 'apache:develop' into develop

de2238b

Merge branch 'apache:develop' into develop

d22756f

Merge branch 'apache:develop' into develop

83d8178

Merge branch 'apache:develop' into develop

a4c94a9

Merge branch 'apache:develop' into develop

7234a41

Merge branch 'apache:develop' into develop

c7708a7

Merge branch 'apache:develop' into develop

166ebbd

Merge branch 'apache:develop' into develop

6b27cc7

Merge branch 'apache:develop' into develop

3b0a27b

rocksdb metadata

80e259e

add unit test

11a0c32

fix testOffsetPersistInMemory

18b909b

fix unit test

2fa8485

fix unit test

2de0cca

remove unused import

0cc5cff

RongtongJin reviewed Jul 30, 2023

View reviewed changes

fujian-zfj and others added 2 commits July 31, 2023 10:09

move RocksDBOffsetSerialize to broker moudle

03d9988

Fix bazel build scripts

1a99cc8

Signed-off-by: Li Zhanhui <[email protected]>

lizhanhui and others added 3 commits July 31, 2023 04:55

Flag QueryMsgByKeyIT as flaky as it fails at frequency: 5 out of 32

e0aae35

Signed-off-by: Li Zhanhui <[email protected]>

change public to private of some inner method

7e4a409

Merge branch 'develop_rocksdb_metadata' of github.com:fujian-zfj/rock…

26eec8f

…etmq into develop_rocksdb_metadata

RongtongJin approved these changes Aug 2, 2023

View reviewed changes

RongtongJin merged commit 3a6ef04 into apache:develop Aug 4, 2023

This was referenced Aug 17, 2023

[Enhancement] Set table reference the same object for setSubscriptionGroupTable method #7203

Closed

[ISSUE #7203] Set table reference the same object for setSubscriptionGroupTable method #7204

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ISSUE #7064] [RIP-66-1] Support KV(RocksDB) Storage for Metadata #7092

[ISSUE #7064] [RIP-66-1] Support KV(RocksDB) Storage for Metadata #7092

fujian-zfj commented Jul 28, 2023

codecov-commenter commented Jul 30, 2023 •

edited

Loading

RongtongJin Jul 30, 2023

fujian-zfj Jul 30, 2023 •

edited

Loading

RongtongJin Jul 30, 2023

fujian-zfj Jul 30, 2023

RongtongJin Jul 31, 2023

fujian-zfj Jul 31, 2023

RongtongJin Jul 30, 2023

fujian-zfj Jul 30, 2023

RongtongJin Jul 31, 2023

fujian-zfj Jul 31, 2023

lizhanhui commented Jul 31, 2023

fujian-zfj commented Jul 31, 2023

[ISSUE #7064] [RIP-66-1] Support KV(RocksDB) Storage for Metadata #7092

[ISSUE #7064] [RIP-66-1] Support KV(RocksDB) Storage for Metadata #7092

Conversation

fujian-zfj commented Jul 28, 2023

Which Issue(s) This PR Fixes

Brief Description

codecov-commenter commented Jul 30, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

fujian-zfj Jul 30, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lizhanhui commented Jul 31, 2023

fujian-zfj commented Jul 31, 2023

codecov-commenter commented Jul 30, 2023 •

edited

Loading

fujian-zfj Jul 30, 2023 •

edited

Loading