New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(remote_wal): split an entry if it's too large #3092

Merged

niebayes merged 20 commits into main from feat/split_large_entry

Jan 5, 2024

Contributor

niebayes commented Jan 3, 2024 •

edited

Loading

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

Split an entry if it's too large.

Reverted an ignored sqlness test, see: #3050

Checklist

I have written the necessary rustdoc comments.
I have added the necessary unit tests and integration tests.
This PR does not require documentation updates.

Refer to a related PR or issue link (optional)


          feat: split an entry if it's too large

955134e

niebayes self-assigned this

github-actions bot added Size: L docs-required labels

sunng87 mentioned this pull request

Update docs for feat: split an entry if it's too large GreptimeTeam/docs#759

Closed

niebayes changed the title ~~feat: split an entry if it's too large~~ feat(remote_wal): split an entry if it's too large

niebayes added docs-not-required and removed docs-required Size: L labels

niebayes mentioned this pull request

Release blocker of v0.5.1 #3048

Closed

14 tasks

niebayes requested review from WenyXu, MichaelScofield, v0y4g3r and evenyag

January 3, 2024 11:56

niebayes linked an issue

that may be closed by this pull request

[remote_wal] Split a log store entry if it cannot fit into a Kafka record #3049

Closed

codecov bot commented Jan 3, 2024 •

edited

Loading

Codecov Report

Attention: 23 lines in your changes are missing coverage. Please review.

Comparison is base (702ea32) 85.63% compared to head (45c1860) 85.67%.
Report is 6 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3092      +/-   ##
==========================================
+ Coverage   85.63%   85.67%   +0.03%     
==========================================
  Files         806      807       +1     
  Lines      131606   132525     +919     
==========================================
+ Hits       112706   113543     +837     
- Misses      18900    18982      +82

WenyXu reviewed

View reviewed changes

src/log-store/src/kafka/log_store.rs Outdated Show resolved Hide resolved

src/log-store/src/kafka/log_store.rs Show resolved Hide resolved

src/log-store/src/kafka/util/record.rs Outdated Show resolved Hide resolved


          chore: rewrite check records

5fa5018

github-actions bot added the Size: L label

MichaelScofield reviewed

View reviewed changes

src/log-store/src/kafka/log_store.rs Outdated Show resolved Hide resolved

src/log-store/src/kafka/log_store.rs Outdated Show resolved Hide resolved

src/log-store/src/kafka/log_store.rs Outdated Show resolved Hide resolved

src/log-store/src/kafka/util/record.rs Outdated Show resolved Hide resolved

src/log-store/src/kafka/util/record.rs Outdated Show resolved Hide resolved

src/log-store/src/kafka/util/record.rs Outdated Show resolved Hide resolved

src/log-store/src/kafka/util/record.rs Outdated Show resolved Hide resolved

src/mito2/src/region_write_ctx.rs Show resolved Hide resolved

src/log-store/src/kafka/util/record.rs Outdated Show resolved Hide resolved

src/log-store/src/kafka/util/record.rs Outdated Show resolved Hide resolved

niebayes added 4 commits

January 4, 2024 15:32


          test: add some unit tests for record


          chore: rewrite entry splitting

729950a


          chore: add unit tests for build records

7fed5e4


          chore: add more unit tests for record

5a3fbee

github-actions bot added Size: XL and removed Size: L labels

WenyXu reviewed

View reviewed changes

src/log-store/src/kafka/util/record.rs Outdated Show resolved Hide resolved

src/log-store/src/kafka/util/record.rs Outdated Show resolved Hide resolved

WenyXu reviewed

View reviewed changes

src/log-store/src/kafka/util/record.rs Outdated Show resolved Hide resolved

src/log-store/src/kafka/util/record.rs Outdated Show resolved Hide resolved

src/log-store/src/kafka/util/record.rs Outdated Show resolved Hide resolved

src/log-store/src/kafka/util/record.rs Outdated Show resolved Hide resolved

niebayes added 2 commits

January 4, 2024 19:38


          chore: rewrite encdec of record

33ea9af


          revert: ignored test

e6dec26


          fix: CR

3be3b97

WenyXu reviewed

View reviewed changes

tests/cases/standalone/common/types/string/scan_big_varchar.sql Show resolved Hide resolved

WenyXu reviewed

View reviewed changes

src/common/config/src/wal/kafka.rs Show resolved Hide resolved

src/common/config/src/wal/kafka.rs Show resolved Hide resolved

niebayes requested review from WenyXu and MichaelScofield

January 5, 2024 08:24

WenyXu reviewed

View reviewed changes

src/datanode/src/datanode.rs Outdated Show resolved Hide resolved

WenyXu reviewed

View reviewed changes

src/log-store/src/kafka/util/record.rs Outdated Show resolved Hide resolved

src/log-store/src/kafka/util/record.rs Outdated Show resolved Hide resolved

src/log-store/src/kafka/util/record.rs Outdated Show resolved Hide resolved

src/log-store/src/kafka/util/record.rs Show resolved Hide resolved

src/log-store/src/kafka/util/record.rs Show resolved Hide resolved

WenyXu reviewed

View reviewed changes

src/log-store/src/lib.rs Outdated Show resolved Hide resolved

evenyag reviewed

View reviewed changes

src/log-store/src/kafka/util/record.rs Outdated Show resolved Hide resolved

v0y4g3r reviewed

View reviewed changes

src/datanode/src/datanode.rs Outdated Show resolved Hide resolved

src/log-store/src/kafka/util/record.rs Show resolved Hide resolved

src/log-store/src/kafka/util/record.rs Outdated Show resolved Hide resolved

src/log-store/src/kafka/util/record.rs Outdated Show resolved Hide resolved

src/log-store/src/kafka/util/record.rs Show resolved Hide resolved

src/log-store/src/kafka/util/record.rs Show resolved Hide resolved

MichaelScofield reviewed

View reviewed changes

src/log-store/src/kafka/util/record.rs Outdated Show resolved Hide resolved

src/log-store/src/kafka/util/record.rs Show resolved Hide resolved

src/log-store/src/kafka/util/record.rs Outdated Show resolved Hide resolved

src/log-store/src/kafka/util/record.rs Show resolved Hide resolved


          fix: properly terminate

aa062f2

niebayes requested review from WenyXu, evenyag, MichaelScofield and v0y4g3r

January 5, 2024 11:51

niebayes added 3 commits

January 5, 2024 19:52


          fix: CR

d64ae3e


          fix: compiling

8aa4068


          fix: sqlness

08338b4

WenyXu approved these changes

View reviewed changes

Member

WenyXu left a comment

LGTM🚀

niebayes added 3 commits

January 5, 2024 20:17


          fix: CR

3778c9c


          fix: license

b09bf25


          fix: license

45c1860

niebayes enabled auto-merge

January 5, 2024 12:31

v0y4g3r approved these changes

View reviewed changes

Contributor

v0y4g3r left a comment

LGTM

niebayes added this pull request to the merge queue

Merged via the queue into main with commit 7830363

23 checks passed

niebayes deleted the feat/split_large_entry branch

January 5, 2024 12:53

niebayes mentioned this pull request

[remote_wal] Add transaction support for producing records #3108

Closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs-not-required