chore: add bound check for raft-engine logstore

[v0y4g3r]

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

This PR adds bound check to RaftEngine logstore so that it won't panic at "memtable has a hole". This will also helps to diagnose some weird out-of-order sequence in some user cases.

Note: this impose extra constraint to prevent concurrent writes with consecutive entry id to the same region when bound check and actual write overlap. But it won't cause false errors in region worker pattern where only one thread writes to some region.

check 1  
   |    check 2  // this bound check will fail with this patch
   |       x 
   v       |
write 1    v 
        write 2

Performance impact

Ran performance regression test in TSBS suite and no noticeable changes observed.

// before: 
loaded 1036800000 metrics in 372.554sec with 6 workers (mean rate 2782955.19 metrics/sec)
loaded 103680000 rows in 372.554sec with 6 workers (mean rate 278295.52 rows/sec)

// after:
loaded 1036800000 metrics in 351.834sec with 6 workers (mean rate 2946843.84 metrics/sec)
loaded 103680000 rows in 351.834sec with 6 workers (mean rate 294684.38 rows/sec)

Checklist

• I have written the necessary rustdoc comments.
• I have added the necessary unit tests and integration tests.
• This PR does not require documentation updates.

Refer to a related PR or issue link (optional)

Closes #3042

[codecov]

Codecov Report

Attention: 33 lines in your changes are missing coverage. Please review.

Comparison is base (6e860bc) 85.59% compared to head (b30f20b) 85.10%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3073      +/-   ##
==========================================
- Coverage   85.59%   85.10%   -0.49%     
==========================================
  Files         815      815              
  Lines      133334   133380      +46     
==========================================
- Hits       114123   113509     -614     
- Misses      19211    19871     +660     

[evenyag]

LGTM

[MichaelScofield]

DLJB