Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(connector): handle update insert and delete + subset columns + handle delete mapping #20238

Merged
merged 9 commits into from
Jan 21, 2025
Merged

fix(connector): handle update insert and delete + subset columns + handle delete mapping #20238

merged 9 commits into from
Jan 21, 2025
+107 −14

Conversation

kwannoel
Copy link
Contributor

@kwannoel kwannoel commented Jan 21, 2025 •
edited
Loading

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Closes #20178. Also the tests were only running on my machine previously and not CI, and hence there were several regressions. I have added them to e2e-sink-test.sh in this PR.

Checklist

  • I have written necessary rustdoc comments.
  • I have added necessary unit tests and integration tests.
  • I have added test labels as necessary.
  • I have added fuzzing tests or opened an issue to track them.
  • My PR contains breaking changes.
  • My PR changes performance-critical code, so I will run (micro) benchmarks and present the results.
  • My PR contains critical fixes that are necessary to be merged into the latest release.

Documentation

  • My PR needs documentation updates.
Release note

@kwannoel Graphite App
Copy link
Contributor Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

@kwannoel kwannoel changed the title fix bugs: handle updateDelete/updateInsert, handle delete assertion mismatch fix(connector): fix postgres sink bugs (handle updateInsert/updateDelete, Jan 21, 2025
@kwannoel kwannoel changed the title fix(connector): fix postgres sink bugs (handle updateInsert/updateDelete, fix(connector): handle update insert and delete + subset pk + handle delete mapping Jan 21, 2025
@kwannoel kwannoel marked this pull request as ready for review January 21, 2025 08:08
@gru-agent Gru Agent
Copy link

gru-agent bot commented Jan 21, 2025

This pull request has been modified. If you want me to regenerate unit test for any of the files related, please find the file in "Files Changed" tab and add a comment @gru-agent. (The github "Comment on this file" feature is in the upper right corner of each file in "Files Changed" tab.)

chenzl25
chenzl25 approved these changes Jan 21, 2025
View reviewed changes
Copy link
Contributor

@chenzl25 chenzl25 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kwannoel kwannoel force-pushed the kwannoel/fix-pg-sink-bugs-1 branch from c5e959e to 5ccd8f0 Compare January 21, 2025 13:51
@kwannoel kwannoel force-pushed the kwannoel/fix-pg-sink-bugs-1 branch from 5ccd8f0 to c3ef510 Compare January 21, 2025 13:51
@kwannoel kwannoel enabled auto-merge January 21, 2025 14:57
@kwannoel kwannoel disabled auto-merge January 21, 2025 16:23
@kwannoel kwannoel enabled auto-merge January 21, 2025 18:06
@kwannoel kwannoel added this pull request to the merge queue Jan 21, 2025
Merged via the queue into main with commit 711fedf Jan 21, 2025
29 of 30 checks passed
@kwannoel kwannoel deleted the kwannoel/fix-pg-sink-bugs-1 branch January 21, 2025 19:35
github-actions bot pushed a commit that referenced this pull request Jan 21, 2025
@kwannoel
fix(connector): handle update insert and delete + subset pk + handle …
9e00966 
…delete mapping (#20238)
@github-actions github-actions bot mentioned this pull request Jan 21, 2025
Merged
github-actions bot pushed a commit that referenced this pull request Jan 21, 2025
@kwannoel
fix(connector): handle update insert and delete + subset pk + handle …
8a02eb7 
…delete mapping (#20238)
@github-actions github-actions bot mentioned this pull request Jan 21, 2025
Open
@fuyufjh fuyufjh changed the title fix(connector): handle update insert and delete + subset pk + handle delete mapping fix(connector): handle update insert and delete + subset columns + handle delete mapping Jan 22, 2025
fuyufjh
fuyufjh reviewed Jan 22, 2025
View reviewed changes
src/stream/src/common/compact_chunk.rs
@@ -232,9 +232,6 @@ impl StreamChunkCompactor {
/// have three kind of patterns Insert, Delete or Update.
/// - For the update (-old row, +old row), when old row is exactly same. The two rowOp will be
/// removed.
///
/// All UPDATE INSERT and UPDATE DELETE will be converted to INSERT and DELETE, and dropped according to
/// certain rules (see `merge_insert` and `merge_delete` for more details).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@st1page

github-merge-queue bot pushed a commit that referenced this pull request Jan 22, 2025
@github-actions @kwannoel
fix(connector): handle update insert and delete + subset columns + ha…
89f9122 
…ndle delete mapping (#20238) (#20246)

Co-authored-by: Noel Kwan <[email protected]>
kwannoel added a commit that referenced this pull request Jan 27, 2025
@kwannoel
fix(connector): handle update insert and delete + subset pk + handle …
912fee9 
…delete mapping (#20238)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@fuyufjh fuyufjh fuyufjh left review comments

@chenzl25 chenzl25 chenzl25 approved these changes

@BugenZhao BugenZhao Awaiting requested review from BugenZhao

Assignees
No one assigned
Labels
ci/main-cron/run-selected ci/run-e2e-sink-tests need-cherry-pick-release-2.1 need-cherry-pick-release-2.2 type/fix Bug fix
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

PG sink can use subset of downstream pg table columns
3 participants
@kwannoel @chenzl25 @fuyufjh