Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TiKV-CDC Memory Quota Reached 512 MB After TiCDC Restart, Changefeed Stuck and Unrecoverable #12036

Closed
wlwilliamx opened this issue Jan 24, 2025 · 1 comment
Labels
area/ticdc Issues or PRs related to TiCDC. type/bug The issue is confirmed as a bug.

Comments

@wlwilliamx
Copy link
Contributor

What did you do?

  1. One CDC instance running in Kubernetes was restarted.
  2. After the restart, the memory usage of TiKV-CDC began to increase continuously.
  3. Eventually, the memory quota (512 MB) was fully utilized, causing the Changefeed to hang.
  4. Restarting the CDC instance multiple times did not resolve the issue. The memory quota of TiKV-CDC remained at 512 MB, and the Changefeed remained stuck.

What did you expect to see?

  • TiKV-CDC should release memory after the restart and operate within the configured memory quota.
  • The Changefeed should resume normal processing after restarting the CDC instance.

What did you see instead?

  • TiKV-CDC's memory usage continued to increase after the restart and eventually hit the 512 MB quota.
  • The Changefeed was stuck and unable to recover.
  • Restarting the CDC instance multiple times did not free up memory or resolve the issue. The memory quota of TiKV-CDC remained at 512 MB.

Versions of the cluster

Upstream TiKV version (execute tikv-server --version):

v7.5.4, with 14 TiKV nodes

TiCDC version (execute cdc version):

v7.5.4, with 2 TiCDC nodes
@wlwilliamx wlwilliamx added area/ticdc Issues or PRs related to TiCDC. type/bug The issue is confirmed as a bug. labels Jan 24, 2025
@wlwilliamx
Copy link
Contributor Author

Report this bug in TiKV #18169

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/ticdc Issues or PRs related to TiCDC. type/bug The issue is confirmed as a bug.
Projects
None yet
Development

No branches or pull requests

1 participant