Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DB archival can cause slowness in full reload over time #154

Open
justjkk opened this issue Mar 5, 2021 · 2 comments
Open

DB archival can cause slowness in full reload over time #154

justjkk opened this issue Mar 5, 2021 · 2 comments
Labels
enhancement New feature or request performance Monitoring, Metrics, Logs, Benchmarks

Comments

@justjkk
Copy link
Contributor

justjkk commented Mar 5, 2021

Multiple tables like communicator.genericsms archive data older than a few months to S3 in order to save space. Over time this can cause more entries in Kafka that slows down a full reload. Need to figure out a way to periodically refresh the Debezium data so that data is stored as snapshots rather than events.
@justjkk
Copy link
Contributor Author

justjkk commented Mar 5, 2021

This issue can also happen due to some tables that are doing a lot of updates like sessions table last_accessed_at column.

@justjkk justjkk added enhancement New feature or request performance Monitoring, Metrics, Logs, Benchmarks labels Mar 5, 2021
@alok87
Copy link
Contributor

alok87 commented Jul 29, 2021

Log compaction can be used here.
It can help keep only the latest record for key. @justjkk

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request performance Monitoring, Metrics, Logs, Benchmarks
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@justjkk @alok87