Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Option to skip merge and direct copy for insert only tables #226

Closed
alok87 opened this issue May 16, 2021 · 0 comments · Fixed by #227
Closed

Option to skip merge and direct copy for insert only tables #226

alok87 opened this issue May 16, 2021 · 0 comments · Fixed by #227
Labels
performance Monitoring, Metrics, Logs, Benchmarks

Comments

@alok87
Copy link
Contributor

alok87 commented May 16, 2021

Some huge tables are insert only tables, you do not perform UPDATES or delete on it. Such tables are huge transaction tables or events tables. User should have option to specify these tables can skip merge and directly load data.

Screenshot 2021-05-16 at 1 50 27 PM

Such huge tables spend major time in doing the delete-common operations, 1.3minutes in the above one while it takes only few seconds to copy the data.

If we can skip merge for such tables it would help speed up the loads and free up the queue for other to load fast.

Should help in #186

This was tried in past and reverted https://github.com/practo/tipoca-stream/pull/88/files later

@alok87 alok87 added the performance Monitoring, Metrics, Logs, Benchmarks label May 16, 2021
@alok87 alok87 changed the title Option to skip merge and direct copy for few tables Option to skip merge and direct copy for insert only tables May 16, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance Monitoring, Metrics, Logs, Benchmarks
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant