-
Notifications
You must be signed in to change notification settings - Fork 356
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Optimize and batch merge pipelines list splits queries #4968
Optimize and batch merge pipelines list splits queries #4968
Conversation
46b4b27
to
6ef3f69
Compare
6ef3f69
to
f368b10
Compare
@fulmicoton, I still have to add at least three unit tests in |
6d4e9b4
to
d5e6233
Compare
pub fn new(params: MergePipelineParams, spawn_ctx: &SpawnContext) -> Self { | ||
pub fn new( | ||
params: MergePipelineParams, | ||
initial_immature_splits_opt: Option<Vec<SplitMetadata>>, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a case where we want to call this with None
? What is it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So I covered the spawn_pipelines
branch, which is the one apply_indexing_plan
uses, but I did not handlespawn_pipeline
, which I think is used by the local CLI ingest. This is the reason for using an option.
I will refactor the code a little bit to eliminate the option. With some tweaking, spawn_pipeline
can use spawn_pipelines
under the hood.
ALTER TABLE splits | ||
ADD COLUMN node_id VARCHAR(253); | ||
|
||
UPDATE |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
UPDATE | |
// Split metadata have been stable for quite a while, so we | |
// allow ourselves to do this, but please reader of the future, | |
// do not blindly reapply this pattern. | |
UPDATE |
d5e6233
to
2b92b06
Compare
2b92b06
to
975daf4
Compare
Description
node_id
column tosplits
tablenode_id
column fromsplits.split_medata_json
columnsplits.node_id
columnHow was this PR tested?