Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[breaking] pruning and reorganizing fields from protos, moving stuff to state #1087

Merged
merged 3 commits into from
Jan 17, 2024

Conversation

heavycrystal
Copy link
Contributor

@heavycrystal heavycrystal commented Jan 16, 2024

⚠️ This change can break existing CDC mirrors!

Performs several breaking changes to our protobufs:

  1. Removes unused fields from FlowConnectionConfigs: table_schema, src_table_id_name_mapping, table_name_schema_mapping, snapshot_sync_mode, cdc_sync_mode, push_batch_size, push_parallelism, metadata_peer. src_table_id_name_mapping and table_name_schema_mapping are now moved to state, setup from SetupFlow and passed accordingly via more proto changes. Order shuffled around to group similar fields together.
  2. Removed unnecessary generalization of table_identifier
  3. Removed unused messages EnsurePullabilityInput and EnsurePullabilityOutput
  4. Changed wording of do_initial_copy and initial_copy_only to better align with other options.
  5. Removed columns from TableSchema and fixed related methods up.
  6. batch_size had 3 different datatypes and was being modified in multiple places, making sure batch_size and idle_timeout_seconds has one source of truth.

nexus/flow-rs/src/grpc.rs Outdated Show resolved Hide resolved
@heavycrystal heavycrystal merged commit 9c447cf into main Jan 17, 2024
12 checks passed
@serprex serprex deleted the flow-pruning-pt1 branch July 19, 2024 15:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants