Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Kinesis] Support the MultistreamTracker API. #1

Draft
wants to merge 4 commits into
base: tubi-main
Choose a base branch
from

Conversation

dluxhu
Copy link
Collaborator

@dluxhu dluxhu commented Sep 19, 2024

PRELIMINARY PR, NOT READY TO MERGE YET

Add support for the new MultistreamTracker API.

This PR contains:

  • A new field in ShardProcessorData: streamIdentifer. This way the consumers of CommittableRecord are able to determine which stream the record came from.
  • Update for the checkpointing logic, now it takes the stream identifier into account, too when grouping the records, not hust the shard id.

Usage

There are minimal changes in the usage of KinesisSchedulerSource:

  • KinesisSchedulerSource is now able to track multiple streams.
  • checkpointRecordsFlow and checkpointRecordBatch now takes an optional maxShardCount parameter, because the maximal shard count can now be over the maximum amount of shards of one flow (500).

Note: this PR contains changes that are to be revertedn before merging to the Akka repository.

TODO

  • Tests
  • Java version
  • Documentation

@dluxhu dluxhu changed the base branch from tubi-main to main September 19, 2024 01:00
@dluxhu dluxhu changed the base branch from main to tubi-main September 19, 2024 01:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant