Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Incremental transformations and sinks #520

Open
yruslan opened this issue Nov 18, 2024 · 0 comments
Open

Incremental transformations and sinks #520

yruslan opened this issue Nov 18, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@yruslan
Copy link
Collaborator

yruslan commented Nov 18, 2024 •
edited
Loading

Background

Incremental ingestion is already supported. Now we need to be able to do processing and send data to sinks incrementally as well.

Feature

Incremental transformations and sinks.

Example [Optional]

To be added...

Proposed Solution [Optional]

To be added...
@yruslan yruslan added the enhancement New feature or request label Nov 18, 2024
yruslan added a commit that referenced this issue Nov 20, 2024
@yruslan
#520 Write batchid information for raw tables in the metastore.
df02404
yruslan added a commit that referenced this issue Nov 20, 2024
@yruslan
#520 Write batchid information for raw tables in the metastore.
a753387
yruslan added a commit that referenced this issue Nov 21, 2024
@yruslan
#520 Fix handling of incremental ingestion for 'raw' format of metast…
07d1cc8 
…ore tables.
yruslan added a commit that referenced this issue Nov 21, 2024
@yruslan
#520 Fix handling of incremental ingestion for 'raw' format of metast…
fe94be8 
…ore tables. (#521)
yruslan added a commit that referenced this issue Nov 21, 2024
@yruslan
#520 Prepare interfaces for incremental transformer processing.
e2034c1
yruslan added a commit that referenced this issue Nov 22, 2024
@yruslan
#520 Prepare interfaces for incremental transformer processing.
6f944bf
yruslan added a commit that referenced this issue Nov 22, 2024
@yruslan
#520 Make the default batchid field for tables having 'raw' format to…
d90bd7d 
… match raw file ingestion.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@yruslan