Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for maxcompute as source type (batch) #130

Merged
merged 28 commits into from
Feb 11, 2025

Conversation

vinoth-gojek
Copy link
Contributor

@vinoth-gojek vinoth-gojek commented Jan 14, 2025

The changes in MR are to enable MaxCompute as a datasource type for batch jobs (offline to online ingestion)

  1. The MR adds the BATCH_MAXCOMPUTE enum
  2. The changes in sdk enable clients to use maxcompute as source type
  3. The changes include triggering spark jobs to fetch records from maxcompute
  4. The prebuilt custom jar to use custom dialect while calling maxcompute is used to overcome issues

With these changes, record with batch_source maxcompute is added to the datasources table

postgres=> select * from data_sources;
42 | BATCH_MAXCOMPUTE | {"table_ref":"datasource:playground.dsp_maxcompute_dogfood"} | {} | created_timestamp | created_timestamp |

@vinoth-gojek vinoth-gojek force-pushed the DAT-3394_add_MAXCOMPUTE_enum branch from f7c55f6 to bc0658b Compare January 16, 2025 06:44
@vinoth-gojek vinoth-gojek changed the title [WIP][DAT-3394] Add support for batch maxcompute [WIP][DAT-3394] Add support for maxcompute as source type (batch) Jan 16, 2025
@vinoth-gojek vinoth-gojek force-pushed the DAT-3394_add_MAXCOMPUTE_enum branch from e0b3dda to bc0658b Compare January 21, 2025 22:46
@vinoth-gojek vinoth-gojek changed the title [WIP][DAT-3394] Add support for maxcompute as source type (batch) [DAT-3394] Add support for maxcompute as source type (batch) Jan 21, 2025
@vinoth-gojek vinoth-gojek changed the title [DAT-3394] Add support for maxcompute as source type (batch) Add support for maxcompute as source type (batch) Feb 10, 2025
Copy link
Contributor

@deadlycoconuts deadlycoconuts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks for poking around the entire Feast codebase, for picking up scala too and of course testing these changes just to get this PR up! I just left a couple of tiny comments but feel free to merge this PR whenever you're satisfied with the changes 🚀

@vinoth-gojek vinoth-gojek merged commit 9e10e3c into main Feb 11, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants