Feature/v0.0.9 #109

ravi-databricks · 2024-10-24T18:29:16Z

Merging feature0.0.9 changes

1.use uc volume paths to push config and resources 2.serverless for onboarding job and dlt

1.Updated all demo's to include unity catalog volumes path instead of dbfs 2.Updated all demo's to use serverless 3.Updated docs for demo options 4.Added demo workflow png files for DAIS and Techsummit demo

2.Added open_job_url function

1.CLI to include serverless for onboarding job 2.CLI to include uc volumes instead of dbfs for uc

1.Incorporated applyChangesFromSnapshot attribute into onboarding 2.Dataflowspec modified to incorporate applyChangesFromSnapshot 3.Unit tests

1.Removed cloud_provider variable from demo 2.Removed dbfs path from demos 3.Modified cloud_provider inside integration tests 4.Modified dbfs path inside inside integration tests

Merging integration tests with serverless and uc volume

…be made directly in editor and don't need to be modified in Databricks and downloaded as .dbc files

…mplifying configs

… and simplification

1.Integration test workflow for apply_changes_from_snapshot 2.Fixed lambda function for next_snapshot_version

Integration Tests and Demo rework

UC Volume + Serverless support for CLI, Integration tests and Demos

1.Onboarding merge issues 2.Unit tests

Revert "Issue 94"

ravi-databricks and others added 30 commits September 25, 2024 12:09

Added uc volume for fanout demo

1fdb82c

Added uc volumes for fanout demo

9ca140c

Added:

0db98c8

1.use uc volume paths to push config and resources 2.serverless for onboarding job and dlt

Removed linting errors

cbf8b29

Added:

3bd552a

1.Updated all demo's to include unity catalog volumes path instead of dbfs 2.Updated all demo's to use serverless 3.Updated docs for demo options 4.Added demo workflow png files for DAIS and Techsummit demo

Added uc_volume_path for eventhub demo

edd2bf1

1.Fixed uc volume path for eventhub

8529333

2.Added open_job_url function

Fixed int tests with uc volume paths and use serverless

d1aa961

Added:

514a974

1.CLI to include serverless for onboarding job 2.CLI to include uc volumes instead of dbfs for uc

Added basic implementation for apply_changes_from_snapshot

6cd66ad

Added:

81a8b1b

1.Incorporated applyChangesFromSnapshot attribute into onboarding 2.Dataflowspec modified to incorporate applyChangesFromSnapshot 3.Unit tests

Updated snapshot initializing exception handling

9b7b406

Corrected:

437ed56

1.Removed cloud_provider variable from demo 2.Removed dbfs path from demos 3.Modified cloud_provider inside integration tests 4.Modified dbfs path inside inside integration tests

added integration test outputs to gitinore

7a6b28f

Corrected dataflowpipeline apply_from_snapshot api

a75ccba

Merge pull request #99 from databrickslabs/issue_97

36ed8ad

Merging integration tests with serverless and uc volume

uc catalog name consistency in sample code

8fc401d

updated runner notebook upload to .py files from .dbc so updates can …

04adc91

…be made directly in editor and don't need to be modified in Databricks and downloaded as .dbc files

lower the source during the arg parse instead of at each time it is used

4b9bffc

Complete rewrite of the arg parser for the intgegration tests

5a38c85

Update args parsing

72fe9dc

begin movign to uc volumes only and getting rid of dbfs as well as si…

c337f59

…mplifying configs

uc resource generation update

3e1e486

continued unused file, and dbfs code removal along with code clean-up…

6a8eb47

… and simplification

Added:

6e1d831

1.Integration test workflow for apply_changes_from_snapshot 2.Fixed lambda function for next_snapshot_version

update upload to databricks

f360160

only upload wheel to workspace location

c678ac8

added uc volume upload of the wheel if provided

d6020b5

wheel upload added, all data uploads reworked

acee79b

initial upload and setup for integration testing for cloud files is done

d316554

dvanderwood and others added 29 commits October 7, 2024 14:17

cloud files testing, added back accidently removed json

5ee1843

cloud files integration test works

e70f85d

formatting

9e77b1f

continuing code simplication between 3 integration run types

e78c751

redundant doc strings removal

1cdb503

remove early exit

37652f9

eventhub_accesskey_name updates

a6249fe

fixed job create

c740093

fixed clean up

b769973

redundant message removed

f6bb5fe

formatting

381531c

non cloud files testing, niether can be confirmed right now

eb44fa5

af cloud demo update

f6954fb

formatting

ec9a317

formatting

ae27548

linting fixes

a60d0e2

formatting and linting

7b17edf

removed match syntax since testing on python 3.9

0d9975a

removed other match statement

386db9d

uc volume path for onboarding file test

05e8b3a

Merge pull request #101 from dvanderwood/issue_97_98

b87620a

Integration Tests and Demo rework

fixed cli related tests and added new unit test for onboard and deploy

27dadc8

Added unit test coverage for cli.py

0da22ce

Merge branch 'issue_86' into issue_97

93b6fa4

Merge pull request #105 from databrickslabs/issue_97

f134a71

UC Volume + Serverless support for CLI, Integration tests and Demos

fixed:

4b4bc91

1.Onboarding merge issues 2.Unit tests

Merge branch 'issue_94' into feature/v0.0.9

024ef5a

Revert "Issue 94"

d0e389d

Merge pull request #110 from databrickslabs/revert-108-issue_94

4307fdf

Revert "Issue 94"

ravi-databricks merged commit 6c8799c into issue_94 Oct 24, 2024
1 check failed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/v0.0.9 #109

Feature/v0.0.9 #109

ravi-databricks commented Oct 24, 2024

Feature/v0.0.9 #109

Feature/v0.0.9 #109

Conversation

ravi-databricks commented Oct 24, 2024