Group migration workflow improvements #3391

JCZuurmond · 2024-11-27T15:59:29Z

Changes

Group migration workflow improvements:

Return early when include_databases is provided
Fix unavailable external catalog on tacl cluster
Check outcome of verify permissions
Setup tacl cluster
Log what to do when group permissions did not verify successfully
Merge duplicate tests

Functionality

modified existing command: databricks labs ucx group-migration-legacy

Tests

modified integration tests

github-actions · 2024-11-27T16:24:00Z

❌ 116/117 passed, 3 flaky, 1 failed, 7 skipped, 1h48m23s total

❌ test_running_real_migrate_groups_job: databricks.sdk.errors.sdk.OperationFailed: failed to reach TERMINATED or SKIPPED, got RunLifeCycleState.INTERNAL_ERROR: Task apply_permissions failed with message: Workload failed, see run output for details. (2m56.324s)

databricks.sdk.errors.sdk.OperationFailed: failed to reach TERMINATED or SKIPPED, got RunLifeCycleState.INTERNAL_ERROR: Task apply_permissions failed with message: Workload failed, see run output for details.
[gw2] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python
09:13 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.eS6G/config.yml) doesn't exist.
09:13 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
09:13 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
09:13 INFO [databricks.labs.ucx.install] Fetching installations...
09:13 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
09:13 DEBUG [tests.integration.conftest] Waiting for clusters to start...
09:13 DEBUG [tests.integration.conftest] Waiting for clusters to start...
09:13 INFO [databricks.labs.ucx.install] Installing UCX v0.50.1+4020241128091336
09:13 INFO [databricks.labs.ucx.install] Creating ucx schemas...
09:13 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-legacy
09:13 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
09:13 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
09:13 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
09:13 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
09:13 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
09:13 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
09:13 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migration-progress-experimental
09:13 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
09:13 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
09:13 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
09:13 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
09:13 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
09:13 INFO [databricks.labs.ucx.install] Creating dashboards...
09:13 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/views...
09:13 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment...
09:13 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration...
09:13 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/progress...
09:13 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/interactive...
09:13 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/estimates...
09:13 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/main...
09:13 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/CLOUD_ENV...
09:13 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/groups...
09:13 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/main...
09:13 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/progress/main...
09:13 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
09:13 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
09:13 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
09:13 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
09:13 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
09:13 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
09:13 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
09:13 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.eS6G/README for the next steps.
09:13 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_seoun.tables] ignoring any existing tables inventory; refresh is forced.
09:13 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_seoun.tables] crawling new set of snapshot data for tables
09:13 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.dummy_sezon] listing tables and views
09:13 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.dummy_sezon.dummy_tvist] fetching table metadata
09:13 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_seoun.tables] found 1 new records for tables
09:13 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-groups job: https://DATABRICKS_HOST#job/1017744430828978
09:13 INFO [databricks.labs.ucx.installer.workflows] Started migrate-groups job: https://DATABRICKS_HOST#job/1017744430828978/runs/673618068368992
09:13 DEBUG [databricks.labs.ucx.installer.workflows] Validating migrate-groups workflow: https://DATABRICKS_HOST#job/1017744430828978
09:13 INFO [databricks.labs.ucx.installer.workflows] Identified a run in progress waiting for run completion
09:13 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.eS6G/config.yml) doesn't exist.
09:13 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
09:13 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
09:13 INFO [databricks.labs.ucx.install] Fetching installations...
09:13 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
09:13 DEBUG [tests.integration.conftest] Waiting for clusters to start...
09:13 DEBUG [tests.integration.conftest] Waiting for clusters to start...
09:13 INFO [databricks.labs.ucx.install] Installing UCX v0.50.1+4020241128091336
09:13 INFO [databricks.labs.ucx.install] Creating ucx schemas...
09:13 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-legacy
09:13 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
09:13 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
09:13 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
09:13 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
09:13 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
09:13 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
09:13 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migration-progress-experimental
09:13 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
09:13 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
09:13 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
09:13 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
09:13 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
09:13 INFO [databricks.labs.ucx.install] Creating dashboards...
09:13 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/views...
09:13 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment...
09:13 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration...
09:13 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/progress...
09:13 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/interactive...
09:13 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/estimates...
09:13 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/main...
09:13 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/CLOUD_ENV...
09:13 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/groups...
09:13 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/main...
09:13 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/progress/main...
09:13 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
09:13 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
09:13 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
09:13 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
09:13 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
09:13 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
09:13 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
09:13 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.eS6G/README for the next steps.
09:13 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_seoun.tables] ignoring any existing tables inventory; refresh is forced.
09:13 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_seoun.tables] crawling new set of snapshot data for tables
09:13 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.dummy_sezon] listing tables and views
09:13 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.dummy_sezon.dummy_tvist] fetching table metadata
09:13 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_seoun.tables] found 1 new records for tables
09:13 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-groups job: https://DATABRICKS_HOST#job/1017744430828978
09:13 INFO [databricks.labs.ucx.installer.workflows] Started migrate-groups job: https://DATABRICKS_HOST#job/1017744430828978/runs/673618068368992
09:13 DEBUG [databricks.labs.ucx.installer.workflows] Validating migrate-groups workflow: https://DATABRICKS_HOST#job/1017744430828978
09:13 INFO [databricks.labs.ucx.installer.workflows] Identified a run in progress waiting for run completion
09:16 INFO [databricks.labs.ucx.install] Deleting UCX v0.50.1+4020241128091336 from https://DATABRICKS_HOST
09:16 INFO [databricks.labs.ucx.install] Deleting inventory database dummy_seoun
09:16 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=218764972043875, as it is no longer needed
09:16 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=368873517396685, as it is no longer needed
09:16 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=733793098905662, as it is no longer needed
09:16 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=223306928807772, as it is no longer needed
09:16 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=1017744430828978, as it is no longer needed
09:16 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=47617835611039, as it is no longer needed
09:16 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=385820953669390, as it is no longer needed
09:16 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=1052617503293399, as it is no longer needed
09:16 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=989738977722536, as it is no longer needed
09:16 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=429852906512322, as it is no longer needed
09:16 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=652618868418263, as it is no longer needed
09:16 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=1016889930294445, as it is no longer needed
09:16 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=983432421926357, as it is no longer needed
09:16 INFO [databricks.labs.ucx.install] Deleting cluster policy
09:16 INFO [databricks.labs.ucx.install] Deleting secret scope
09:16 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
[gw2] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

Flaky tests:

🤪 test_table_migration_job_publishes_remaining_tables[regular] (3m8.301s)
🤪 test_hiveserde_table_in_place_migration_job[hiveserde] (4m25.751s)
🤪 test_table_migration_job_refreshes_migration_status[hiveserde-migrate-external-tables-ctas] (4m29.661s)

_{Running from acceptance #7563}

… in legacy group migration workflow

JCZuurmond added 24 commits November 27, 2024 10:11

Test group existence in migrate groups integration test

5ce4442

Type hint test

b2a0974

Add TODO

4c31276

Add validate workflow

04685ad

Add skip job wait

7012628

Add setup tacl to migrate-groups-legacy workflow

0b74d65

Let workspace and account group function group manager

ef959e0

Move check for original workspace group up

a38cfc3

Fix test

13f716d

Add TODO's

204ec50

Name variables more clear and fix condition

89e6c73

Remove redundant check of account group to exist

b5eee38

Move include databases check out of try-except

1f1aa30

Fix test

e32cb21

Check if all group permission validation came back positive

5e30d12

Raise value error in validate group permission steps when not succesful

bb9e059

Format

0be6389

Merge duplicate tests

192f4d3

Remove redundant include group names

d13bc78

Remove redundant import

940c5a8

Fix logging verify group permissions

239b6a9

Fix threads.gather

08e599f

Make log error a warning

ecdbf19

Format

8b4400b

JCZuurmond added the internal this pull request won't appear in release notes label Nov 27, 2024

JCZuurmond requested a review from a team as a code owner November 27, 2024 15:59

JCZuurmond temporarily deployed to account-admin November 27, 2024 15:59 — with GitHub Actions Inactive

JCZuurmond requested a review from asnare November 27, 2024 15:59

JCZuurmond self-assigned this Nov 27, 2024

JCZuurmond requested a review from FastLee November 27, 2024 15:59

Reference validate-group-permissions workflow when validation fails…

641b211

… in legacy group migration workflow

JCZuurmond had a problem deploying to account-admin November 28, 2024 08:58 — with GitHub Actions Failure

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Group migration workflow improvements #3391

Group migration workflow improvements #3391

JCZuurmond commented Nov 27, 2024 •

edited

Loading

github-actions bot commented Nov 27, 2024 •

edited

Loading

Group migration workflow improvements #3391

Are you sure you want to change the base?

Group migration workflow improvements #3391

Conversation

JCZuurmond commented Nov 27, 2024 • edited Loading

Changes

Functionality

Tests

github-actions bot commented Nov 27, 2024 • edited Loading

JCZuurmond commented Nov 27, 2024 •

edited

Loading

github-actions bot commented Nov 27, 2024 •

edited

Loading