Add `UsedTableProgressEncoder` #3266

JCZuurmond · 2024-11-13T09:07:07Z

Changes

Add UsedTableProgressEncoder to add the used table to the history log for using it in the migration progress

Linked issues

Progresses #3045
Breaks up #3112

Functionality

modified existing workflow: table-migration-experimental

Tests

modified unit tests
modified integration tests

src/databricks/labs/ucx/hive_metastore/table_ownership.py

github-actions · 2024-11-13T13:02:52Z

❌ 9/112 passed, 103 failed, 1 skipped, 1h20m8s total

❌ test_create_catalog_schema_when_users_group_in_warehouse_acl: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (999ms)

... (skipped 126 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw2] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_all_grant_types: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (1.093s)

... (skipped 53 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw8] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_grant_findings: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (1.348s)

... (skipped 53 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw6] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_external_locations: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (1.393s)

... (skipped 53 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw5] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_grant_ownership: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (619ms)

... (skipped 53 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw8] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_save_external_location_mapping_missing_location: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (513ms)

... (skipped 53 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw5] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_create_catalog_schema_with_principal_acl_aws: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (3.262s)

... (skipped 53 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw7] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_creating_lakeview_dashboard_permissions: TypeError: LakeviewAPI.create() got an unexpected keyword argument 'serialized_dashboard' (180ms)

... (skipped 122 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw1] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_migration_job_ext_hms[regular]: databricks.sdk.errors.platform.ResourceDoesNotExist: Cluster TEST_EXT_HMS_CLUSTER_ID does not exist (2.26s)

... (skipped 86 bytes)
oes not exist
databricks.sdk.errors.platform.ResourceDoesNotExist: Cluster TEST_EXT_HMS_CLUSTER_ID does not exist

❌ test_migrate_external_tables_with_principal_acl_aws_warehouse: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (1.398s)

... (skipped 53 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw1] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_migrate_table_in_mount: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (260ms)

... (skipped 53 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw1] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_migrate_external_tables_with_spn_CLOUD_ENV: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (1.526s)

... (skipped 53 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw1] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_migration_index_deleted_source: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (286ms)

... (skipped 126 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw1] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_table_migration_ownership: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (272ms)

... (skipped 53 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw1] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_fallback_admin_user: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (8.737s)

... (skipped 54576 bytes)
47659649002239')], id='8957403854939895', name=None, roles=[], schemas=None, user_name='[email protected]')]

❌ test_move_views: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (839ms)

... (skipped 126 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw0] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_move_tables: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (7.981s)

... (skipped 126 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw1] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_alias_tables: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (5.857s)

... (skipped 126 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw0] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_move_tables_table_properties_mismatch_preserves_original: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (737ms)

... (skipped 126 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw0] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_describe_all_tables_in_databases: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (247ms)

... (skipped 53 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw0] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_describe_all_udfs_in_databases: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (265ms)

... (skipped 53 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw0] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_udf_ownership: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (263ms)

... (skipped 53 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw0] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_table_migration_job_refreshes_migration_status[regular-migrate-tables]: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (2.137s)

... (skipped 74 bytes)
E_ID was not found.
databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found.

❌ test_table_migration_job_refreshes_migration_status[hiveserde-migrate-external-hiveserde-tables-in-place-experimental]: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (1.654s)

... (skipped 74 bytes)
E_ID was not found.
databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found.

❌ test_table_migration_job_refreshes_migration_status[hiveserde-migrate-external-tables-ctas]: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (1.639s)

... (skipped 74 bytes)
E_ID was not found.
databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found.

❌ test_table_migration_for_managed_table[managed-migrate-tables]: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (1.674s)

... (skipped 74 bytes)
E_ID was not found.
databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found.

❌ test_hiveserde_table_in_place_migration_job[hiveserde]: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (1.64s)

... (skipped 74 bytes)
E_ID was not found.
databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found.

❌ test_hiveserde_table_ctas_migration_job[hiveserde]: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (2.056s)

... (skipped 74 bytes)
E_ID was not found.
databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found.

❌ test_table_migration_job_publishes_remaining_tables[regular]: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (2.471s)

... (skipped 74 bytes)
E_ID was not found.
databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found.

❌ test_progress_tracking_installer_creates_table[workflow_runs]: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (589ms)

... (skipped 126 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw0] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_progress_tracking_installer_creates_table[historical]: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (535ms)

... (skipped 126 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw0] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_workflow_run_recorder_records_workflow_run: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (1.754s)

... (skipped 147 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw0] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_migrate_managed_tables: TimeoutError: Timed out after 0:02:00 (2m3.981s)

... (skipped 70 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw8] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_migrate_tables_with_cache_should_not_create_table: TimeoutError: Timed out after 0:02:00 (2m3.339s)

... (skipped 70 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw2] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_migrate_external_table_hiveserde_ctas: TimeoutError: Timed out after 0:02:00 (2m5.425s)

... (skipped 70 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw5] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_create_ucx_catalog_creates_catalog: TimeoutError: Timed out after 0:02:00 (2m7.332s)

... (skipped 70 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw4] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_move_tables_no_to_schema: TimeoutError: Timed out after 0:02:00 (2m4.035s)

... (skipped 70 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw1] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_create_all_catalogs_schemas: TimeoutError: Timed out after 0:03:00 (3m9.503s)

... (skipped 70 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw3] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_create_catalog_schema_with_legacy_hive_metastore_privileges: TimeoutError: Timed out after 0:03:00 (3m8.673s)

... (skipped 70 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw7] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_migrate_managed_tables_with_acl: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (314ms)

... (skipped 126 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw7] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_create_catalog_schema_with_principal_acl_CLOUD_ENV: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (4.963s)

... (skipped 53 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw3] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_query_dfsa_ownership: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (284ms)

... (skipped 53 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw3] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_path_dfsa_ownership: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (344ms)

... (skipped 53 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw3] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_migrate_external_tables_with_principal_acl_CLOUD_ENV: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (2.836s)

... (skipped 53 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw7] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_migrate_external_tables_with_principal_acl_aws: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (1.255s)

... (skipped 53 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw7] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_job_task_linter_library_not_installed_cluster: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (3.227s)

... (skipped 147 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw3] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_job_task_linter_library_installed_cluster: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (2.774s)

... (skipped 147 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw3] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_job_linter_some_notebook_graph_with_problems: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (2.972s)

... (skipped 147 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw3] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_workflow_linter_lints_job_with_import_pypi_library: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (3.535s)

... (skipped 147 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw3] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_lint_local_code: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (28.922s)

... (skipped 1503200 bytes)
 at 0x7f9364274760>.
13:40 DEBUG [databricks.labs.ucx.source_code.linters.directfs] Could not infer value of *row

❌ test_migrate_dbfs_non_delta_tables: TimeoutError: Timed out after 0:02:00 (2m3.232s)

... (skipped 70 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw8] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_graph_computes_magic_run_route_recursively_in_parent_folder[order0]: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (2.069s)

... (skipped 126 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw3] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_migrate_external_table: TimeoutError: Timed out after 0:02:00 (2m5.762s)

... (skipped 70 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw2] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_graph_computes_magic_run_route_recursively_in_parent_folder[order1]: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (1.478s)

... (skipped 126 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw3] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_relative_grand_parent_path_is_supported: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (4.53s)

... (skipped 126 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw8] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_graph_computes_magic_run_route_recursively_in_parent_folder[order3]: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (1.552s)

... (skipped 126 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw3] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_migrate_managed_table_to_external_table_without_conversion: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (1.81s)

... (skipped 126 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw2] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_revert_migrated_table: TimeoutError: Timed out after 0:02:00 (2m4.876s)

... (skipped 70 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw4] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_graph_computes_magic_run_route_recursively_in_parent_folder[order2]: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (1.224s)

... (skipped 126 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw8] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_graph_computes_magic_run_route_recursively_in_parent_folder[order4]: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (1.832s)

... (skipped 126 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw2] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_graph_computes_magic_run_route_recursively_in_parent_folder[order5]: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (2.169s)

... (skipped 126 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw3] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_workflow_linter_lints_job_with_workspace_requirements_dependency: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (3.387s)

... (skipped 147 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw8] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_workflow_linter_lints_job_with_dbfs_requirements_dependency: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (3.506s)

... (skipped 147 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw3] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_workflow_linter_lints_job_with_workspace_egg_dependency: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (4.236s)

... (skipped 147 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw2] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_workflow_linter_lints_job_with_wheel_dependency: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (545ms)

... (skipped 147 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw3] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_workflow_linter_lints_job_with_dbfs_egg_dependency: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (3.262s)

... (skipped 147 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw8] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_job_spark_python_task_linter_unhappy_path: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (2.627s)

... (skipped 147 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw3] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_job_spark_python_task_linter_happy_path: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (2.423s)

... (skipped 147 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw8] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_partitioned_tables: TimeoutError: Timed out after 0:02:00 (2m4.862s)

... (skipped 70 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw1] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_table_ownership: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (280ms)

... (skipped 53 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw1] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_job_spark_python_task_workspace_linter_happy_path: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (2.201s)

... (skipped 147 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw3] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_job_spark_python_task_dbfs_linter_happy_path: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (3.023s)

... (skipped 147 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw8] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_job_dlt_task_linter_happy_path: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (2.405s)

... (skipped 147 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw1] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_job_spark_python_task_linter_notebook_handling: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (2.937s)

... (skipped 147 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw3] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_job_dlt_task_linter_unhappy_path: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (2.533s)

... (skipped 147 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw8] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_job_dependency_problem_egg_dbr14plus: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (2.784s)

... (skipped 147 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw1] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_build_notebook_dependency_graphs_installs_wheel_with_pip_cell_in_notebook[pip_install_demo_wheel]: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (1.854s)

... (skipped 1392 bytes)
alled thingy-0.0.1


13:40 DEBUG [databricks.labs.ucx.source_code.linters.files] Resolving unknown import: thingy

❌ test_build_notebook_dependency_graphs_installs_wheel_with_pip_cell_in_notebook[pip_install_demo_wheel_and_pytest]: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (1.672s)

... (skipped 3560 bytes)
-0.0.1 tomli-2.1.0


13:40 DEBUG [databricks.labs.ucx.source_code.linters.files] Resolving unknown import: thingy

❌ test_build_notebook_dependency_graphs_installs_wheel_with_pip_cell_in_notebook[pip_install_demo_wheel_with_target_directory]: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (1.318s)

... (skipped 1460 bytes)
alled thingy-0.0.1


13:40 DEBUG [databricks.labs.ucx.source_code.linters.files] Resolving unknown import: thingy

❌ test_build_notebook_dependency_graphs_installs_wheel_with_pip_cell_in_notebook[pip_install_demo_wheel_many_flags]: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (1.417s)

... (skipped 1642 bytes)
alled thingy-0.0.1


13:40 DEBUG [databricks.labs.ucx.source_code.linters.files] Resolving unknown import: thingy

❌ test_build_notebook_dependency_graphs_fails_installing_when_spaces[pip_install_demo_wheel_with_spaces_in_target_directory]: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (1.25s)

... (skipped 1474 bytes)
alled thingy-0.0.1


13:40 DEBUG [databricks.labs.ucx.source_code.linters.files] Resolving unknown import: thingy

❌ test_build_notebook_dependency_graphs_when_installing_notebooks_twice[pip_install_demo_wheel]: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (1.609s)

... (skipped 2658 bytes)
alled thingy-0.0.1


13:40 DEBUG [databricks.labs.ucx.source_code.linters.files] Resolving unknown import: thingy

❌ test_build_notebook_dependency_graphs_installs_pytest_from_index_url: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (4.474s)

... (skipped 3064 bytes)

Successfully installed exceptiongroup-1.2.2 iniconfig-2.0.0 packaging-24.2 pluggy-1.5.0 pytest-8.3.3 tomli-2.1.0

❌ test_build_notebook_dependency_graphs_when_installing_notebooks_twice[pip_install_pytest_with_index_url]: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (3.517s)

... (skipped 6006 bytes)

Successfully installed exceptiongroup-1.2.2 iniconfig-2.0.0 packaging-24.2 pluggy-1.5.0 pytest-8.3.3 tomli-2.1.0

❌ test_build_notebook_dependency_graphs_when_installing_pytest_twice: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (2.631s)

... (skipped 838 bytes)
d: ['pip', '--disable-pip-version-check', 'install', 'pytest', '--quiet', '-t', '/tmp/ucx-pac4n971', '--upgrade']

❌ test_fix_dashboard: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (695ms)

... (skipped 147 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw8] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_query_linter_lints_queries_and_stores_dfsas_and_tables: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (4.298s)

... (skipped 771 bytes)
shboard Created ucx_DhpA6_ra78a57d37: https://DATABRICKS_HOST/sql/dashboards/5ad006f5-7e14-4bca-af7c-538b21e77a2c

❌ test_build_notebook_dependency_graphs_installs_pypi_packages: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (32.417s)

... (skipped 1252 bytes)
.
databricks-labs-ucx 0.49.0 requires sqlglot<25.30,>=25.5.0, but you have sqlglot 25.30.0 which is incompatible.

❌ test_migrate_view: TimeoutError: Timed out after 0:03:00 (3m7.617s)

... (skipped 70 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw5] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_linter_from_context: TimeoutError: Timed out after 0:02:00 (2m14.313s)

... (skipped 91 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw7] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_job_linter_no_problems: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (2.458s)

... (skipped 147 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw7] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_all_grants_in_databases: TimeoutError: Timed out after 0:03:00 (5m38.601s)

... (skipped 70 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw6] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_all_grants_for_udfs_in_databases: TimeoutError: Timed out after 0:03:00 (5m40.523s)

... (skipped 70 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw9] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_migrate_managed_table_to_external_table_with_clone: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (2.238s)

... (skipped 126 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw6] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_build_notebook_dependency_graphs_when_installing_notebooks_twice[pip_install_multiple_packages]: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (1m3.016s)

... (skipped 2380 bytes)
.
databricks-labs-ucx 0.49.0 requires sqlglot<25.30,>=25.5.0, but you have sqlglot 25.30.0 which is incompatible.

❌ test_migrate_external_table_failed_sync: TimeoutError: Timed out after 0:01:00 (1m0.495s)

... (skipped 70 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw6] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_migrate_view_alias_test: TimeoutError: Timed out after 0:03:00 (3m7.443s)

... (skipped 70 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw5] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_migrate_external_table_hiveserde_in_place: TimeoutError: Timed out after 0:02:00 (2m5.51s)

... (skipped 70 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw6] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_mapping_skips_tables_databases: TimeoutError: Timed out after 0:05:00 (5m4.601s)

... (skipped 70 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw4] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_all_grants_for_other_objects: TimeoutError: Timed out after 0:03:00 (5m19.419s)

... (skipped 70 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw9] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_mapping_reverts_table: TimeoutError: Timed out after 0:02:00 (2m5.85s)

... (skipped 70 bytes)
ome/runner/work/ucx/ucx/.venv/bin/python
[gw4] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_running_real_migration_progress_job: TimeoutError: Timed out after 0:12:00 (12m6.626s)

... (skipped 91 bytes)
e DATABRICKS_WAREHOUSE_ID was not found.
[gw0] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_malformed_pip_cell_is_supported: databricks.sdk.errors.platform.NotFound: The warehouse DATABRICKS_WAREHOUSE_ID was not found. (6.882s)

... (skipped 2514 bytes)
ompatible.
types-requests 2.31.0.20240406 requires urllib3>=2, but you have urllib3 1.26.9 which is incompatible.

_{Running from acceptance #7368}

nfx

don't do indirect dependencies.
it makes no sense to have a parametrised testcase with just a single line in a test table.

src/databricks/labs/ucx/contexts/application.py

nfx · 2024-11-13T15:00:12Z

src/databricks/labs/ucx/framework/owners.py

@@ -201,7 +201,7 @@ def __init__(self, administrator_locator: AdministratorLocator, ws: WorkspaceCli
        super().__init__(administrator_locator)
        self._ws = ws

-    def owner_of_path(self, path: str) -> str:
+    def owner_of_path(self, path: str) -> str:  # TODO: Why is this `owner_of_path` and not


and not what?.. it'll become an issue after being merged and there's no clear description of the issue.

src/databricks/labs/ucx/hive_metastore/table_migrate.py

src/databricks/labs/ucx/hive_metastore/table_ownership.py

tests/unit/progress/test_tables.py

asnare

I haven't reviewed the implementation, but have a rather more fundamental question: should this encoder (for UsedTable) actually exist?

To me this looks like what we had with TableMigrationStatus in that these records need to be attached to the progress of the logical thing that contains the reference to the used table.

Guessing a bit, but we can have these for:

Queries (→ Dashboards)
Paths (→ Jobs)
Notebooks (→ Jobs)

JCZuurmond · 2024-11-14T13:34:35Z

#3266 (review) @asnare, that is a good question, when do we attach it to a different object and when not? Note that the UsedTable is a linting resource, where migration status was not

@nfx : What do you think?

… write

nfx

attach UsedTable names to jobs/pipelines/queries/views only if those tables are in HMS. UsedTable is an internal entity for linting.

JCZuurmond · 2024-11-19T13:00:15Z

Will tackle the changes in another PR

JCZuurmond added the feat/migration-progress Issues related to the migration progress workflow label Nov 13, 2024

JCZuurmond self-assigned this Nov 13, 2024

JCZuurmond requested a review from nfx November 13, 2024 10:45

JCZuurmond force-pushed the feat/add-used-table-progress=encoder branch from 7c19acf to 0fba883 Compare November 13, 2024 10:48

JCZuurmond marked this pull request as ready for review November 13, 2024 10:48

JCZuurmond requested a review from a team as a code owner November 13, 2024 10:48

JCZuurmond had a problem deploying to account-admin November 13, 2024 10:48 — with GitHub Actions Error

JCZuurmond commented Nov 13, 2024

View reviewed changes

src/databricks/labs/ucx/hive_metastore/table_ownership.py Show resolved Hide resolved

JCZuurmond had a problem deploying to account-admin November 13, 2024 11:15 — with GitHub Actions Failure

JCZuurmond requested a review from asnare November 13, 2024 14:07

nfx requested changes Nov 13, 2024

View reviewed changes

JCZuurmond mentioned this pull request Nov 13, 2024

Add QueryProblemProgressEncoder #3267

Open

5 tasks

asnare reviewed Nov 13, 2024

View reviewed changes

JCZuurmond added 15 commits November 14, 2024 14:35

Move TableOwnership to separate module

9aebf2a

Move table migration ownership to table_migrate module

1add260

Add UsedTableOwnership

a989e73

Test UsedTableOwnership to fall back on workspace administrator

b26bde9

Test UsedTableOwnership to fallback on workspace administrator if not…

0f90dc5

… write

Test UsedTableOwnership gets owner from query

cd758f0

Format

6398f39

Move used table to fixture

e91a283

Test propagating used table ownership from code

9d90fb3

Test unknown code source type fallback to workspace admin

45cdcd5

Format

14ebeec

Add asserts on autospec mocks

5fd8a63

Add docstring to UsedTableOwnership

a6c5deb

Format

059d0f7

Add from_table classmethod to UsedTable

35177f8

JCZuurmond added 11 commits November 14, 2024 14:35

Reuse UsedTableOwnership in TableOwnership

a0f016e

Add UsedTableProgressEncoder

7d4b712

Add UsedTableProgressEncoder to RuntimeContext

bdc4233

Add UsedTableProgressEncoder to MigrationProgress workflow

57df37f

Move Used table unit tests to unit tests module

709e3fd

Move base unit tests to unit tests module

fd95c15

Format

1f733a8

Add missing part to comment

0f5ae17

Avoid indirect administrator locator

b7f32de

Format

f07aebb

Remove redundant pytest.mark.paramtrize

61bf68c

JCZuurmond force-pushed the feat/add-used-table-progress=encoder branch from fac963f to 61bf68c Compare November 14, 2024 13:35

JCZuurmond had a problem deploying to account-admin November 14, 2024 13:35 — with GitHub Actions Failure

JCZuurmond requested a review from nfx November 14, 2024 13:35

This was referenced Nov 15, 2024

Add linting resources to migration progress dashboard #3314

Draft

Add DirectFsAccess progress encoder #3315

Closed

nfx requested changes Nov 18, 2024

View reviewed changes

JCZuurmond requested a review from a team as a code owner November 19, 2024 12:56

Remove UsedTableProgressEncoder

743847d

JCZuurmond force-pushed the feat/add-used-table-progress=encoder branch from fcd5286 to 743847d Compare November 19, 2024 12:57

JCZuurmond closed this Nov 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `UsedTableProgressEncoder` #3266

Add `UsedTableProgressEncoder` #3266

JCZuurmond commented Nov 13, 2024 •

edited

Loading

github-actions bot commented Nov 13, 2024 •

edited

Loading

nfx left a comment

nfx Nov 13, 2024

JCZuurmond Nov 13, 2024

asnare left a comment

JCZuurmond commented Nov 14, 2024

nfx left a comment

JCZuurmond commented Nov 19, 2024

Add UsedTableProgressEncoder #3266

Add UsedTableProgressEncoder #3266

Conversation

JCZuurmond commented Nov 13, 2024 • edited Loading

Changes

Linked issues

Functionality

Tests

github-actions bot commented Nov 13, 2024 • edited Loading

nfx left a comment

Choose a reason for hiding this comment

nfx Nov 13, 2024

Choose a reason for hiding this comment

JCZuurmond Nov 13, 2024

Choose a reason for hiding this comment

asnare left a comment

Choose a reason for hiding this comment

JCZuurmond commented Nov 14, 2024

nfx left a comment

Choose a reason for hiding this comment

JCZuurmond commented Nov 19, 2024

Add `UsedTableProgressEncoder` #3266

Add `UsedTableProgressEncoder` #3266

JCZuurmond commented Nov 13, 2024 •

edited

Loading

github-actions bot commented Nov 13, 2024 •

edited

Loading