Release v0.33.0

* Added `validate-table-locations` command for checking overlapping tables across workspaces ([#2341](#2341)). A new command, `validate-table-locations`, has been added to check for overlapping table locations across workspaces before migrating tables. This command is intended to ensure that tables can be migrated across workspaces without issues. The new command is part of the table migration workflows and uses a `LocationTrie` data structure to efficiently search for overlapping table locations. If any overlaps are found, the command logs a warning message and adds the conflicting tables to a list of all conflicts. This list is returned at the end of the command. The `validate-table-locations` command is intended to be run before migrating tables to ensure that the tables can be migrated without conflicts. The command includes a `workspace-ids` flag, which allows users to specify a list of workspace IDs to include in the validation. If this flag is not provided, the command will include all workspaces present in the account. This new command resolves issue [#673](#673). The `validate_table_locations` method is added to the `AccountAggregate` class and the `ExternalLocations` class has been updated to use the new `LocationTrie` class. The import section has also been updated to include new modules such as `LocationTrie` and `Table` from `databricks.labs.ucx.hive_metastore.locations` and `databricks.labs.ucx.hive_metastore.tables` respectively. Additionally, test cases have been added to ensure the correct functioning of the `LocationTrie` class.
* Added references to hive_metastore catalog in all table references an… ([#2419](#2419)). In this release, we have updated various methods and functions across multiple files to include explicit references to the `hive_metastore` catalog in table references. This change aims to improve the accuracy and consistency of table references in the codebase, enhancing reliability and maintainability. Affected files include `azure.py`, `init_scripts.py`, `pipelines.py`, and others in the `databricks/labs/ucx/assessment` module, as well as test files in the `tests/unit/assessment` and `tests/unit/azure` directories. The `_try_fetch` method has been updated to include the catalog name in table references in all instances, ensuring the correct catalog is referenced in all queries. Additionally, various test functions in affected files have been updated to reference the `hive_metastore` catalog in SQL queries. This update is part of the resolution of issue [#2207](#2207) and promotes robust handling of catalog, schema, and table naming scenarios in hive metastore migration status management.
* Added support for skipping views when migrating tables and views ([#2343](#2343)). In this release, we've added support for skipping both tables and views during the migration process in the `databricks labs ucx` command, addressing issue [#1937](#1937). The `skip` command has been enhanced to support skipping views, and new functions `skip_table_or_view` and `load_one` have been introduced to the `Table` class. Appropriate error handling and tests, including unit tests and integration tests, have been implemented to ensure the functionality works as expected. With these changes, users can now skip views during migration and have more flexibility when working with tables in the Unity Catalog.
* Avoid false positives when linting for pyspark patterns ([#2381](#2381)). This release includes enhancements to the PySpark linter aimed at reducing false positives during linting. The linter has been updated to check the originating module when detecting PySpark calls, ensuring that warnings are triggered only for relevant nodes from the pyspark or dbutils modules. Specifically, the `ReturnValueMatcher` and `DirectFilesystemAccessMatcher` classes have been modified to include this new check. These changes improve the overall accuracy of the PySpark linter, ensuring that only pertinent warnings are surfaced during linting. Additionally, the commit includes updated unit tests to verify the correct behavior of the modified linter. Specific improvements have been made to avoid false positives when detecting the `listTables` function in the PySpark catalog, ensuring that the warning is only triggered for the actual PySpark `listTables` method call.
* Bug: Generate custom warning when doing table size check and encountering DELTA_INVALID_FORMAT exception ([#2426](#2426)). A modification has been implemented in the `_safe_get_table_size` method within the `table_size.py` file of the `hive_metastore` package. This change addresses an issue ([#1913](#1913)) concerning the occurrence of a `DELTA_INVALID_FORMAT` exception while determining the size of a Delta table. Instead of raising an error, the exception is now converted into a warning, and the function proceeds to process the rest of the table. A corresponding warning message has been added to inform users about the issue and suggest checking the table structure. No new methods have been introduced, and existing functionalities have been updated to handle this specific exception more gracefully. The changes have been thoroughly tested with unit tests for the table size check when encountering a `DELTA_INVALID_FORMAT` error, employing a mock backend and a mock Spark session to simulate the error conversion. This change does not affect user documentation, CLI commands, workflows, or tables, and is solely intended for software engineers adopting the project.
* Clean up left over uber principal resources for Azure ([#2370](#2370)). This commit includes modifications to the Azure access module of the UCX project to clean up resources if the creation of the uber principal fails midway. It addresses issues [#2360](#2360) (Azure part) and [#2363](#2363), and modifies the command `databricks labs ucx create-uber-principal` to include this functionality. The changes include adding new methods and modifying existing ones for working with Azure resources, such as `StorageAccount`, `AccessConnector`, and `AzureRoleAssignment`. Additionally, new unit and integration tests have been added and manually tested to ensure that the changes work as intended. The commit also includes new fixtures for testing storage accounts and access connectors, and a test case for getting, applying, and deleting storage permissions. The `azure_api_client` function has been updated to handle different input argument lengths and methods such as "get", "put", and "post". A new managed identity, "appIduser1", has been added to the Azure mappings file, and the corresponding role assignments have been updated. The changes include error handling mechanisms for certain scenarios that may arise during the creation of the uber service principal.
* Crawlers: Use `TRUNCATE TABLE` instead of `DELETE FROM` when resetting crawler tables ([#2392](#2392)). In this release, the `.reset()` method for crawlers has been updated to use `TRUNCATE TABLE` instead of `DELETE FROM` when clearing out crawler tables, resulting in more efficient and idiomatic code. This change affects the existing `migrate-data-reconciliation` workflow and is accompanied by updated unit and integration tests to ensure correct functionality. The `reset()` method now accepts a table name argument, which is passed to the newly introduced `escape_sql_identifier()` utility function from the `databricks.labs.ucx.framework.utils` module for added safety. The migration status is now refreshed using the `TRUNCATE TABLE` command, which removes all records from the table, providing improved performance compared to the previous implementation. The `SHOW DATABASES` and `TRUNCATE TABLE` queries are validated in the `refresh_migration_status` workflow test, which now checks if the `TRUNCATE TABLE` query is used instead of `DELETE FROM` when resetting crawler tables.
* Detect tables that are not present in the mapping file ([#2205](#2205)). In this release, we have introduced a new method `get_remaining_tables()` that returns a list of tables in the Hive metastore that have not been processed by the migration tool. This method performs a full refresh of the index and checks each table in the Hive metastore against the index to determine if it has been migrated. We have also added a new private method `_is_migrated()` to check if a given table has already been migrated. Additionally, we have replaced the `refresh_migration_status` method with `update_migration_status` in several workflows to present a more accurate representation of the migration process in the dashboard. A new SQL script, 04_1_remaining_hms_tables.sql, has been added to list the remaining tables in Hive Metastore which are not present in the mapping file. We have also added a new test for the table migration job that verifies that tables not present in the mapping file are detected and reported. A new test function `test_refresh_migration_status_published_remained_tables` has been added to ensure that the migration process correctly handles the case where tables have been published to the target metadata store but still remain in the source metadata store. These changes are intended to improve the functionality of the migration tool for Hive metastore tables and resolve issue [#1221](#1221).
* Fixed ConcurrentDeleteReadException in migrate-view task during table migration ([#2282](#2282)). In this release, we have implemented a fix for the ConcurrentDeleteReadException that occurred during the `migrate-view` command's migration task. The solution involved moving the refreshing of migration status between batches from within the batches. Along with the fix, we added a new method `index()` to the `TableMigrate` class, which checks if a table has been migrated or not. This method is utilized in the `_view_can_be_migrated` method to ensure that all dependencies of a view have been migrated before migrating the view. The `index_full_refresh()` method, which earlier performed this check, has been modified to refresh the index between batches instead of within batches. It is worth noting that the changes made have been manually tested, but no unit tests, integration tests, or verification on staging environments have been added. The target audience for this release is software engineers who adopt this project. No new documentation, commands, workflows, or tables have been added or modified in this release.
* Fixed documentation typos: `create-missing-pricipals` -> `create-missing-principals` ([#2357](#2357)). This pull request resolves typographical errors in the `create-missing-principals` command documentation, correcting the mistaken usage of `create-missing-pricipals` throughout the project documentation. The changes encompass the command description, the UCX command section, and the manual process documentation for AWS storage credentials. The `create-missing-principals` command, utilized by Cloud Admins to create and configure new AWS roles for Unity Catalog access, remains functionally unaltered.
* Fixed linting for Spark Python workflow tasks ([#2349](#2349)). This commit updates the linter to support PySpark tasks in workflows by modifying the existing `experimental-workflow-linter` to correctly handle these tasks. Previously, the linter assumed Python files were Jupyter notebooks, but PySpark tasks are top-level Python files run as `__main__`. This change introduces a new `ImportFileResolver` class to resolve imports for PySpark tasks, and updates to the `DependencyResolver` class to properly handle them. Additionally, unit and integration tests have been updated and added to ensure the correct behavior of the linter. The `DependencyResolver` constructor now accepts an additional `import_resolver` argument in some instances. This commit resolves issue [#2213](#2213) and improves the accuracy and versatility of the linter for different types of Python files.
* Fixed missing `security_policy` when updating SQL warehouse config ([#2409](#2409)). In this release, we have added new methods `GetWorkspaceWarehouseConfigResponseSecurityPolicy` and `SetWorkspaceWarehouseConfigRequestSecurityPolicy` to improve handling of SQL warehouse config security policy. We have introduced a new variable `security_policy` to store the security policy value, which is used when updating the SQL warehouse configuration, ensuring that the required security policy is set and fixing the `InvalidParameterValue: Endpoint security policy is required and must be one of NONE, DATA_ACCESS_CONTROL, PASSTHROUGH` error. Additionally, when the `enable_serverless_compute` error occurs, the new SQL warehouse data access config is printed in the log, allowing users to manually configure the uber principal in the UI. We have also updated the `create_uber_principal` method to set the security policy correctly and added parameterized tests to test the setting of the warehouse configuration security policy. The `test_create_global_spn` method has been updated to include the `security_policy` parameter in the `create_global_spn` method call, and new test cases have been added to verify the warehouse config's security policy is correctly updated. These enhancements will help make the system more robust and user-friendly.
* Fixed raise logs `ResourceDoesNotExists` when iterating the log paths ([#2382](#2382)). In this commit, we have improved the handling of the `ResourceDoesNotExist` exception when iterating through log paths in the open-source library. Previously, the exception was not being properly raised or handled, resulting in unreliable code behavior. To address this, we have added unit tests in the `test_install.py` file that accurately reflect the actual behavior when iterating log paths. We have also modified the test to raise the `ResourceDoesNotExist` exception when the result is iterated over, rather than when the method is called. Additionally, we have introduced the `ResourceDoesNotExistIter` class to make it easier to simulate the error during testing. These changes ensure that the code can gracefully handle cases where the specified log path does not exist, improving the overall reliability and robustness of the library. Co-authored by Andrew Snare.
* Generate custom error during installation due to external metastore connectivity issues ([#2425](#2425)). In this release, we have added a new custom error `OperationFailed` to the `InstallUcxError` enumeration in the `databricks/labs/ucx/install.py` file. This change is accompanied by an exception handler that checks if the error message contains a specific string related to AWS credentials, indicating an issue with external metastore connectivity. If this condition is met, a new `OperationFailed` error is raised with a custom message, providing instructions for resolving the external metastore connectivity issue and re-running the UCX installation. This enhancement aims to provide a more user-friendly error message for users encountering issues during UCX installation due to external metastore connectivity. The functionality of the `_create_database` method has been modified to include this new error handling mechanism, with no alterations made to the existing functionality. Although the changes have not been tested using unit tests, integration tests, or staging environments, they have been manually tested on a workspace with incorrect external metastore connectivity.
* Improve logging when waiting for workflows to complete ([#2364](#2364)). This pull request enhances the logging and error handling of the databricks labs ucx project, specifically when executing workflows. It corrects a bug where `skip_job_wait` was not handled correctly, and now allows for a specified timeout period instead of the default 20 minutes. Job logs are replicated into the local logfile if a workflow times out. The existing `databricks labs ucx ensure-assessment-run` and `databricks labs ucx migrate-tables` commands have been updated, and unit and integration tests have been added. These improvements provide more detailed and informative logs, and ensure the quality of the code through added tests. The behavior of the `wait_get_run_job_terminated_or_skipped` method has been modified in the tests, and the `assign_metastore` function has also been updated, but the specific changes are not specified in the commit message.
* Lint dependencies consistently ([#2400](#2400)). In this release, the `files.py` module in the `databricks/labs/ucx/source_code/linters` package has been updated to ensure consistent linting of jobs. Previously, the linting sequence was not consistent and did not provide inherited context when linting jobs. These issues have been resolved by modifying the `_lint_one` function to build an inherited tree when linting files or notebooks, and by adding a `FileLinter` object to determine which file/notebook linter to use for linting. The `_lint_task` function has also been updated to yield a `LocatedAdvice` object instead of a tuple, and takes an additional argument, `linted_paths`, which is a set of paths that have been previously linted. Additionally, the `_lint_notebook` and `_lint_file` functions have been removed as their functionality is now encompassed by the `_lint_one` function. These changes ensure consistent linting of jobs and inherited context. Unit tests were run and passed. Co-authored by Eric Vergnaud.
* Make Lakeview names unique ([#2354](#2354)). A change has been implemented to guarantee the uniqueness of dataset names in Lakeview dashboards, addressing issue [#2345](#2345) where non-unique names caused system errors. This change includes renaming the `count` dataset to `fourtytwo` in the `datasets` field and updating the `name` field for a widget query from `count` to 'counter'. These internal adjustments streamline the Lakeview system's functionality while ensuring consistent and unique dataset naming.
* Optimisation: when detecting if a file is a notebook only read the start instead of the whole file ([#2390](#2390)). In this release, we have optimized the detection of notebook files during linting in a non-Workspace path in our open-source library. This has been achieved by modifying the `is_a_notebook` function in the `base.py` file to improve efficiency. The function now checks the start of the file instead of loading the entire file, which is more resource-intensive. If the content of the file is not available, it will attempt to read the file header when opening the file, instead of returning False if there's an error. The `magic_header` is used to determine if the file is a notebook or not. If the content is available, the function will check if it starts with the `magic_header` to determine if a file is a notebook or not, without having to read the entire file, which can be time-consuming and resource-intensive. This change will improve the linting performance and reduce resource usage, making the library more efficient and user-friendly for software engineers.
* Retry dashboard install on DeadlineExceeded ([#2379](#2379)). In this release, we have added the DeadlineExceeded exception to the @Retried decorator in the _create_dashboard method, which is used to create a lakeview dashboard from SQL queries in a folder. This modification to the existing functionality is intended to improve the reliability of dashboard installation by retrying the operation up to four minutes in case of a DeadlineExceeded error. This change resolves issues [#2376](#2376), [#2377](#2377), and [#2389](#2389), which were related to dashboard installation timeouts. Software engineers will benefit from this update as it will ensure successful installation of dashboards, even in scenarios where timeouts were previously encountered.
* Updated databricks-labs-lsql requirement from <0.8,>=0.5 to >=0.5,<0.9 ([#2416](#2416)). In this update, we have modified the version requirement for the `databricks-labs-lsql` library to allow version 0.8, which includes bug fixes and improvements in dashboard creation and deployment. We have changed the version constraint from '<0.8,>=0.5' to '>=0.5,<0.9' to accommodate the latest version while preventing future major version upgrades. This change enhances the overall design of the system and simplifies the code for managing dashboard deployment. Additionally, we have introduced a new test that verifies the `deploy_dashboard` method is no longer being used, utilizing the `deprecated_call` function from pytest to ensure that calling the method raises a deprecation warning. This ensures that the system remains maintainable and up-to-date with the latest version of the `databricks-labs-lsql` library.
* Updated ext hms detection to include more conf attributes ([#2414](#2414)). In this enhancement, the code for installing UCX has been updated to include additional configuration attributes for detecting external Hive Metastore (HMS). The previous implementation failed to recognize specific attributes like "spark.hadoop.hive.metastore.uris". This update introduces a new list of recognized prefixes for external HMS spark attributes, namely "spark_conf.spark.sql.hive.metastore", "spark_conf.spark.hadoop.hive.metastore", "spark_conf.spark.hadoop.javax.jdo.option", and "spark_conf.spark.databricks.hive.metastore". This change enables the extraction of a broader set of configuration attributes during installation, thereby improving the overall detection and configuration of external HMS attributes.
* Updated sqlglot requirement from <25.11,>=25.5.0 to >=25.5.0,<25.12 ([#2415](#2415)). In this pull request, we have updated the `sqlglot` dependency to a new version range, `>=25.5.0,<25.12`, to allow for the latest version of `sqlglot` to be used while ensuring that the version does not exceed 25.12. This change includes several breaking changes, new features, and bug fixes. New features include support for ALTER VIEW AS SELECT, UNLOAD, and other SQL commands, as well as improvements in performance. Bug fixes include resolutions for issues related to OUTER/CROSS APPLY parsing, GENERATE_TIMESTAMP_ARRAY, and other features. Additionally, there are changes that improve the handling of various SQL dialects, including BigQuery, DuckDB, Snowflake, Oracle, and ClickHouse. The changelog and commits also reveal fixes for bugs and improvements in the library since the last version used in the project. By updating the `sqlglot` dependency to the new version range, the project can leverage the newly added features, bug fixes, and performance improvements while ensuring that the version does not exceed 25.12.
* Updated sqlglot requirement from <25.9,>=25.5.0 to >=25.5.0,<25.11 ([#2403](#2403)). In this release, we have updated the required version range for the `sqlglot` library from '[25.5.0, 25.9)' to '[25.5.0, 25.11)'. This update allows us to utilize newer versions of `sqlglot` while maintaining compatibility with previous versions. The `sqlglot` team's latest release, version 25.10.0, includes several breaking changes, new features, bug fixes, and refactors. Notable changes include support for STREAMING tables in Databricks, transpilation of Snowflake's CONVERT_TIMEZONE function for DuckDB, support for GENERATE_TIMESTAMP_ARRAY in BigQuery, and the ability to parse RENAME TABLE as a Command in Teradata. Due to these updates, we strongly advise conducting thorough testing before deploying to production, as these changes may impact the functionality of the project.
* Use `load_table` instead of deprecated `is_view` in failing integration test `test_mapping_skips_tables_databases` ([#2412](#2412)). In this release, the `is_view` parameter of the `skip_table_or_view` method in the `test_mapping_skips_tables_databases` integration test has been deprecated and replaced with a more flexible `load_table` parameter. This change allows for greater control and customization when specifying how a table should be loaded during the test. The `load_table` parameter is a callable that returns a `Table` object, which contains information about the table's schema, name, object type, and table format. This improvement removes the use of a deprecated parameter, enhancing the maintainability of the test code.

Dependency updates:

 * Updated sqlglot requirement from <25.9,>=25.5.0 to >=25.5.0,<25.11 ([#2403](#2403)).
 * Updated databricks-labs-lsql requirement from <0.8,>=0.5 to >=0.5,<0.9 ([#2416](#2416)).
 * Updated sqlglot requirement from <25.11,>=25.5.0 to >=25.5.0,<25.12 ([#2415](#2415)).