Release v0.56.0

* Added documentation to use Delta Live Tables migration ([#3587](#3587)). In this documentation update, we introduce a new section for migrating Delta Live Table pipelines to the Unity Catalog as part of the migration process. This workflow allows for the original and cloned pipelines to run independently after the cloned pipeline reaches the `RUNNING` state. The update includes an example of stopping and renaming an existing HMS DLT pipeline, and creating a new cloned pipeline. Additionally, known issues and limitations are outlined, such as supported streaming sources, maintenance pausing, and querying by timestamp. To streamline the migration process, the `migrate-dlt-pipelines` command is introduced with optional parameters for including or excluding specific pipeline IDs. This feature is intended for developers and administrators managing data pipelines and handling table aliasing issues. Relevant user documentation has been added and the changes have been manually tested.
* Added support for MSSQL and POSTGRESQL to HMS Federation ([#3701](#3701)). In this enhancement, the open-source library now supports Microsoft SQL Server (MSSQL) and PostgreSQL databases in the Hive Metastore Federation (HMS Federation) feature. This update introduces classes for handling external Hive Metastore instances and their versions, and refactors a regex pattern for better support of various JDBC URL formats. A new `supported_databases_port` class variable is added to map supported databases to default ports, allowing the code to handle SQL Server's distinct default port. Additionally, a `supported_hms_versions` class variable is created, outlining supported Hive Metastore versions. The `_external_hms` method is updated to extract HMS version information more accurately, and the `_split_jdbc_url` method is refactored for better URL format compatibility and parameter extraction. The test file `test_federation.py` has been updated with new unit tests for external catalog creation with MSSQL and PostgreSQL, further enhancing compatibility with various databases and expanding HMS Federation's capabilities.
* Added the CLI command for migrating DLT pipelines ([#3579](#3579)). A new CLI command, "migrate-dlt-pipelines," has been added for migrating DLT pipelines from HMS to UC using the DLT Migration API. This command allows users to include or exclude specific pipeline IDs during migration using the `--include-pipeline-ids` and `--exclude-pipeline-ids` flags, respectively. The change impacts the `PipelinesMigrator` class, which has been updated to accept and use these new parameters. Currently, there is no information available about testing, but the changes are expected to be manually tested and accompanied by corresponding unit and integration tests in the future. The changes are isolated to the `PipelinesMigrator` class and related functionality, with no impact on existing methods or functionality.
* Addressed Bug with Dashboard migration ([#3663](#3663)). In this release, the `_crawl` method in `dashboards.py` has been enhanced to exclude SDK dashboards that lack IDs during the dashboard migration process. This modification enhances migration efficiency by avoiding unnecessary processing of incomplete dashboards. Additionally, the `_list_dashboards` method now includes a check for dashboards with no IDs while iterating through the `dashboards_iterator`. If a dashboard with no ID is found, the method fetches the dashboard details using the `_get_dashboard` method and adds them to the `dashboards` list, ensuring proper processing. Furthermore, a bug fix for issue [#3663](#3663) has been implemented in the `RedashDashboardCrawler` class in `assessment/test_dashboards.py`. The `get` method has been added as a side effect to the `WorkspaceClient` mock's `dashboards` attribute, enabling the retrieval of individual dashboard objects by their IDs. This modification ensures that the `RedashDashboardCrawler` can correctly retrieve and process dashboard objects from the `WorkspaceClient` mock, preventing errors due to missing dashboard objects.
* Broaden safe read text caught exception scope ([#3705](#3705)). In this release, the `safe_read_text` function has been enhanced to handle a broader range of exceptions that may occur while reading a text file, including `OSError` and `UnicodeError`, making it more robust and safe. The function previously caught specific exceptions such as `FileNotFoundError`, `UnicodeDecodeError`, and `PermissionError`. Additionally, the codebase has been improved with updated unit tests, ensuring that the new functionality works correctly. The linting parts of the code have also been updated, enhancing the readability and maintainability of the project for other software engineers. A new method, `safe_read_text`, has been added to the `source_code` module, with several new test cases designed to ensure that the method handles edge cases correctly, such as when the file does not exist, when the path is a directory, or when an OSError occurs. These changes make the open-source library more reliable and robust for various use cases.
* Case sensitive/insensitive table validation ([#3580](#3580)). In this release, the library has been updated to enable more flexible and customizable metadata comparison for tables. A case sensitive flag has been introduced for metadata comparison, which allows for consideration or ignoring of column name case during validation. The `TableMetadataRetriever` abstract base class now includes a new parameter `column_name_transformer` in the `get_metadata` method, which is a callable that can be used to transform column names as needed for comparison. Additionally, a new `case_sensitive` parameter has been added to the `StandardSchemaComparator` constructor to determine whether column names should be compared case sensitively or not. A new parametrized test function `test_schema_comparison_case` has also been included to ensure that this functionality works as expected. These changes provide users with more control over the metadata comparison process and improve the library's handling of cases where column names in the source and target tables may have different cases.
* Catch `AttributeError` in `InfferedValue._safe_infer_internal` ([#3684](#3684)). In this release, we have implemented a change to the `_safe_infer_internal` method in the `InferredValue` class to catch `AttributeError`. This change addresses an issue in the Astroid library reported in their GitHub repository (<pylint-dev/astroid#2683>) and resolves issue [#3659](#3659) in our project. By handling `AttributeError` during the inference process, we have made the code more robust and safer. When an exception occurs, an error message is logged with debug-level logging, and the method yields the `Uninferable` sentinel value to indicate that inference failed for the node. This enhancement strengthens the source code linting code through value inference in our open-source library.
* Document to run `validate-groups-membership` before groups migration, not after ([#3631](#3631)). In this release, we have updated the order of executing the `validate-groups-membership` command in the group migration process. Previously, the command was recommended to be run after the groups migration, but it has been updated to be executed before the migration. This change ensures that the groups have the correct membership and the number of groups and users in the workspace and account are the same before migration, providing an extra level of safety. Additionally, we have updated the `remove-workspace-local-backup-groups` command to remove workspace-level backup groups and their permissions only after confirming the successful migration of all groups. We have also updated the spelling of the `validate-group-membership` command to `validate-groups-membership` in a documentation file. This release is aimed at software engineers who are adopting the project and looking to migrate their groups to the account level.
* Extend code migration progress documentation ([#3588](#3588)). In this documentation update, we have added two new sections, `Code Migration` and "Final details," to the open-source library's migration process documentation. The `Code Migration` section provides a detailed walkthrough of the steps to migrate code after completing table migration and data reconciliation, including using the linter to investigate compatibility issues and linted workspace resources. The "[linter advices](/docs/reference/linter_codes)" provide codes and messages on detected issues and resolution methods. The migrated code can then be prioritized and tracked using the `migration-progress` dashboard, and migrated using the `migrate-` commands. The `Final details` section outlines the steps to take once code migration is complete, including running the `cluster-remap` command to remap clusters to be Unity Catalog compatible. This update resolves issue [#2231](#2231) and includes updated user documentation, with new methods for linting and migrating local code, managing dashboard migrations, and syncing workspace information. Additional commands for creating and validating table mappings, migrating locations, and assigning metastores are also included, with the aim of improving the code migration process by providing more detailed documentation and new commands for managing the migration.
* Fixed Skip/Unskip schema functionality ([#3567](#3567)). In this release, we have addressed the improper handling of skip/unskip schema functionality in our open-source library. The `skip_schema` and `unskip_schema` methods in the `mapping.py` file have been updated to include the `hive_metastore` schema prefix while setting or unsetting the database property that determines whether a schema should be skipped. Additionally, the `_get_database_in_scope_task` and `_get_table_in_scope_task` methods have been modified to parse table properties as a dictionary, allowing for more straightforward lookup of the skip property for a table. The `test_skip_with_schema` and `test_unskip_with_schema` methods in the `tests/unit/test_cli.py` file have also been updated. The `test_skip_with_schema` method now includes the catalog name `hive_metastore` in the `ALTER SCHEMA` statement, ensuring that the schema is properly skipped. The `test_unskip_with_schema` method has been modified to use the `SET DBPROPERTIES` statement to set the value of the `databricks.labs.ucx.skip` property to `false`, effectively unskipping the schema. Furthermore, the `execute` method in the `sbe` module and the queries in the `mock_backend` module have been updated to match the new commands. These changes address the issue of improperly skipping schemas and ensure that the code functions as intended, allowing users to skip and unskip schemas as needed. Overall, these modifications improve the reliability and correctness of the skip/unskip schema functionality, ensuring that it behaves as expected in different scenarios.
* Fixed `Total Tables` widget in assessment to only show table counts ([#3738](#3738)). In this release, we have addressed the issue with the `Total Tables` widget in the assessment dashboard as part of resolving [#3738](#3738) and in relation to [#3252](#3252). The revised `00_3_count_total_tables.sql` query in the `src/databricks/labs/ucx/queries/assessment/main/` directory now includes a WHERE clause to filter out views from the table count query. By excluding views and only displaying table counts in the `Total Tables` widget, the scope of changes is limited to the SQL query itself. The diff reflects the addition of the WHERE clause and necessary indentation. The commit has been manually tested as part of our quality assurance process, and the successful test results are documented in the `Tests` section of the commit message.
* Fixed broken anchor for doc release ([#3720](#3720)). In this release, we have developed and implemented fixes to address issues with the Databricks workflows documentation used in the migration process. The previous version contained a broken anchor reference for the workflow process, which has now been corrected. This improvement includes the addition of a manual test to verify the fix. The revised documentation enables users to view the status of deployed workflows and rerun failed workflows using the `workflows` and `repair-run` commands, respectively. These updates simplify the management and troubleshooting of workflows, enhancing the overall user experience.
* Fixed broken anchors in documentation ([#3712](#3712)). In this release, we have made significant improvements to the UCX process documentation, addressing issues related to broken anchors, outdated command names, and syntax. The commands `enable_hms_federation` and `create_federated_catalog` have been renamed to `enable-hms-federation` and `create-federated-catalog`, respectively. These updates include corresponding changes to the command syntax and have been manually tested to ensure accuracy. Additionally, we have added a new command, `validate-groups-membership`, which can be executed prior to the group migration workflow for added confidence. In case of no matching account group in the UCX-installed workspace, the `create-account-groups` command is now available. This release also includes updates to the section titles and links to enhance clarity and reflect current functionality.
* Fixed notebook sources with `NotebookLinter.apply` ([#3693](#3693)). A new `Github.py` file has been added to the `databricks/labs/ucx/` directory, providing functionality for working with GitHub issues. It includes an `IssueType` enum, a `construct_new_issue_url` function, and constants for constructing URLs to the documentation and GitHub repository. The `NotebookLinter` class has been updated to include notebook fixing functionality, and the `PythonLinter` class has been introduced to run `apply` on an Abstract Syntax Tree (AST) tree. The `Notebook.apply` method has been implemented to apply changes to notebook sources and the legacy `NotebookMigrator` has been removed. These changes also include various unit and integration tests and modifications to the existing `databricks labs ucx migrate-local-code` command. The `DOCS_URL` method has been added to the `databricks.labs.ucx.github` module, and the error message for external metastore connectivity issues now includes a link to the UCX installation instruction in the documentation.
* Fixed the broken documentation links in dashboards ([#3726](#3726)). This revision updates documentation links in various dashboards to correct broken links and enhance the user experience. Specifically, it addresses issues [#3725](#3725) and [#3726](#3726) by updating links in the "Assessment Overview," "Assessment Summary," and `Compute summary` dashboards, as well as the `group migration` and `table upgrade` documentation. The changes include replacing local Markdown file links with online documentation links and updating links to point to the correct documentation sections in the UCX GitHub repository. Although the changes have been manually tested, no unit or integration tests have been added, and staging environment verification has not been performed. Despite this, the revisions ensure accurate and up-to-date documentation links, improving the usability of the dashboards.
* Force `MaybeDependency` to have a `Dependency` OR `list[Problem]`, not neither nor both ([#3635](#3635)). This commit enforces the `MaybeDependency` object to have either a `Dependency` or a `list[Problem]`, but not neither or both, in order to handle known libraries during import registration. It resolves issue [#3585](#3585), breaks up issue [#3626](#3626), and progresses issue [#1527](#1527), while modifying code linting logic and updating unit tests to accommodate these changes. Specifically, new classes like `KnownLoader`, `KnownDependency`, and `KnownProblem` have been introduced, and the `_resolve_allow_list` method has been updated to reflect the new enforcement. Additionally, tests have been added and modified to ensure the correct behavior of the modified logic, with a focus on handling directories, resolving children in context, and detecting known problems in imported libraries.
* HMS Federation Documentation ([#3688](#3688)). The HMS Federation feature allows Hive Metastore (HMS) to be federated to a catalog, acting as a step towards migrating to Unity Catalog or as a hybrid solution where both HMS and UC access to the data is required. This feature provides an alternative to the table migration process, eliminating the need for table mapping, creating catalogs and schemas, and migrating Hive metastore data objects. The `enable_hms_federation` command enables the Hive Metastore federation process, while the `create_federated_catalog` command creates a UC catalog that mirrors all the schemas and tables in the source Hive Metastore. The `migrate-glue-credentials` command, which is AWS-only, creates a UC Service Credential for GLUE. These new commands are documented in the HMS Federation Documentation section and are now part of the migration process documentation with the data reconciliation step following it. To enable HMS Federation, use the `enable-hms-federation` and `create-federated-catalog` commands.
* Make `MaybeTree` the main Python AST entrypoint for constructing the syntax tree ([#3550](#3550)). In this release, the main entry point for constructing the Python AST syntax tree has been changed from `Tree` to `MaybeTree` in the open-source library. This change involves moving class methods and static methods that construct a `MaybeTree` from the `Tree` class to the `MaybeTree` class, and making the class method that normalizes the source code before parsing the only entry point. The `normalized_parse` method has been renamed to `from_source_code` to match the commonly used naming for class methods within UCX. The `walk` and `first_statement` methods have been removed from `MaybeTree` as they were repetitions from `Tree`'s methods. These changes aim to enforce normalization and improve code consistency. Additionally, unit tests have been added and the Python linting related code has been modified to work with the new `MaybeTree` class. This change resolves issues [#3457](#3457) and [#3213](#3213).
* Make fixer diagnostic codes unique ([#3582](#3582)). This commit modifies the `databricks labs ucx migrate-local-code` command to make fixer diagnostic codes unique, ensuring accurate code migration and fixing. Two new methods have been added for modifying and adding unit and integration tests. Diagnostic codes for the `table-migrated-to-uc` issue are now unique depending on the context where the table is referenced: SQL, Python, or Python-SQL. This ensures the appropriate fixer is applied when addressing code migration issues, improving overall functionality and user experience. Additionally, the commit updates the documentation to include the new postfixes for the `table-migrated-to-uc` linter code and their descriptions, making it clearer for developers to diagnose and resolve issues related to table migration.
* Removed the linting false positive for missing table format warning when using `spark.table` ([#3589](#3589)). In this release, linting false positives related to missing table format warnings when using `spark.table` have been addressed, resolving issue [#3545](#3545). The linting logic and unit tests have been updated to handle changes in the default format for table references in Databricks Runtime 8.0, which now uses Delta as the default format. These changes improve the accuracy of the linting process, reducing unnecessary warnings and enhancing the overall developer experience. Additionally, the `test_linting_walker_populates_paths` unit test in the `test_jobs.py` file has been updated to use a different file path for testing.
* Removed tree from `PythonSequentialLinter` ([#3535](#3535)). In this release, the `PythonSequentialLinter` has been refactored to no longer manipulate the code tree, and instead, the tree manipulation logic has been moved to `NotebookLinter`. This change improves the separation of concerns between the two components, resulting in a more modular and maintainable codebase. The `NotebookLinter` now handles early failure when resolving the code used by a notebook and attaches `%run` notebook trees as a child tree to the cell that calls the notebook. The code linting functionality has been modified, and the `databricks labs ucx lint-local-code` command has been updated. These changes resolve [#3543](#3543) and progress [#3514](#3514) and are dependent on PRs [#3529](#3529) and [#3550](#3550). The changes have been manually tested and include added and modified unit tests. Additionally, the `Advice` class has been updated to include a type variable `T`, which allows for more specific type hinting when creating instances of the class and its subclasses.
* Rename file language helper function ([#3661](#3661)). In this code change, the helper function for determining the file language and checking its support by the linter has been renamed and refactored. The function, previously called `file_language`, has been updated and now named `infer_file_language_if_supported`. This change clarifies the function's purpose as it not only infers the file language but also checks if the file is supported by the linter, acting as a filter. The function returns a `Language` object if the file is supported or `None` if it is not. The `infer_file_language_if_supported` function has been used in other parts of the codebase, such as the `is_a_notebook` function. This change improves the codebase's readability and maintainability by making the helper function's purpose more explicit. The related code has been updated to use the new function accordingly.
* Scope crawled jobs in `JobsCrawler` with `include_job_ids` ([#3658](#3658)). In this release, the `JobsCrawler` class in the `workflow_task.py` file has been updated to include a new optional parameter `include_job_ids` in the constructor. This parameter allows users to specify a list of job IDs to include in the crawling process, improving efficiency in large workspaces. Additionally, a check has been added to the `_assess_jobs` method to skip jobs whose IDs are not in the list of included IDs. Integration tests have been added to ensure the correct behavior of the new feature. This change resolves issue [#3656](#3656), which requested the ability to crawl jobs based on a specific list of job IDs. It is recommended to add a comment to the code explaining the purpose and usage of the `include_job_ids` parameter and update the documentation accordingly.
* Support fixing `LocalFile`'s with `FileLinter` ([#3660](#3660)). In this release, we have added new methods `write_text`, `safe_write_text`, `back_up_path`, and `revert_back_up_path` to the `base.py` file to support fixing files in `LocalFile` containers and adding unit tests and integration tests. The `LocalFile` class in the "files.py" file has been extended to include new methods and properties, such as `apply`, `migrated_code`, `back_up_path`, and `back_up_original_and_flush_migrated_code`, enabling fixing files using linters and writing changes back to the container. The `databricks labs ucx migrate-local-code` command has also been updated to utilize the new functionality. These changes address issue [#3514](#3514), ensuring the proper handling of errors during file writing and providing automated fixing of code issues within LocalFiles.
* Updated `migate-local-code` to use latest linter functionality ([#3700](#3700)). In this update, the `migrate-local-code` command has been enhanced by incorporating the latest linter functionality. The `LocalFileMigrator` and `LocalCodeLinter` classes have been merged, and the interfaces of `.fix` and `.apply` methods have been aligned. A new `FixerWalker` has been introduced to address dependencies in the dependency graph, and the existing `databricks labs ucx migrate-local-code` command has been updated accordingly. Relevant unit tests and integration tests have been added and modified to ensure the correctness of the changes, which resolve issue [#3514](#3514) and supersede issue [#3520](#3520). The `lint-local-code` command has also been updated with a flag to specify the path for linting. The `migate-local-code` command now lints local code and generates advice on how to make it compatible with the Unity Catalog, and can also apply local code fixes to make them compatible.
* Updated sqlglot requirement from <26.3,>=25.5.0 to >=25.5.0,<26.4 ([#3572](#3572)). In this pull request, we have updated the requirement for the `sqlglot` library in the 'pyproject.toml' file, changing it from being greater than or equal to version 25.5.0 and less than 26.3, to being greater than or equal to version 25.5.0 and less than 26.4. This change is part of issue [#3572](#3572) and was made to allow for the use of the latest version of 'sqlglot'. The pull request includes a changelog from the `sqlglot` repository, detailing the changes made in each version between 25.5.0 and 26.4. The commits relevant to this update include bumping the version of `sqlglotrs` to various versions between 0.3.7 and 0.3.14. This pull request was automatically generated by Dependabot, a tool that creates pull requests to update the dependencies in a project. It is now ready for review and merging.
* Updated sqlglot requirement from <26.4,>=25.5.0 to >=25.5.0,<26.7 ([#3677](#3677)). In this release, we have updated the `sqlglot` dependency from version `>=25.5.0,<26.4` to `>=25.5.0,<26.7`. This change allows us to leverage the latest version of `sqlglot`, which includes various bug fixes and improvements, such as avoiding redundant casts in FROM/TO_UTC_TIMESTAMP and enhancing UUID support. Although there are some breaking changes introduced in the latest version, they should not affect our project's functionality. Additionally, this update includes several bug fixes and improvements for specific dialects such as Redshift, BigQuery, and TSQL. Overall, this update enhances the performance and functionality of the `sqlglot` library, ensuring compatibility with the latest version.
* Use cached property for table migration index on local checkout context ([#3711](#3711)). In this release, we introduce a new cached property, `_migration_index`, to the `LocalCheckoutContext` class, designed to store the table migration index for the local checkout context. This change aims to prevent multiple recrawling when the migration index is empty. The `linter_context_factory` method has been refactored to utilize the new `_migration_index` property, and the `CurrentSessionState` parameter is removed. Additionally, the `local_code_linter` method has been updated to leverage the new `LinterContext` instance with the `_migration_index` property, instead of using the `linter_context_factory` method. The `LocalCodeLinter` object now accepts a new callable lambda function, returning a `LinterContext` instance with the `_migration_index` property. These enhancements improve code performance by reducing the migration index crawls in the local checkout context and simplify the code by eliminating the `CurrentSessionState` parameter.
* [DOCS] Explain when to run `remove-workspace-local-backup-groups` workflow ([#3707](#3707)). In this release, the UCX component of the application has been enhanced with new Databricks workflows for orchestrating the group migration process. The `workflows` command displays the status of the workflows, and the `repair-run` command allows for rerunning failed workflows. The group migration workflow is specifically designed to be executed after a successful assessment workflow, and running it is followed by an optional `remove-workspace-local-backup-groups` workflow. This final step removes unnecessary workspace-level backup groups and their associated permissions, keeping the workspace clean and organized. The `remove-workspace-local-backup-groups` workflow should only be executed after confirming the successful migration of all groups involved.

Dependency updates:

 * Updated sqlglot requirement from <26.3,>=25.5.0 to >=25.5.0,<26.4 ([#3572](#3572)).
 * Updated sqlglot requirement from <26.4,>=25.5.0 to >=25.5.0,<26.7 ([#3677](#3677)).