diff --git a/docs/ucx/docs/process/index.mdx b/docs/ucx/docs/process/index.mdx index 5933392d93..4c265cfd11 100644 --- a/docs/ucx/docs/process/index.mdx +++ b/docs/ucx/docs/process/index.mdx @@ -8,7 +8,8 @@ On a high level, the steps in migration process are: 1. [assessment](/docs/reference/workflows#assessment-workflow) 2. [group migration](/docs/reference/workflows#group-migration-workflow) 3. [table migration](/docs/process/#table-migration-process) -4. [data reconciliation](/docs/reference/workflows#post-migration-data-reconciliation-workflow) +4. [HMS Federation](/docs/process/#hms-federation) +5. [data reconciliation](/docs/reference/workflows#post-migration-data-reconciliation-workflow) 6. [code migration](/docs/reference/commands#code-migration-commands) 7. [delta live table pipeline migration](/docs/process#delta-live-table-pipeline-migration-process) 8. [final details](#final-details) @@ -290,6 +291,26 @@ The [`revert-migrated-tables` command](/docs/reference/commands#revert-migrated- the `upgraded_to` property on the source object. Use this command to allow for migrating a table or view again. +## HMS Federation + +Hive Metastore (HMS) Federation is a feature that allows HMS to be federated to a catalog. +More information is available in the [Databricks documentation](https://docs.databricks.com/en/data-governance/unity-catalog/hms-federation/index.html). +HMS Federation is used in two scenarios: +1. A step to ease the migration to Unity Catalog. +2. A Hybrid solution where both HMS and UC access to the data is required. + +> You are required to complete the [assessment workflow](/docs/reference/workflows#assessment-workflow) before starting the table migration workflow. +HMS Federation is an alternative to table migration. +You are required to complete all the steps documented in the [table migration process](/docs/process#table-migration-process) except for: +- [Table Mapping](/docs/process#table-mapping) +- [Create Catalogs and Schemas](/docs/process/#step-2-create-catalogs-and-schemas) +- [Migrate Hive metastore data objects](/docs/process#migrate-hive-metastore-data-objects) + +There are two steps to enable HMS Federation: +1. [enable-hms-federation](/docs/reference/commands#enable-hms-federation) +2. [create-federated-catalog](/docs/reference/commands#create-federated-catalog) + + ## Code Migration After you're done with the [table migration](#table-migration-process) and diff --git a/docs/ucx/docs/reference/commands/index.mdx b/docs/ucx/docs/reference/commands/index.mdx index 12509bc94c..4d738054ea 100644 --- a/docs/ucx/docs/reference/commands/index.mdx +++ b/docs/ucx/docs/reference/commands/index.mdx @@ -667,10 +667,53 @@ These commands are for [pipeline migration](/docs/process#delta-live-table-pipel ### `migrate-dlt-pipelines` -```text +```commandline $ databricks labs ucx migrate-dlt-pipelines [--include-pipeline-ids ] [--exclude-pipeline-ids ] ``` +## HMS (Hive Metastore) federation commands + +These commands are for [Hive Metastore federation](/docs/process#hive-metastore-federation-process) and require the [assessment workflow](/docs/reference/workflows#assessment-workflow) to be completed. +Hive Metastore federation creates a UC catalog that mirrors all the schemas and tables in the source Hive Metastore. It is an alternative to the [table migration process](/docs/process#table-migration-process). + +### `enable_hms_federation` + +```commandline +$ databricks labs ucx enable_hms_federation +``` + +This command sets the `enable_hms_federation` flag in the config.yml file to `True`.
+It is used to enable the Hive Metastore federation process. +It is also used by the `migrate-locations` command to determine whether to set the [Fallback Mode](https://docs.databricks.com/en/data-governance/unity-catalog/hms-federation/hms-federation-internal.html) to "True" in the newly created external locations. +In AWS it also also allows the creation a GLUE role if one is required using the `create-missing-principals` command. + +### `create_federated_catalog` + +```commandline +$ databricks labs ucx create_federated_catalog +``` + +This command performs the following steps: +1. Prompts the user for a name for the federated catalog and connection. +2. Create an HMS "Connection" in the Unity Catalog. The HMS connection will point to the HMS used by the workspace that was identified in the installation process. It supports: + - Internal Workspace Hive Metastore + - AWS Glue Data Catalog + - External Database Hive Metastore (Postgres, SQL Server, MySQL) +3. Create a federated catalog in Unity Catalog. The federated catalog will mirror all the schemas and tables in the source Hive Metastore. +4. Whitelist the external location identified in the "Assessment" phase. + +### `migrate-glue-credentials` + +```commandline +$ databricks labs ucx migrate-glue-credentials +``` + +AWS Only. +This command is used for HMS Federation and creates a UC Service Credential for GLUE. It requires an existing UC compatible GLUE role. +The role can be created using the `create-missing-principals` command. + + + ## Utility commands ### `logs`