[internal] port over existing UCX integration tests #44
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This PR breaks backwards compatibility for databrickslabs/blueprint downstream. See build logs for more details. Running from downstreams #16 |
|
✅ 35/35 passed, 3 skipped, 2m19s total Running from acceptance #56 |
nfx
added a commit
that referenced
this pull request
Sep 18, 2024
* Moved remaining UCX integration tests and fixtures ([#45](#45)). In this release, we have made significant changes to the UCX integration tests and fixtures, as indicated by multiple commit messages. Firstly, we have moved remaining UCX integration tests and fixtures, introducing a new PyTest fixture called `Installation` in the README.md file, providing instructions on how to add `databricks-labs-pytester` as a test-time dependency when using `hatch` as the build system. Additionally, we have added the `make_feature_table` fixture, which creates a Databricks feature table and cleans it up after the test, taking optional parameters for customization. We have also modified the `mypy` configuration in the `pyproject.toml` file to allow untyped imports during the type-checking process. In the `compute.py` file, we have updated the `make_job` fixture to return a function that creates a `databricks.sdk.service.jobs.Job` instance, and modified the `create` function to return the `databricks.sdk.service.jobs.Job` instance directly. We have also added a new fixture called `make_feature_table` in the plugin file, which simulates the lifecycle of a feature table in the machine learning service, with functions to generate a unique name and create/remove the feature table. In the `test_catalog.py` file, we have made changes to clean up the file and ensure proper logging of test events and errors. Overall, these changes aim to refactor, expand functionality, and improve user-friendliness for the adopters of the project, ensuring proper logging and debugging capabilities. * [internal] port over existing UCX integration tests ([#44](#44)). Three new integration tests have been added to the UCX project to verify the functionality of the `RemoveAfter` property for tables and schemas. The `test_remove_after_property_table` and `test_remove_after_property_schema` tests create new tables and schemas, respectively, and check if the `RemoveAfter` property is included in their properties. However, these tests are still marked as `TODO` due to existing issues with the `tables.get` and `schemas.get` functions. In addition, existing UCX integration tests have been ported over, which include new functions for testing the removal of resources based on the `RemoveAfter` tag. These tests are located in the `tests/integration/fixtures/test_compute.py` file and test the removal of various types of resources, including jobs, clusters, warehouses, and instance pools. The tests ensure that the time until purge is less than the `TEST_RESOURCE_PURGE_TIMEOUT` value plus one hour and import the `datetime` module and the `TEST_RESOURCE_PURGE_TIMEOUT` constant from the `watchdog` fixture, as well as the `logging` and `databricks.sdk.service.iam` modules.
Merged
nfx
added a commit
that referenced
this pull request
Sep 18, 2024
* Moved remaining UCX integration tests and fixtures ([#45](#45)). In this release, we have made significant changes to the UCX integration tests and fixtures, as indicated by multiple commit messages. Firstly, we have moved remaining UCX integration tests and fixtures, introducing a new PyTest fixture called `Installation` in the README.md file, providing instructions on how to add `databricks-labs-pytester` as a test-time dependency when using `hatch` as the build system. Additionally, we have added the `make_feature_table` fixture, which creates a Databricks feature table and cleans it up after the test, taking optional parameters for customization. We have also modified the `mypy` configuration in the `pyproject.toml` file to allow untyped imports during the type-checking process. In the `compute.py` file, we have updated the `make_job` fixture to return a function that creates a `databricks.sdk.service.jobs.Job` instance, and modified the `create` function to return the `databricks.sdk.service.jobs.Job` instance directly. We have also added a new fixture called `make_feature_table` in the plugin file, which simulates the lifecycle of a feature table in the machine learning service, with functions to generate a unique name and create/remove the feature table. In the `test_catalog.py` file, we have made changes to clean up the file and ensure proper logging of test events and errors. Overall, these changes aim to refactor, expand functionality, and improve user-friendliness for the adopters of the project, ensuring proper logging and debugging capabilities. * [internal] port over existing UCX integration tests ([#44](#44)). Three new integration tests have been added to the UCX project to verify the functionality of the `RemoveAfter` property for tables and schemas. The `test_remove_after_property_table` and `test_remove_after_property_schema` tests create new tables and schemas, respectively, and check if the `RemoveAfter` property is included in their properties. However, these tests are still marked as `TODO` due to existing issues with the `tables.get` and `schemas.get` functions. In addition, existing UCX integration tests have been ported over, which include new functions for testing the removal of resources based on the `RemoveAfter` tag. These tests are located in the `tests/integration/fixtures/test_compute.py` file and test the removal of various types of resources, including jobs, clusters, warehouses, and instance pools. The tests ensure that the time until purge is less than the `TEST_RESOURCE_PURGE_TIMEOUT` value plus one hour and import the `datetime` module and the `TEST_RESOURCE_PURGE_TIMEOUT` constant from the `watchdog` fixture, as well as the `logging` and `databricks.sdk.service.iam` modules.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.