Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Installation
in the README.md file, providing instructions on how to adddatabricks-labs-pytester
as a test-time dependency when usinghatch
as the build system. Additionally, we have added themake_feature_table
fixture, which creates a Databricks feature table and cleans it up after the test, taking optional parameters for customization. We have also modified themypy
configuration in thepyproject.toml
file to allow untyped imports during the type-checking process. In thecompute.py
file, we have updated themake_job
fixture to return a function that creates adatabricks.sdk.service.jobs.Job
instance, and modified thecreate
function to return thedatabricks.sdk.service.jobs.Job
instance directly. We have also added a new fixture calledmake_feature_table
in the plugin file, which simulates the lifecycle of a feature table in the machine learning service, with functions to generate a unique name and create/remove the feature table. In thetest_catalog.py
file, we have made changes to clean up the file and ensure proper logging of test events and errors. Overall, these changes aim to refactor, expand functionality, and improve user-friendliness for the adopters of the project, ensuring proper logging and debugging capabilities.RemoveAfter
property for tables and schemas. Thetest_remove_after_property_table
andtest_remove_after_property_schema
tests create new tables and schemas, respectively, and check if theRemoveAfter
property is included in their properties. However, these tests are still marked asTODO
due to existing issues with thetables.get
andschemas.get
functions. In addition, existing UCX integration tests have been ported over, which include new functions for testing the removal of resources based on theRemoveAfter
tag. These tests are located in thetests/integration/fixtures/test_compute.py
file and test the removal of various types of resources, including jobs, clusters, warehouses, and instance pools. The tests ensure that the time until purge is less than theTEST_RESOURCE_PURGE_TIMEOUT
value plus one hour and import thedatetime
module and theTEST_RESOURCE_PURGE_TIMEOUT
constant from thewatchdog
fixture, as well as thelogging
anddatabricks.sdk.service.iam
modules.