Skip to content
This repository has been archived by the owner on Aug 25, 2023. It is now read-only.

location-based delta names when in debug #203

Open
mrmasterplan opened this issue Nov 18, 2022 · 0 comments
Open

location-based delta names when in debug #203

mrmasterplan opened this issue Nov 18, 2022 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@mrmasterplan
Copy link
Contributor

mrmasterplan commented Nov 18, 2022

When the Configurator is set to debug, maybe it would be advantaveous to not create hive-table entries.
So when the Configurator is in debug, AND a table is marked as format:delta in yaml, then return a name of the format

    delta.`path`

Then a user has the following possiblity:
If the paths are set up like /{MNT}/{ID}/myDB/myTable, Then is no hive tables are set up, the user can quickly erase the temporary tables like dbutils.fs.rm('/tmp/<ID here>',True). SQL CREATE TABLE statements still work with such delta names. The user just has to make sure to move database creation to another sql that is then maybe not executed in the tests.

Since this feature changes existing behavior, maybe it should be opt-in with a flag Configurator().enable_path_based_debug_delta_names()

@mrmasterplan mrmasterplan added the enhancement New feature or request label Nov 18, 2022
@mrmasterplan mrmasterplan self-assigned this Nov 18, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant