Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(ingestion/lookml): resolve access notation for LookML Constant #12277

Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
fa4f5f2
lookml parameter support
sid-acryl Dec 19, 2024
865b5ff
fix: pr comment
sagar-salvi-apptware Jan 8, 2025
36f746f
test: updated golden files
sagar-salvi-apptware Jan 8, 2025
4a9ec88
fix: minor change
sagar-salvi-apptware Jan 8, 2025
750cf33
fix: pr comments
sagar-salvi-apptware Jan 8, 2025
4e7bc94
fix: pr comment
sagar-salvi-apptware Jan 10, 2025
ae605d9
fix: refactor changes to lookml constant
sagar-salvi-apptware Jan 15, 2025
a7cc9d5
test: updated test for lookml constant
sagar-salvi-apptware Jan 15, 2025
46316ca
fix: fixed lint issue
sagar-salvi-apptware Jan 15, 2025
2c16fc6
fix: pr comments
sagar-salvi-apptware Jan 16, 2025
0b54bb6
test: updated test for lookml constant
sagar-salvi-apptware Jan 16, 2025
f08b827
fix: pr comment
sagar-salvi-apptware Jan 17, 2025
f1daf12
fix: rename liquid_variable -> liquid_variables
sagar-salvi-apptware Jan 18, 2025
0b85dbd
fix: pr comments
sagar-salvi-apptware Jan 21, 2025
22b5af3
test: add unit tests for lookml constant trasnformer
sagar-salvi-apptware Jan 21, 2025
d68fd5d
fix: minor change
sagar-salvi-apptware Jan 21, 2025
896bf90
fix: minor comments
sagar-salvi-apptware Jan 21, 2025
69fed36
fix: pr comment
sagar-salvi-apptware Jan 22, 2025
8adeb4c
fix: minor pr comments
sagar-salvi-apptware Jan 23, 2025
baf4278
Update metadata-ingestion/docs/sources/looker/lookml_post.md
sagar-salvi-apptware Jan 23, 2025
41422a7
Update metadata-ingestion/docs/sources/looker/lookml_post.md
sagar-salvi-apptware Jan 23, 2025
f4fb5d7
Merge branch 'master' into fix/ING-784-lookml-parameter
mayurinehate Jan 23, 2025
ee02fcd
fix: added report info
sagar-salvi-apptware Jan 24, 2025
6634d1e
test: fixed ut
sagar-salvi-apptware Jan 24, 2025
aa89283
fix: added minor changes
sagar-salvi-apptware Jan 24, 2025
5726a26
fix: Constant Resolution
sagar-salvi-apptware Jan 24, 2025
00d1392
test: Constant Resolution
sagar-salvi-apptware Jan 24, 2025
3d1443a
Update metadata-ingestion/src/datahub/ingestion/source/looker/looker_…
sagar-salvi-apptware Jan 27, 2025
0a4cd06
Update metadata-ingestion/src/datahub/ingestion/source/looker/looker_…
sagar-salvi-apptware Jan 27, 2025
250b876
Update metadata-ingestion/tests/integration/lookml/test_lookml.py
sagar-salvi-apptware Jan 27, 2025
c5d8b2d
fix: minor pr comments
sagar-salvi-apptware Jan 27, 2025
d592a52
Merge branch 'master' into fix/ING-784-lookml-parameter
mayurinehate Jan 27, 2025
31a1152
Merge branch 'master' into fix/ING-784-lookml-parameter
mayurinehate Jan 28, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix: rename liquid_variable -> liquid_variables
  • Loading branch information
sagar-salvi-apptware committed Jan 18, 2025
commit f1daf12f21ba547d03b80d7ddd2cc279e2ce984d
2 changes: 1 addition & 1 deletion metadata-ingestion/docs/sources/looker/looker_recipe.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ source:
client_secret: ${LOOKER_CLIENT_SECRET}

# Liquid variables
# liquid_variable:
# liquid_variables:
# _user_attributes:
# looker_env: "dev"
# dev_database_prefix: "employee"
Expand Down
4 changes: 2 additions & 2 deletions metadata-ingestion/docs/sources/looker/lookml_post.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
#### Configuration Notes

1. If a view contains a liquid template (e.g. `sql_table_name: {{ user_attributes['db']}}.kafka_streaming.events }}`, with `db=ANALYTICS_PROD`), then you will need to specify the values of those variables in the `liquid_variable` config as shown below:
1. If a view contains a liquid template (e.g. `sql_table_name: {{ user_attributes['db']}}.kafka_streaming.events }}`, with `db=ANALYTICS_PROD`), then you will need to specify the values of those variables in the `liquid_variables` config as shown below:

```yml
liquid_variable:
liquid_variables:
user_attributes:
db: ANALYTICS_PROD
```
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -269,7 +269,7 @@
def _apply_transformation(self, value: str, view: dict) -> str:
return resolve_liquid_variable(
text=value,
liquid_variable=self.source_config.liquid_variable,
liquid_variable=self.source_config.liquid_variables,
report=self.reporter,
)

Expand Down Expand Up @@ -385,13 +385,13 @@
return str(self.source_config.lookml_constants.get(key))

# Check if it's a misplaced lookml constant
if key in self.source_config.liquid_variable:
if key in self.source_config.liquid_variables:
self.reporter.report_warning(
title="Misplaced lookml constant",
message="Misplaced lookml constant, Use 'lookml_constants' instead of 'liquid_variables'.",
context=f"Key {key}",
)
return f"@{{{key}}}"

Check warning on line 394 in metadata-ingestion/src/datahub/ingestion/source/looker/looker_template_language.py

View check run for this annotation

Codecov / codecov/patch

metadata-ingestion/src/datahub/ingestion/source/looker/looker_template_language.py#L394

Added line #L394 was not covered by tests

logger.warning(f"Constant '@{{{key}}}' not found in configuration.")
return "NULL"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -158,13 +158,17 @@ class LookMLSourceConfig(
description="When enabled, looker refinement will be processed to adapt an existing view.",
)

liquid_variable: Dict[Any, Any] = Field(
liquid_variables: Dict[Any, Any] = Field(
{},
description="A dictionary containing Liquid variables, Liquid logic, and LookML parameters with their corresponding values, utilized in SQL-defined "
"derived views. The Liquid template will be resolved in view.derived_table.sql and "
"view.sql_table_name. Defaults to an empty dictionary.",
)

_liquid_variable_deprecated = pydantic_renamed_field(
old_name="liquid_variable", new_name="liquid_variables", print_warning=True
)

lookml_constants: Dict[str, str] = Field(
{},
description=(
Expand Down
Loading