You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As explained in os-climate/os_c_data_commons#232 the dbt directory needs to now be created for ingestion pipelines that want to use Open Metadata. Cell #18 fails without such a directory:
---------------------------------------------------------------------------
FileNotFoundError Traceback (most recent call last)
Input In [18], in <cell line: 8>()
1 shutil.rmtree("/opt/app-root/src/PCAF-sovereign-footprint/dbt/pcaf_transform/models", ignore_errors=True)
2 #path = "/opt/app-root/src/sovereign-footprint"
3 #dir_list = os.listdir(path)
4 #print("Files and directories in '", path, "' :")
5 # print the list
6 #print(dir_list)
----> 8 os.mkdir("/opt/app-root/src/PCAF-sovereign-footprint/dbt/pcaf_transform/models", mode=0o755)
9 custom_meta_content = {
10 'data provider': df[['data_provider']],
11 'description': 'Description',
12 }
13 custom_meta_fields= {}
FileNotFoundError: [Errno 2] No such file or directory: '/opt/app-root/src/PCAF-sovereign-footprint/dbt/pcaf_transform/models'
Actually, more must be done than merely creating the dbt directory. The pcaf_transform subdirectory must be created, copying the pattern of https://github.com/os-climate/wri-gppd-ingestion-pipeline and/or https://github.com/os-climate/essd-ingest-pipeline/tree/iceberg-dbt
There has also been some motion between sqlalchemy, dbt-trino, and the trino client. sqlalchemy versions >= 1.4.42 must be matched with trino>=0.319.0. Presently the latest version of dbt-trino (1.3.1) forces the installation of trino==0.318.0, which won't work. I will report the inconsistencies to the upstream projects and I expect it will all resolve relatively soon. Until then, best to use sqlalchemy<1.4.42.
The text was updated successfully, but these errors were encountered:
As explained in os-climate/os_c_data_commons#232 the
dbt
directory needs to now be created for ingestion pipelines that want to use Open Metadata. Cell #18 fails without such a directory:Actually, more must be done than merely creating the
dbt
directory. The pcaf_transform subdirectory must be created, copying the pattern of https://github.com/os-climate/wri-gppd-ingestion-pipeline and/or https://github.com/os-climate/essd-ingest-pipeline/tree/iceberg-dbtThere has also been some motion between sqlalchemy, dbt-trino, and the trino client. sqlalchemy versions >= 1.4.42 must be matched with trino>=0.319.0. Presently the latest version of dbt-trino (1.3.1) forces the installation of trino==0.318.0, which won't work. I will report the inconsistencies to the upstream projects and I expect it will all resolve relatively soon. Until then, best to use sqlalchemy<1.4.42.
The text was updated successfully, but these errors were encountered: