Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Need to create dbt directory #14

Open
MichaelTiemannOSC opened this issue Nov 16, 2022 · 1 comment
Open

Need to create dbt directory #14

MichaelTiemannOSC opened this issue Nov 16, 2022 · 1 comment
Assignees

Comments

@MichaelTiemannOSC
Copy link
Contributor

MichaelTiemannOSC commented Nov 16, 2022

As explained in os-climate/os_c_data_commons#232 the dbt directory needs to now be created for ingestion pipelines that want to use Open Metadata. Cell #18 fails without such a directory:

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Input In [18], in <cell line: 8>()
      1 shutil.rmtree("/opt/app-root/src/PCAF-sovereign-footprint/dbt/pcaf_transform/models", ignore_errors=True)
      2 #path = "/opt/app-root/src/sovereign-footprint"
      3 #dir_list = os.listdir(path)
      4 #print("Files and directories in '", path, "' :") 
      5 # print the list
      6 #print(dir_list)
----> 8 os.mkdir("/opt/app-root/src/PCAF-sovereign-footprint/dbt/pcaf_transform/models", mode=0o755)
      9 custom_meta_content = {
     10     'data provider': df[['data_provider']],
     11      'description': 'Description',
     12 }
     13 custom_meta_fields= {}

FileNotFoundError: [Errno 2] No such file or directory: '/opt/app-root/src/PCAF-sovereign-footprint/dbt/pcaf_transform/models'

Actually, more must be done than merely creating the dbt directory. The pcaf_transform subdirectory must be created, copying the pattern of https://github.com/os-climate/wri-gppd-ingestion-pipeline and/or https://github.com/os-climate/essd-ingest-pipeline/tree/iceberg-dbt
There has also been some motion between sqlalchemy, dbt-trino, and the trino client. sqlalchemy versions >= 1.4.42 must be matched with trino>=0.319.0. Presently the latest version of dbt-trino (1.3.1) forces the installation of trino==0.318.0, which won't work. I will report the inconsistencies to the upstream projects and I expect it will all resolve relatively soon. Until then, best to use sqlalchemy<1.4.42.

@MichaelTiemannOSC
Copy link
Contributor Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants