Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature automate data transformation #137

Merged
merged 13 commits into from
Nov 20, 2024

Conversation

SwordSaintLancelot
Copy link
Collaborator

@SwordSaintLancelot SwordSaintLancelot commented Oct 28, 2024

Description

This PR is part of this PR.
To separate the VEDA and GHG environment we separated the transformation plugins that are to be used by the SM2A DAG. This PR creates a folder for the plugins which are pushed to GHG specific AWS account. The DAG will then fetch the transformation plugin to transform the netCDF files toCOGs
Tasks accomplished-:

  • Created plugins for older datasets (Sedac population, OCO2-Geos, Ecco-Darwin, TM5 inverse flux)
  • Created plugin for a new dataset (GOSAT-CH4)
  • Tested the delivery of plugins to S3 bucket (only if the plugin does not exist on S3)
  • Ran the DAG to test out the transformation of the new dataset.

@SwordSaintLancelot SwordSaintLancelot self-assigned this Oct 28, 2024
Copy link

github-actions bot commented Oct 28, 2024

PR Preview Action v1.4.8
🚀 Deployed preview to https://US-GHG-Center.github.io/ghgc-docs/pr-preview/pr-137/
on branch gh-pages at 2024-11-20 16:03 UTC

Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@siddharth0248
Copy link
Collaborator

@amarouane-ABDELHAK is this ok to be merged ?

@amarouane-ABDELHAK
Copy link

I will be reviewing it today. In the meantime, could you add the missing newlines?

@SwordSaintLancelot
Copy link
Collaborator Author

@amarouane-ABDELHAK what do you mean by newlines?

@SwordSaintLancelot SwordSaintLancelot merged commit f35694d into staging Nov 20, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants