-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* [WPS-69]: use 512px tile sizes for mapbox * [WPSI-62]: implementing google action
- Loading branch information
1 parent
a8c1e8e
commit 985c221
Showing
12 changed files
with
175 additions
and
7 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,76 @@ | ||
| name: Update 24-month SPI Anomaly (Long-term) | ||
|
|
||
| on: | ||
| workflow_dispatch: | ||
| # Allows manual triggering of the action | ||
| schedule: | ||
| - cron: '0 12 * * *' # Runs daily at 12:00 UTC, adjust as needed | ||
|
|
||
| jobs: | ||
| spi-24-month: | ||
| runs-on: ubuntu-latest | ||
|
|
||
| steps: | ||
| - name: Checkout Repository | ||
| uses: actions/checkout@v4 | ||
| - uses: actions/setup-python@v3 | ||
| with: | ||
| python-version: '3.13' | ||
|
|
||
| - name: Upgrade pip & Install dependencies | ||
| run: | | ||
| python -m pip install --upgrade pip | ||
| echo "Installing Python dependencies" | ||
| if [ -f ./python/requirements.txt ]; then pip install -r ./python/requirements.txt; fi | ||
| pip install rasterio | ||
| pip install earthengine-api | ||
| pip install earthengine-api --upgrade | ||
| # Get file source name and put it on env | ||
| # - name: Run Python Script on File | ||
| # run: | | ||
| # python python/getFilename24month.py | ||
|
|
||
| # # Copy from Google storage to local | ||
| # - name: Copy from Google storage to local | ||
| # uses: actions-hub/gcloud@master | ||
| # env: | ||
| # PROJECT_ID: ${{secrets.GCP_PROJECT_ID}} | ||
| # APPLICATION_CREDENTIALS: ${{ secrets.GOOGLE_APPLICATION_CREDENTIALS }} | ||
| # with: | ||
| # args: cp $WPS_24_MONTH_GS $WPS_24_MONTH_FILENAME | ||
| # cli: gsutil | ||
|
|
||
| # # Get file source name and put it on env | ||
| # - name: Project downloaded file | ||
| # run: | | ||
| # python python/project.py | ||
|
|
||
| # # Upload to Google storage | ||
| # - name: Upload to Google storage | ||
| # uses: actions-hub/gcloud@master | ||
| # env: | ||
| # PROJECT_ID: ${{secrets.GCP_PROJECT_ID}} | ||
| # APPLICATION_CREDENTIALS: ${{ secrets.GOOGLE_APPLICATION_CREDENTIALS }} | ||
| # with: | ||
| # args: cp $WPS_24_MONTH_FILENAME $WPS_24_MONTH_GS_PROJ | ||
| # cli: gsutil | ||
|
|
||
|
|
||
| - name: 'auth' | ||
| uses: 'google-github-actions/auth@v2' | ||
| with: | ||
| token_format: 'access_token' | ||
| project_id: ${{ secrets.GCP_PROJECT_ID }} | ||
| workload_identity_provider: 'projects/297470958436/locations/global/workloadIdentityPools/wps-github-action-automations' | ||
| service_account: '[email protected]' | ||
|
|
||
| - name: 'Use gcloud CLI' | ||
| run: 'gcloud info' | ||
|
|
||
| # upload to GEE | ||
| - name: Upload to GEE | ||
| run: | | ||
| python python/earthengine.py | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -44,6 +44,8 @@ coverage | |
| .env.local | ||
| .env | ||
|
|
||
| .secrets | ||
|
|
||
| # vscode and vendor cache | ||
| .vscode/ | ||
| untracked | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| act -P ubuntu-latest=ubuntu-builder --pull=false --container-architecture linux/amd64 --secret-file .secrets workflow_dispatch --workflows '.github/workflows/cron_update_SPI24.yml' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| print("***************HELLOO******************") |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| from datetime import datetime | ||
|
|
||
| print(f"{datetime.now().year}{datetime.now().month}") |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| FROM ubuntu:20.04 | ||
|
|
||
| RUN apt-get update \ | ||
| && apt-get upgrade -y \ | ||
| && DEBIAN_FRONTEND=noninteractive apt-get install -y \ | ||
| build-essential \ | ||
| curl \ | ||
| nodejs \ | ||
| npm \ | ||
| && rm -rf /var/lib/apt/lists/* |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| import ee | ||
|
|
||
| ee.Authenticate() | ||
|
|
||
| ee.Initialize(project="wpsi-208318") | ||
|
|
||
| print(ee.String('Hello from the Earth Engine servers!').getInfo()) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| from datetime import datetime | ||
| import os | ||
|
|
||
| # 24 month | ||
|
|
||
| def getPrevMonth(month, delta): | ||
| if month+delta <= 0: | ||
| return month+delta+12 | ||
| else: | ||
| return (month+delta)%12 | ||
|
|
||
| current_year_month = f"{datetime.now().year}{getPrevMonth(datetime.now().month, -1):02}" | ||
| prev_year_month = f"{datetime.now().year}{getPrevMonth(datetime.now().month, -2):02}" | ||
|
|
||
| file = 'ERA5_SPI24_{year_month}' | ||
|
|
||
| env_file = os.getenv('GITHUB_ENV') # Get the path of the runner file | ||
|
|
||
| # write to the file | ||
| with open(env_file, "a") as env_file: | ||
| env_file.write(f"WPS_24_MONTH_GS=gs://wps_pillar1a/data_sources/Deltares/Data{current_year_month}/{file.format(year_month=prev_year_month)}.tif\n") | ||
| env_file.write(f"WPS_24_MONTH_GS_PROJ=gs://wps_pillar1a/data_sources/Deltares/Data{current_year_month}/proj_{file.format(year_month=prev_year_month)}.tif\n") | ||
| env_file.write(f"WPS_24_MONTH_FILENAME={file.format(year_month=prev_year_month)}.tif") | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| import rasterio | ||
| from rasterio.crs import CRS | ||
| import os | ||
|
|
||
| FILE = os.environ["WPS_24_MONTH_FILENAME"] | ||
|
|
||
|
|
||
| if FILE: | ||
| print('The filename evn WPS_24_MONTH_FILENAME={FILE} successfully loaded!') | ||
| else: | ||
| print('Please check the env WPS_24_MONTH_FILENAME was not loaded') | ||
|
|
||
| with rasterio.open(FILE, "r+") as rds: | ||
| rds.crs = CRS.from_epsg(4326) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| ## Using act to locally run github actions | ||
|
|
||
| you have to create a local image with the tag ubuntu-builder by running | ||
|
|
||
|
|
||
|
|
||
| ```bash | ||
| docker build -f python/docker/Act.Dockerfile -t ubuntu-builder . | ||
| ``` | ||
|
|
||
| ```bash | ||
| act -P ubuntu-latest=ubuntu-builder --pull=false --container-architecture linux/amd64 --secret-file .secrets workflow_dispatch --workflows '.github/workflows/cron_update_SPI24.yml' | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| LMIPy==0.6.2 | ||
| pandas==2.2.3 | ||
| python-dotenv==1.0.1 | ||
| Requests==2.32.3 |