Skip to content

Commit

Permalink
add biodiversity preprocessing
Browse files Browse the repository at this point in the history
  • Loading branch information
elpamart authored and alexeh committed Sep 27, 2023
1 parent b07c429 commit 3dbd55c
Show file tree
Hide file tree
Showing 3 changed files with 83 additions and 0 deletions.
1 change: 1 addition & 0 deletions data/h3_data_importer/data_checksums/biodiversity
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
683cfc1d67d2fbd28673965b10f5bbeb33e4a7bb7fdef3d7807967e0faaef82b natural_crop_flii_loss_by_human_lu_50km_10km.tif
30 changes: 30 additions & 0 deletions data/preprocessing/biodiversity/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Variables
data_dir=data
resampling_resolution="0.083333"
checksums_dir=../../../h3_data_importer/data_checksums
AWS_S3_BUCKET_URL=s3://landgriffon-raw-data

# Targets
.PHONY: all download_forest_landscape_integrity_loss resample_forest_landscape_integrity_loss upload_results write_checksum

all: download_forest_landscape_integrity_loss resample_forest_landscape_integrity_loss upload_results write_checksum

download_forest_landscape_integrity_loss:
mkdir -p $(data_dir)
gsutil -m cp \
"gs://landgriffon-gee-bucket/landscape_indicators_20230821/natural_crop_flii_loss_by_human_lu_50km_1000m.tif" \
$(data_dir)

resample_forest_landscape_integrity_loss:
rio warp \
$(data_dir)/natural_crop_flii_loss_by_human_lu_50km_1000m.tif \
$(data_dir)/natural_crop_flii_loss_by_human_lu_50km_10km.tif \
--resampling average \
--res $(resampling_resolution) \
--overwrite

upload_results:
aws s3 cp $(data_dir)/natural_crop_flii_loss_by_human_lu_50km_10km.tif ${AWS_S3_BUCKET_URL}/processed/biodiversity/

write_checksum:
cd $(data_dir) && sha256sum natural_crop_flii_loss_by_human_lu_50km_10km.tif > $(checksums_dir)/biodiversity
52 changes: 52 additions & 0 deletions data/preprocessing/biodiversity/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# Biodiversity Indicators
## Forest Landscape Integrity Loss Processing

This folder contains a set of Makefile targets to download, process and upload the Biodiversity - Forest Landscape Integrity Loss data. The data is downloaded from Google Cloud Storage (GCS), resampled and then uploaded to an Amazon Web Serice (AWS) S3 bucket.


## Prerequisites

Before running the pipeline, ensure you have the following prerequisites in place:

1. **Google Cloud SDK (gsutil)**: You'll need gsutil to download data from Google Cloud Storage. Make sure it's installed and authenticated.

2. **Rasterio (rio)**: You'll need rio (Rasterio) for resampling the raster data. Install it using pip if it's not already installed: `pip install rasterio`.

3. **Amazon Web Services (AWS) CLI**: You must have the AWS CLI installed and configured with the necessary permissions to upload data to your AWS S3 bucket.

## Usage

### 1. Download data

This target downloads the forest landscape integrity loss data from GCS and stores it in a specified data directory.

```
make download_forest_landscape_integrity_loss
```

### 2. Resample dorest landscape integrity loss

This target resamples the downloaded data to a different resolution using Rasterio. It takes the input file from the data directory, performs resampling, and overwrites the file in the data directory with the resampled version

```
make resample_forest_landscape_integrity_loss
```

### 3. Upload results

This target uploads the processed forest landscape integrity loss data to an AWS S3 bucket. Make sure to set the `AWS_S3_BUCKET_URL` environment variable to the destination S3 bucket URL.

```
make upload_results
```

### 4. Write checksums

This target generates a SHA256 checksum for the processed data and writes it to a file in the specified checksums directory.

```
make write_checksum
```

`Note`: Make sure you have the necessary permissions and access to the data sources and AWS resources mentioned in this README before running the pipeline.

0 comments on commit 3dbd55c

Please sign in to comment.