LandTrendr is a set of spectral-temporal segmentation algorithms that focuses on removing the natural spectral variations in a time series of Landsat Images. Stabling the natural variation in a time series, gives emphasis on how a landscape is evolving with time. This is useful in many pursuits as it gives information on the state of a landscape, be it growing, remaining stable, or on a decline. LandTrendr is mostly used in Google Earth Engine (GEE), an online image processing console, where it is readily available for use.
A large obstacle in using LandTrendr in GEE, is knowing which configuration of LandTrendr parameters to use. The LandTrendr GEE function uses 9 arguments: 8 parameters that control how spectral-temporal segmentation is executed, and an annual image collection on which to assess and remove the natural variations. The original LandTrendr journal illustrates the effect and sensitivity of changing some of these values. The default parameters for the LandTrendr GEE algorithm do a satisfactory job in many circumstances, but extensive testing and time is needed to hone the parameter selection to get the best segmentation out of the LandTrendr algorithm for a given region. Thus, augmenting the LandTrendr parameter selection process would save time and standardize a method to choose parameters, but we also aim to take this augmentation a step further.
Traditionally, LandTrendr is run over an image collection with a single LandTrendr parameter configuration and is able to remove natural variation for every pixel time series in an image. But no individual LandTrendr parameter configuration is best for all surface conditions, where forest may respond well to one configuration, but many under or over emphasize stabilization in another land class. Thus here we aim to delineate patches of spectrally similar pixels from the imagery, find what LandTrendr parameters work best for each patch group, and run LandTrendr on each patch group location with that best parameter configuration.
GEE link open with Emapr Account for dependencies
NOTE: The below work flow is of a Oregon test region. So file paths to data sets are symbolic. To Islay they are real from vol/v1/proj.
First, we want to break up an image into spectrally simular chucks, patches of pixels that are like one another. These could be pixels that make up a pond or a stand of forest. This is what the SNIC script does for us. One thing to note is that ever patch is independent of the other patches even if two patches represent the same land class. From this script we get a seed image, which represents the starting point of each patch. The Seed Image has several bands pretaining to mean spectral values of that Seed's patch.
0. Script location
./LTOP_Oregon/scripts/GEEjs/01SNICPatches.js
1. Copy and paste script in GEE console
2. Make sure you all needed dependencies (emapr GEE account has all dependencies)
3. Review in script parameters. Lines 35-39, lines 47-49 (SNIC years), lines 83,84 (SNIC)
4. Run script (01SNICPatches)
5. Run tasks
Here, we move our SNIC datasets to a server for further processing .
1. Open terminal on Islay in a VNC
2. Activate conda environment “py35”. What if they dont have Islay?
conda activate py35
3. This script bring data from the Google drive to Islay
./LTOP_Oregon/scripts/GEEjs/00_get_chunks_from_gdrive.py
4. Run script
python ./LTOP_Oregon/scripts/GEEjs/00_get_chunks_from_gdrive.py LTOP_Oregon_SNIC_v1 ./LTOP_Oregon/rasters/01_SNIC/
5. Check data at download destination.
./LTOP_Oregon/rasters/01_SNIC/
Since, some areas have several images we need to merge them, and to save a little space we make a virtual raster. To make a virual raster from the many Seed Image chuncks follow the step below.
[add seed image]
1. Activate conda environment
a) conda activate gdal37. conda enviroment with GDAL and python 3.7
2. Build VRT SNIC seed image
a) make text file of file path is folder (only tiffs in the folder)
Linux:
ls -d "$PWD"/* > listOfTiffs.txt
b) build vrt raster with text file
gdalbuildvrt -srcnodata 0 snic_seed.vrt -input_file_list listOfTiffs.txt
3. Inspect and process the merged imagery.
a) Data location:
./LTOP_Oregon/rasters/01_SNIC/snic_seed.vrt
4 Change the raster calc SNIC Seed Image into a vector of points. Each point corresponds to a seed pixel.
Here we change every pixel in the Seed Image to a point vector except pixels with no data values.(You can use GDAL or QGIS)
1. Qgis tool - Raster pixels to points
2. Input
./snic_seed_pixels.tif
3. Output
./snic_seed_pixel_points.shp
GDAL Option:
1.GDAL Command
gdal_polygonize.py snic_seed.vrt ltop_snic_seed_polygons.shp
ogr2ogr -f "ESRI Shapefile" -sql "SELECT ST_PointOnSurface(geometry), * FROM ltop_snic_seed_polygons" -dialect sqlite snic_seed_pixel_points.shp ltop_snic_seed_polygons.shp
After sampling we select a subset of points. The size of the subset is arbitraray choosen to a size what works in GEE.
0. Qgis tool - Random extract within subsets
1. Input
./LTOP_Oregon/vectors/01_SNIC/01_snic_seed_pixel_points/snic_seed_pixel_points.shp
2. Number of selection
75000
Note: the value above is arbitrary and it did not break GEE
3. Save selected features as:
./snic_seed_pixel_points_75k.shp
GDAL Option
ogr2ogr -f "ESRI Shapefile" -sql "SELECT * FROM 01_snic_seed_pixel_points ORDER BY RANDOM() LIMIT 75000;" -dialect sqlite snic_seed_pixel_points_75k.shp snic_seed_pixel_points.shp
With point generated in the pervious step we extract the pixel values from the Seed Image across all bands and save them to a attribute table of the point vector file.
0. Qgis tool - Sample Raster Values ~3362.35 secs)
1. Input point layer
./snic_seed_pixel_points_75k.shp
2. Raster layer
./LTOP_Oregon/rasters/01_SNIC/snic_seed.vrt
3. Output column prefix
seed_
4. Output location
./snic_seed_pixel_points_75k_att.shp
Here we zip and upload the subset of vector points to GEE.
1. file location
./snic_seed_pixel_points_75k_att.shp
2. Zip shape files in directory
zip -r snic_seed_pixel_points_75k_att.zip snic_seed_pixel_points_75k_att/
3. Up load to GEE as asset
Remember in step 1 when I said the SNIC patch are independent of one another even if they represent the same land class. Well, it is here, where we link the patches that are simular to one another with the Kmeans alogroithm.
1. script local location
./LTOP_Oregon/scripts/GEEjs/02kMeansCluster.js
2. copy and paste script into GEE console
2. Make sure you all needed dependencies
3. Review in script parameters.
number of Kmean Clusters : 5000
4. Run script
5. Run tasks
task to drive
Kmeans Cluster seed image to Google drive
this snic seed image with K means cluster ids will be used to sample the data space for the landTrendr parammeter selection process.
task to assets
kmeans cluster image to GEE assets
This image asset will be used in the final screipt as the masking locations
Download the Kmeans Seed image. This image looks like the SNIC Seed Image but the pixels values are Kmeans Cluster IDs. This IDs are the links between simular patches.
0. Open terminal on Islay in a VNC
1. Script location
./LTOP_Oregon/scripts/GEEjs/
2. Activate conda environment “py35”
conda activate py35
3. Python script syntax
python 00_get_chunks_from_gdrive.py <google drive folder name> <local directory>
4. Python Command
python ./LTOP_Oregon/scripts/GEEjs/00_get_chunks_from_gdrive.py LTOP_Kmeans ./LTOP_Oregon/rasters/02_Kmeans/gee/
3. output location
./LTOP_Oregon/rasters/02_Kmeans/gee/
Using the SNIC Seed point vector dataset, the output from step 5, we sample the Kmeans Seed Image.
1. Build VRT K means seed image
a) make text file of file path is folder (only tiffs in the folder)
Linux:
ls -d "$PWD"/* > listOfTiffs.txt
b) build vrt raster with text file
gdalbuildvrt -srcnodata 0 kmeans_clusters.vrt -input_file_list listOfTiffs.txt
2. Qgis (TOOL: Sample Raster Values)
a)Input
./kmeans_clusters.vrt
./snic_seed_pixel_points.shp
b) Output column prefix
cluster_id
c) output
./LTOP_Kmeans_Cluster_IDs.shp
Since our sample contains points for every pixel in the Seed Image there are duplicate Kmeans Cluster ID values. These duplicates represent Kmeans Clusters that are spectrally simular, linked land class if you will. But we only need vector point with a unique Cluster ID. So here we randomly select points each of which has a unique Kmeans Cluster ID. So we end up with 5000 points for 5000 Kmeans Cluster IDs.
1) location
./LTOP_Oregon/scripts/kMeanClustering/randomDistinctSampleOfKmeansClusterIDs_v2.py
2) Edit in script parameters
a) input shp file:
./LTOP_Oregon_Kmeans_Cluster_IDs.shp
b) output shp file:
./LTOP_Kmeans_Cluster_IDs_5k_sample.shp
3) conda
conda activate geo_env
4) run script
python ./LTOP_Oregon/scripts/kMeanClustering/randomDistinctSampleOfKmeansClusterIDs_v2.py
Move the random subset of the Kmeans sample points up to GEE.
1) location
./LTOP_Kmeans_Cluster_IDs_5k_sample.shp
2) zip folder
zip -r .LTOP_Kmeans_Cluster_IDs_5k_sample.zip ./LTOP_Kmeans_Cluster_IDs_5k_sample/
3) upload to GEE
With the subset sample of Kmeans Cluster points, a point for each cluster ID, sample a time series of Landsat Imagery (TCB TCG TCW NBR and NDVI). This sample is exported as a table from GEE.
[Image of Table]
1. script local location
./LTOP_Oregon/scripts/GEEjs/03abstractSampler.js
2. copy and paste script into GEE console
2. Make sure you all needed dependencies
3. Review in script parameters.
4. Run script
5. Run tasks
task to drive
LTOP_Oregon_Abstract_Sample_annualSRcollection_Tranformed_NBRTCWTCGNDVIB5_v1.csv
Download the table
1) Download from Google Drive
LTOP_Oregon_Abstract_Sample_annualSRcollection_Tranformed_NBRTCWTCGNDVIB5_v1.csv
2) location (islay)
./LTOP_Oregon/tables/abstract_sample_gee/
Here we create an abstract image. We start with the table that contains a time series of spretral values for 5000 points. These points locations are moved to be adjsent to one aonther, and are turned into pixels with each observation in the time series a new image of pixels. This script exports a TIFF image for every year in the time series and a new point vector file at each pixel locaton.
[image of orignal points] [image of move points] [image of abstract image]
1) Script n
./csv_to_abstract_images_v2.py
2) Input
./LTOP_Oregon_Abstract_Sample_annualSRcollection_Tranformed_NBRTCWTCGNDVIB5_v1.csv
3) Outputs
a) image directory
./LTOP_Oregon/rasters/03_AbstractImage/
b) SHP directory
./LTOP_Oregon/vectors/03_abstract_image_pixel_points/
4) Conda
conda activate geo_env
5) Run Command
python csv_to_abstract_images.py
We then upload the abstarct images to GEE
1) Raster location
./LTOP_Oregon/rasters/03_AbstractImage/
2) make folder in GEE assets to hold all the images
3) Upload all images to assets folder
4) Make image collection in GEE assets tab
5) add each abstract image to image collection
Upload the shp file that acompanied the abstract image.
1) SHP file location
./LTOP_Oregon/vectors
2) zip files
zip -r 03_abstract_image_pixel_points.zip 03_abstract_image_pixel_points/
3) Upload to GEE
1. script local location
./LTOP_Oregon/scripts/GEEjs/04abstractImager.js
2. copy and paste script into GEE console
2. Make sure you all needed dependencies
3. Review in script parameters.
a) check to make sure runParams pasted correctly (super long list)
b) run script for each index 'NBR', 'NDVI', 'TCG', 'TCW', 'B5'
i) editing line 18 to change index name
4. Run script
5. Run tasks
task to drive (CSV)
LTOP_Oregon_abstractImageSamples_5000pts_v2/
LTOP_Oregon_abstractImageSample_5000pts_lt_144params_B5_v2.csv
LTOP_Oregon_abstractImageSample_5000pts_lt_144params_NBR_v2.csv
LTOP_Oregon_abstractImageSample_5000pts_lt_144params_NDVI_v2.csv
LTOP_Oregon_abstractImageSample_5000pts_lt_144params_TCG_v2.csv
LTOP_Oregon_abstractImageSample_5000pts_lt_144params_TCW_v2.csv
1) script location
./LTOP_Oregon/scripts/GEEjs/00_get_chunks_from_gdrive.py
2) Run Command
conda activate py35
python 00_get_chunks_from_gdrive.py LTOP_Oregon_abstractImageSamples_5000pts_v2 ./LTOP_Oregon/tables/LTOP_Oregon_Abstract_Image_LT_data/
3) output location
./LTOP_Oregon/tables/LTOP_Oregon_Abstract_Image_LT_data/
1) script locaton
./LTOP_Oregon/scripts/lt_seletor/01_ltop_lt_parameter_scoring.py
2) Edit line 119 as the input directory of csv files
a) input directory
./LTOP_Oregon/tables/LTOP_Oregon_Abstract_Image_LT_data/
3) Edit line 653 as the output csv file
a) output line 563
./LTOP_Oregon/tables/LTOP_Oregon_selected_config/LTOP_Oregon_selected_config.csv
4) run script
conda activate geo_env
python ./LTOP_Oregon/scripts/lt_seletor/01_ltop_lt_parameter_scoring.py
1) script location
./LTOP_Oregon/scripts/lt_seletor/02_ltop_select_top_parameter_configuration.py
2) Edit and review script
input file path line 6
./LTOP_Oregon/tables/LTOP_Oregon_config_scores/LTOP_Oregon_config_scores.csv
output file path line 7
./LTOP_Oregon/tables/LTOP_Oregon_selected_configurations/LTOP_Oregon_config_selected.csv
3) run script
conda base
python ./LTOP_Oregon/scripts/lt_seletor/02_ltop_select_top_parameter_configuration.py
1) CSV location
./LTOP_Oregon/tables/LTOP_Oregon_selected_configurations/LTOP_Oregon_config_selected.csv
2) Upload CSV as an asset to GEE
1) script location
./LTOP_Oregon/scripts/GEEjs/05lt-Optumum-Imager.js
2) Edit and review script
3) run script
4) Run Task
asset task
to drive task
0. Open terminal on Islay in a VNC
1. Script location
./LTOP_Oregon/scripts/GEEjs/
2. Activate conda environment “py35”
conda activate py35
3. Python script syntax
python 00_get_chunks_from_gdrive.py <google drive folder name> <local directory>
4. Run script
python 00_get_chunks_from_gdrive.py LTOP_Oregon_image_withVertYrs_NBR /LTOP_Oregon/rasters/04_LTOP_Image_NBR/
5. Check data at download destination.
./LTOP_Oregon/rasters/01_SNIC/