demo/test: add SanFranBaySenD42 for GMTSAR (#1229)

+ docs: add SanFranBaySenD42 for GMTSAR - add docs/templates/SanFranBaySenD42.txt for GMTSAR - demo_dataset.md: add SanFranBaySenD42 for GMTSAR with figure (with DOI from zenodo) - use demo fig from github/insarlab/design-doc repo, instead of Yunjun's obsolete personal website, for better independency. + tests: add GMTSAR into the regular circle ci testing using example dataset from zenodo - add tests/configs/SanFranBaySenD42.txt - tests/smallbaselineApp.py: add SanFranBaySenD42 dataset with a false URL for place holder - add SanFranBaySenD42 to the regular Circle CI integration testing + load water mask for gmtsar, by adding: - objects.stackDict.geometryDict.write2hdf5(): convert NaN value to zero. - prep_gmtsar.prepare_geometry(): add all supported geometry data files, including water mask file - prep_gmtsar: add comments on input/output parameters for all functions. + utils.readfile._attribute_gmtsar2roipac: bugfix for ORBIT_DIRECTION + optional dependency: pin gdal<3.9 for GMTSAR, as gdal-3.9 could not read the geo-coordinate info properly from GMT *.grd files, which is generated by GMTSAR.
insarlab · Jul 19, 2024 · 1a1f3e1 · 1a1f3e1
1 parent a8471b3
commit 1a1f3e1
Show file tree

Hide file tree

Showing 12 changed files with 203 additions and 61 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -33,7 +33,8 @@ jobs:
           command: |
             export PYTHONUNBUFFERED=1
             # install dependencies and source code
-            mamba install --verbose --yes gdal">=3" --file ${MINTPY_HOME}/requirements.txt
+            # pin gdal version because gdal-3.9 could not read the GMT-6 *.grd file properly
+            mamba install --verbose --yes --file ${MINTPY_HOME}/requirements.txt gdal"<3.9"
             python -m pip install ${MINTPY_HOME}
             # test installation
             smallbaselineApp.py -h
@@ -49,31 +50,37 @@ jobs:
             ${MINTPY_HOME}/tests/dem_error.py
 
       - run:
-          name: Integration Test 1/4 - FernandinaSenDT128 (ISCE/topsStack)
+          name: Integration Test 1 - FernandinaSenDT128 (ISCE2/topsStack)
           command: |
             mkdir -p ${HOME}/data
             ${MINTPY_HOME}/tests/smallbaselineApp.py --dir ${HOME}/data --dset FernandinaSenDT128
 
       - run:
-          name: Integration Test 2/4 - SanFranSenDT42 (ARIA)
+          name: Integration Test 2 - SanFranSenDT42 (ARIA)
           command: |
             mkdir -p ${HOME}/data
             ${MINTPY_HOME}/tests/smallbaselineApp.py --dir ${HOME}/data --dset SanFranSenDT42
 
       - run:
-          name: Integration Test 3/5 - RidgecrestSenDT71 (HyP3)
+          name: Integration Test 3 - RidgecrestSenDT71 (ASF HyP3)
           command: |
             mkdir -p ${HOME}/data
             ${MINTPY_HOME}/tests/smallbaselineApp.py --dir ${HOME}/data --dset RidgecrestSenDT71
 
       - run:
-          name: Integration Test 4/5 - WellsEnvD2T399 (Gamma)
+          name: Integration Test 4 - SanFranBaySenD42 (GMTSAR)
+          command: |
+            mkdir -p ${HOME}/data
+            ${MINTPY_HOME}/tests/smallbaselineApp.py --dir ${HOME}/data --dset SanFranBaySenD42
+
+      - run:
+          name: Integration Test 5 - WellsEnvD2T399 (Gamma)
           command: |
             mkdir -p ${HOME}/data
             ${MINTPY_HOME}/tests/smallbaselineApp.py --dir ${HOME}/data --dset WellsEnvD2T399
 
       - run:
-          name: Integration Test 5/5 - WCapeSenAT29 (SNAP)
+          name: Integration Test 6 - WCapeSenAT29 (SNAP)
           command: |
             mkdir -p ${HOME}/data
             ${MINTPY_HOME}/tests/smallbaselineApp.py --dir ${HOME}/data --dset WCapeSenAT29

diff --git a/docs/README.md b/docs/README.md
@@ -51,7 +51,7 @@ smallbaselineApp.py ${MINTPY_HOME}/docs/templates/FernandinaSenDT128.txt
 ```
 
 <p align="left">
-  <img width="600" src="https://yunjunzhang.files.wordpress.com/2019/06/fernandinasendt128_poi.jpg">
+  <img width="600" src="https://github.com/insarlab/design-docs/blob/main/docs/FernandinaSenDT128-ISCE2.jpg">
 </p>
 
 Results are plotted in **./pic** folder. To explore more data information and visualization, try the following scripts:

diff --git a/docs/demo_dataset.md b/docs/demo_dataset.md
@@ -1,6 +1,6 @@
 Here are example interferogram stacks pre-processed using different InSAR processors.
 
-### Sentinel-1 on Fernandina with ISCE ###
+### Sentinel-1 on Fernandina with ISCE2/topsStack ###
 
 + Area: Fernandina volcano at Galápagos Islands, Ecuador
 + Data: Sentinel-1 A/B descending track 128 during Dec 2014 - June 2018 (98 acquisitions; [Zenodo](https://zenodo.org/record/3952953))
@@ -14,7 +14,7 @@ smallbaselineApp.py ${MINTPY_HOME}/docs/templates/FernandinaSenDT128.txt
 ```
 
 <p align="left">
-  <img width="650" src="https://yunjunzhang.files.wordpress.com/2019/06/fernandinasendt128_poi.jpg">
+  <img width="650" src="https://github.com/insarlab/design-docs/blob/main/docs/FernandinaSenDT128-ISCE2.jpg">
 </p>
 
 Relevant literature:
@@ -35,7 +35,41 @@ smallbaselineApp.py ${MINTPY_HOME}/docs/templates/SanFranSenDT42.txt
 ```
 
 <p align="left">
-  <img width="650" src="https://yunjunzhang.files.wordpress.com/2020/11/sanfransendt42_transect.jpg">
+  <img width="650" src="https://github.com/insarlab/design-docs/blob/main/docs/SanFranSenDT42-ARIA.jpg">
+</p>
+
+Relevant literature:
+
++ Chaussard, E., R. Bürgmann, H. Fattahi, R. M. Nadeau, T. Taira, C. W. Johnson, and I. Johanson (2015), Potential for larger earthquakes in the East San Francisco Bay Area due to the direct connection between the Hayward and Calaveras Faults, _Geophysical Research Letters,_ 42(8), 2734-2741, doi:10.1002/2015GL063575.
+
+### Sentinel-1 of the 2019 Ridgecrest, California earthquake sequence with ASF HyP3 ###
+
++ Area: Owens Valley, California, USA ([USGS event page](https://earthquake.usgs.gov/earthquakes/eventpage/ci38457511/executive))
++ Data: Sentinel-1 descending track 71 during June - August 2019 (7 acquisitions; [Zenodo](https://zenodo.org/record/11049257))
++ Size: ~240 MB
+
+```bash
+wget https://zenodo.org/record/11049257/files/RidgecrestSenDT71.tar.xz
+tar -xvJf RidgecrestSenDT71.tar.xz
+cd RidgecrestSenDT71
+smallbaselineApp.py ${MINTPY_HOME}/docs/templates/RidgecrestSenDT71.txt
+```
+
+### Sentinel-1 on San Francisco Bay with GMTSAR ###
+
++ Area: San Francisco Bay, California, USA
++ Data: Sentinel-1 A/B descending track 42 during December 2014 - June 2024 (333 acquisitoins; [Zenodo](https://zenodo.org/records/12773014))
++ Size: ~2.3 GB
+
+```bash
+wget https://zenodo.org/records/12773014/files/SanFranBaySenD42.tar.xz
+tar -xvJf SanFranBaySenD42.tar.xz
+cd SanFranBaySenD42
+smallbaselineApp.py ${MINTPY_HOME}/docs/templates/SanFranBaySenD42.txt
+```
+
+<p align="left">
+  <img width="600" src="https://github.com/insarlab/design-docs/blob/main/docs/SanFranBaySenD42-GMTSAR.jpg">
 </p>
 
 Relevant literature:
@@ -56,26 +90,13 @@ smallbaselineApp.py ${MINTPY_HOME}/docs/templates/WellsEnvD2T399.txt
 ```
 
 <p align="left">
-  <img width="650" src="https://yunjunzhang.files.wordpress.com/2019/06/wellsenvd2t399_co_poi.jpg">
+  <img width="650" src="https://github.com/insarlab/design-docs/blob/main/docs/WellsEnvD2T399-Gamma.jpg">
 </p>
 
 Relevant literature:
 
 + Nealy, J. L., H. M. Benz, G. P. Hayes, E. A. Bergman, and W. D. Barnhart (2017), The 2008 Wells, Nevada, Earthquake Sequence: Source Constraints Using Calibrated Multiple‐Event Relocation and InSAR, _Bulletin of the Seismological Society of America_, 107(3), 1107-1117, doi:10.1785/0120160298.
 
-### Sentinel-1 of the 2019 Ridgecrest, California earthquake sequence with HyP3 ###
-
-+ Area: Owens Valley, California, USA ([USGS event page](https://earthquake.usgs.gov/earthquakes/eventpage/ci38457511/executive))
-+ Data: Sentinel-1 descending track 71 during June - August 2019 (7 acquisitions; [Zenodo](https://zenodo.org/record/11049257))
-+ Size: ~240 MB
-
-```bash
-wget https://zenodo.org/record/11049257/files/RidgecrestSenDT71.tar.xz
-tar -xvJf RidgecrestSenDT71.tar.xz
-cd RidgecrestSenDT71
-smallbaselineApp.py ${MINTPY_HOME}/docs/templates/RidgecrestSenDT71.txt
-```
-
 ### Sentinel-1 on Western Cape, South Africa with SNAP ###
 
 + Area: West coast of Western Cape province, South Africa
@@ -103,7 +124,7 @@ smallbaselineApp.py ${MINTPY_HOME}/docs/templates/KujuAlosAT422F650.txt
 ```
 
 <p align="left">
-  <img width="650" src="https://yunjunzhang.files.wordpress.com/2019/06/kujualosat422f650_vel.jpg">
+  <img width="650" src="https://github.com/insarlab/design-docs/blob/main/docs/KujuAlosAT422F650-ROIPAC.jpg">
 </p>
 
 Relevant literature:

diff --git a/docs/dir_structure.md b/docs/dir_structure.md
@@ -463,9 +463,10 @@ $DATA_DIR/SanFranBaySenD42
 ├── baseline_table.dat
 ├── supermaster.PRM
 ├── geometry
-|   ├── azimuth_angle.grd
 |   ├── dem.grd
-│   └── incidence_angle.grd
+|   ├── azimuth_angle.grd
+│   ├── incidence_angle.grd
+│   └── water_mask.grd
 ├── interferograms
 │   ├── 2004114_2004324
 │   │   ├── corr_ll.grd
@@ -485,16 +486,17 @@ RLOOKS          = 32         #[int], number of looks in the range direction
 HEADING         = -168.0     #[float], satellite heading angle, measured from the north in clockwise as positive
                              # One could open the *.kml file in Google Earth and measure it manually
 
-mintpy.load.processor    = gmtsar
-mintpy.load.metaFile     = $DATA_DIR/SanFranBaySenD42/supermaster.PRM
-mintpy.load.baselineDir  = $DATA_DIR/SanFranBaySenD42/baseline_table.dat
+mintpy.load.processor     = gmtsar
+mintpy.load.metaFile      = $DATA_DIR/SanFranBaySenD42/supermaster.PRM
+mintpy.load.baselineDir   = $DATA_DIR/SanFranBaySenD42/baseline_table.dat
 ##---------interferogram datasets:
-mintpy.load.unwFile      = $DATA_DIR/SanFranBaySenD42/interferograms/*/unwrap_ll*.grd
-mintpy.load.corFile      = $DATA_DIR/SanFranBaySenD42/interferograms/*/corr_ll*.grd
+mintpy.load.unwFile       = $DATA_DIR/SanFranBaySenD42/interferograms/*/unwrap_ll*.grd
+mintpy.load.corFile       = $DATA_DIR/SanFranBaySenD42/interferograms/*/corr_ll*.grd
 ##---------geometry datasets:
-mintpy.load.demFile      = $DATA_DIR/SanFranBaySenD42/geometry/dem*.grd
-mintpy.load.incAngleFile = $DATA_DIR/SanFranBaySenD42/geometry/incidence_angle.grd
-mintpy.load.azAngleFile  = $DATA_DIR/SanFranBaySenD42/geometry/azimuth_angle.grd
+mintpy.load.demFile       = $DATA_DIR/SanFranBaySenD42/geometry/dem.grd
+mintpy.load.incAngleFile  = $DATA_DIR/SanFranBaySenD42/geometry/incidence_angle.grd
+mintpy.load.azAngleFile   = $DATA_DIR/SanFranBaySenD42/geometry/azimuth_angle.grd
+mintpy.load.waterMaskFile = $DATA_DIR/SanFranBaySenD42/geometry/water_mask.grd
 ```
 
 ### Gamma ###

diff --git a/docs/installation.md b/docs/installation.md
@@ -79,16 +79,18 @@ bash Miniforge3-Linux-x86_64.sh -b -p ~/tools/miniforge
 Install dependencies into a new environment, e.g. named "insar":
 
 ```bash
-# Add "gdal'>=3'" below to install extra dependencies if you use ARIA, FRInGE, HyP3 or GMTSAR
-# Add "isce2"     below to install extra dependencies if you use ISCE-2
+# Add "isce2"      below to install extra dependencies if you use ISCE-2
+# Add "gdal"       below to install extra dependencies if you use ARIA, FRInGE or HyP3
+# Add "gdal'<3.9'" below to install extra dependencies if you use GMTSAR
 mamba create --name insar --file ~/tools/MintPy/requirements.txt
 ```
 
 or install dependencies into an existing environment:
 
 ```bash
-# Add "gdal'>=3'" below to install extra dependencies if you use ARIA, FRInGE, HyP3 or GMTSAR
-# Add "isce2"     below to install extra dependencies if you use ISCE-2
+# Add "isce2"      below to install extra dependencies if you use ISCE-2
+# Add "gdal"       below to install extra dependencies if you use ARIA, FRInGE or HyP3
+# Add "gdal'<3.9'" below to install extra dependencies if you use GMTSAR
 mamba update --name my-existing-env --file ~/tools/MintPy/requirements.txt
 ```
 

diff --git a/docs/templates/SanFranBaySenD42.txt b/docs/templates/SanFranBaySenD42.txt
@@ -0,0 +1,28 @@
+# vim: set filetype=cfg:
+## manually add the following metadata
+HEADING = -168
+ALOOKS  = 8
+RLOOKS  = 32
+
+########## computing resource configuration
+mintpy.compute.maxMemory    = 8
+mintpy.compute.cluster      = local
+mintpy.compute.numWorker    = 4
+
+########## 1. Load Data (--load to exit after this step)
+mintpy.load.processor       = gmtsar    #[isce, aria, hyp3, gmtsar, snap, gamma, roipac], auto for isce
+mintpy.load.metaFile        = ../supermaster.PRM
+mintpy.load.baselineDir     = ../baseline_table.dat
+##---------interferogram datasets:
+mintpy.load.unwFile         = ../interferograms/*/unwrap_ll*.grd
+mintpy.load.corFile         = ../interferograms/*/corr_ll*.grd
+##---------geometry datasets:
+mintpy.load.demFile         = ../geometry/dem.grd
+mintpy.load.incAngleFile    = ../geometry/incidence_angle.grd
+mintpy.load.azAngleFile     = ../geometry/azimuth_angle.grd
+mintpy.load.waterMaskFile   = ../geometry/water_mask.grd
+
+mintpy.reference.lalo           = 37.69, -122.07
+mintpy.troposphericDelay.method = pyaps
+mintpy.deramp                   = no
+mintpy.topographicResidual      = yes
diff --git a/src/mintpy/objects/stackDict.py b/src/mintpy/objects/stackDict.py
@@ -695,10 +695,7 @@ def write2hdf5(self, outputFile='geometryRadar.h5', access_mode='w', box=None, x
                                                              c=str(compression)))
 
                     # read
-                    data = np.array(self.read(family=dsName,
-                                              box=box,
-                                              xstep=xstep,
-                                              ystep=ystep)[0], dtype=dsDataType)
+                    data = self.read(family=dsName, box=box, xstep=xstep, ystep=ystep)[0]
 
                     # water body: -1/True  for water and 0/False for land
                     # water mask:  0/False for water and 1/True  for land
@@ -708,6 +705,12 @@ def write2hdf5(self, outputFile='geometryRadar.h5', access_mode='w', box=None, x
                         print('    input file "{}" is water body (True/False for water/land), '
                               'convert to water mask (False/True for water/land).'.format(fname))
 
+                    elif dsName == 'waterMask':
+                        # GMTSAR water/land mask: 1 for land, and nan for water / no data
+                        if np.sum(np.isnan(data)) > 0:
+                            print('    convert NaN value for waterMask to zero.')
+                            data[np.isnan(data)] = 0
+
                     elif dsName == 'height':
                         noDataValueDEM = -32768
                         if np.any(data == noDataValueDEM):
@@ -745,6 +748,7 @@ def write2hdf5(self, outputFile='geometryRadar.h5', access_mode='w', box=None, x
                                 data = ut.wrap(data, wrap_range=[-180, 180])    # rewrap within -180 to 180
 
                     # write
+                    data = np.array(data, dtype=dsDataType)
                     ds = f.create_dataset(dsName,
                                           data=data,
                                           chunks=True,