Framing Convention

[cmarshak]

@dbekaert:

Had a very productive conversation at the Hyp3-ARIA coordination meeting. Basically topo is taking too long with current frame definitions and our “jerry-rigging” of ISCE2 using the region-of-interest parameter: the bug ticket #93 has more details

Some reasonable solutions could be:

1. Decreasing DEM size: we are doing floor/ceil on the dem bounds because it was suggested ISCE2 needed such integer bounds - see reference in Bug ticket
2. Decreasing frame size - currently we are using 10 bursts with 2 overlaps. Might be able to get away with less since the region of interest bounding box (I believe) adds a burst to the GUNW and ensures larger overlap.

Marin suggested running topo on GPU, but such a feature would be a huge lift even if Marin led it. If there are other parameters to help, please let me know.

Questions:

1. What’s the minimum frame size we could reasonably expect good results (particularly with unwrapping) given that our “region-of-interest” approach is likely going to increase the GUNW size and the overlap between GUNW?
2. Could we decrease the overlap to 1 burst between GUNW products?

Any other ideas generally?

[cmarshak]

Including the comment from the PR here: #92 (comment)

This is from an 8 burst with 1 burst overlap guaranteed.

Sharing the GUNW using the frame-id (example from the README) [link]. For additional context, here are some geojsons of the frame 22438 and its bounding box.

I noted in the bug ticket that there were confounding factors with available resources, but currently, I do not notice any issue with processing time.

The frame above is taking ~1.5 hours for the processing as expected.

TopsApp Steps:   0%|          | 0/24 [00:00<?, ?it/s]
TopsApp Steps:   4%|▍         | 1/24 [00:00<00:10,  2.27it/s]
TopsApp Steps:   8%|▊         | 2/24 [01:39<21:26, 58.49s/it]
TopsApp Steps:  12%|█▎        | 3/24 [01:42<11:30, 32.90s/it]
TopsApp Steps:  17%|█▋        | 4/24 [01:42<06:41, 20.10s/it]
TopsApp Steps:  21%|██        | 5/24 [34:40<3:49:59, 726.31s/it]
TopsApp Steps:  25%|██▌       | 6/24 [34:41<2:23:51, 479.53s/it]
TopsApp Steps:  29%|██▉       | 7/24 [34:41<1:31:29, 322.92s/it]
TopsApp Steps:  33%|███▎      | 8/24 [34:42<58:44, 220.28s/it]  
TopsApp Steps:  38%|███▊      | 9/24 [34:42<37:53, 151.56s/it]
TopsApp Steps:  42%|████▏     | 10/24 [34:43<24:28, 104.92s/it]
TopsApp Steps:  46%|████▌     | 11/24 [34:43<15:48, 72.96s/it] 
TopsApp Steps:  50%|█████     | 12/24 [34:44<10:10, 50.91s/it]
TopsApp Steps:  54%|█████▍    | 13/24 [51:00<1:00:43, 331.19s/it]
TopsApp Steps:  58%|█████▊    | 14/24 [1:01:39<1:10:42, 424.28s/it]
TopsApp Steps:  62%|██████▎   | 15/24 [1:01:42<44:33, 297.04s/it]  
TopsApp Steps:  67%|██████▋   | 16/24 [1:17:04<1:04:41, 485.14s/it]
TopsApp Steps:  71%|███████   | 17/24 [1:18:40<42:57, 368.27s/it]  
TopsApp Steps:  75%|███████▌  | 18/24 [1:18:53<26:09, 261.56s/it]
TopsApp Steps:  79%|███████▉  | 19/24 [1:25:19<24:55, 299.03s/it]
TopsApp Steps:  83%|████████▎ | 20/24 [1:25:20<13:57, 209.39s/it]
TopsApp Steps:  88%|████████▊ | 21/24 [1:26:41<08:32, 170.99s/it]
TopsApp Steps:  92%|█████████▏| 22/24 [1:26:42<03:59, 119.83s/it]
TopsApp Steps:  96%|█████████▌| 23/24 [1:26:42<01:24, 84.02s/it] 
TopsApp Steps: 100%|██████████| 24/24 [1:26:43<00:00, 58.96s/it]
TopsApp Steps: 100%|██████████| 24/24 [1:26:43<00:00, 216.81s/it]

As noted in the isce.log there are about 11 to 12 bursts per swath i.e. 1-2 on each boundary are added due to our need to specify a bbox in ISCE2.

####################################################################################################
    runComputeBaseline
----------------------------------------------------------------------------------------------------
baseline.IW-1 Bpar at midrange for first common burst = -36.04160300831371
baseline.IW-1 Bpar at midrange for last common burst = -39.4209536637765
baseline.IW-1 Bperp at midrange for first common burst = 47.352539109543734
baseline.IW-1 Bperp at midrange for last common burst = 53.76018676783484
baseline.IW-1 First common burst in reference = 0
baseline.IW-1 First common burst in secondary = 0
baseline.IW-1 Last common burst in reference = 12
baseline.IW-1 Last common burst in secondary = 12
baseline.IW-1 Number of bursts in reference = 12
baseline.IW-1 Number of bursts in secondary = 12
baseline.IW-1 Number of common bursts = 12
baseline.IW-2 Bpar at midrange for first common burst = -39.87334932713999
baseline.IW-2 Bpar at midrange for last common burst = -43.7524507360454
baseline.IW-2 Bperp at midrange for first common burst = 44.477662323577086
baseline.IW-2 Bperp at midrange for last common burst = 50.595975915712884
baseline.IW-2 First common burst in reference = 0
baseline.IW-2 First common burst in secondary = 0
baseline.IW-2 Last common burst in reference = 12
baseline.IW-2 Last common burst in secondary = 12
baseline.IW-2 Number of bursts in reference = 12
baseline.IW-2 Number of bursts in secondary = 12
baseline.IW-2 Number of common bursts = 12
baseline.IW-3 Bpar at midrange for first common burst = -43.33677388123395
baseline.IW-3 Bpar at midrange for last common burst = -47.22672798274666
baseline.IW-3 Bperp at midrange for first common burst = 42.378930259911115
baseline.IW-3 Bperp at midrange for last common burst = 47.688060313717685
baseline.IW-3 First common burst in reference = 0
baseline.IW-3 First common burst in secondary = 0
baseline.IW-3 Last common burst in reference = 11
baseline.IW-3 Last common burst in secondary = 11
baseline.IW-3 Number of bursts in reference = 11
baseline.IW-3 Number of bursts in secondary = 11
baseline.IW-3 Number of common bursts = 11

[dbekaert]

The GPU approach goes two way.
1)The instances types are more in use, generally more expensive and you still need to run the remainder part of the workflow that is not GPU enabled. Not sure about the cost benefit here to spend the investment on it.
2) GPU might be attractive for the case we go to urgent response and run the workflow with gpu enabled for both the TOPO and the dense-offsets etc.

[dbekaert]

For the unwrapping the question should be more, what is a too large frame size that you dont have the computational resources at hand. Again the 1-2 burst should not be the issue, but could it be more burst sometimes? Just get a feel for yourself when things start to break down resource wize is good.

[cmarshak]

Ok - will look into.

Would be a good experiment in the cloud, but we just removed "region-of-interest" to rely more heavily on a fixed frame convention. Therefore, will have to be a manual approach.

Will have to do such an experiment on leffe.

I do think our current approach with 8 bursts and 1 overlap for each adjacent frame will match the current costs/size of GUNWs and make sure that there is no edge disk space issues due to not requiring 3 SLCs for reference/secondary. So this should not be a blocker.

[cmarshak]

Here is overlapping frame from above:

Here is the same data for an overlapping frame

GUNW: https://gunw-development-testing.s3.us-west-2.amazonaws.com/frame-testing/S1-GUNW-D-R-144-tops-20230125_20221220-140045-00121W_00035N-PP-4569-v2_0_6.nc
Frame 22439: https://gunw-development-testing.s3.us-west-2.amazonaws.com/frame-testing/22439_frame.geojson
Bbox: https://gunw-development-testing.s3.us-west-2.amazonaws.com/frame-testing/22439_box.geojson

Weirdly, one of the bursts is omitted or at least relative to the ESA burst map. There is still plenty of overlap.

Here are some screenshots from QGIS.

The gray vector area is the frame from fixed frames we have:

And here is the overlap - both amplitude images are transparent so the darker area is where they overlap:

[cmarshak]

Using the current hyp3 test job, I generated some test GUNWs using this framing convention with an ionosphere correction layer.

For 18 products over Tibet, I saw the following runtime stats:

2.53 hours +/-  7.25  minutes
Max:  2.77  hours
Min:  2.45  hours

I think this sounds reasonable in terms of processing time given without ionosphere we saw about 1.5 hours. This is adding ~67% extra time on average.

[cmarshak]

Generating 15 GUNWs at an area over Los Angeles, the processing time with the current frame convention and ionosphere was a little more than double.

3 hours, 12 minutes and 2.32 seconds +/- 14 minutes and 6.70 seconds

The maximum was 3 hours, 26 minutes and 49.43 seconds.