QA reports for science-mode L0B files?

[gmgunter]

@rad-eng-59 relayed a request for adding some level of QA support for NISAR L0Bs (specifically science-mode L0Bs -- not DM1 or DM2 products that have significantly different data layouts).

I wonder if our QA can support L0B product and generate similar plots to RSLC such as raster , range spectrum. and histogram of I/Q. Basically, leave this part of L0B QA to L1 QA given our functionalities are codes are already available in L1 QA and perhaps, our QA can apply the same functionality to L0B QA. We simply leave L0B QA (Alice's tool) to deal with telemetry info and key calibration parameters such as chirp correlator, caltone, etc. I'm working on putting together some script to assess these key Calib fields in L0B as part of Alice's QA. What do you say? Thanks

This will save a lot of time and hassle for SDS while fully covering the list of requests from us and SE team in regards to processing range lines as part of L0B QA.

Please check out the items 4-7 highlighted in red in https://wiki.jpl.nasa.gov/pages/viewpage.action?pageId=892336039

@gmgunter will investigate the feasibility of integrating a product reader for science-mode L0B products into the nisarqa product reader class hierarchy. The reader only needs to access the minimal subset of product contents necessary to generate plots such as spectra, histograms, etc.

A possible alternative to adding a QA module for L0B products would be to add nisarqa as a dependency of the existing L0B QA tool so that that software can use the existing functions to plot spectra, histograms, etc (assuming that these functions are sufficiently modular/reusable).

cc @hfattahi @nemo794

[gmgunter]

A few thoughts--

In order to generate backscatter/phase histograms, the hypothetical L0B QA module would want to call the following nisarqa functions:

• process_backscatter_and_phase_histograms()
• process_range_spectra()

In general, L0B products may have non-uniform PRIs so generating azimuth spectra is nontrivial. If we want to be able to compute azimuth spectra for L0B products with fixed PRI, however, the module should additionally call

• process_azimuth_spectra()

process_backscatter_and_phase_histograms currently takes any NonInsarProduct, whereas process_range_spectra (and process_azimuth_spectra) currently only operate on RSLC products. But these type annotations could be relaxed a bit. In practice, the input product object only needs to implement the following attributes and methods in order for us to compute histograms and range spectra:

• product.band
• product.get_pols(freq=...)
• product.get_raster(freq=..., pol=...)
• product.get_slant_range_spacing(freq=...)
• product.get_processed_center_frequency(freq=...)

In order to compute azimuth spectra for products with fixed PRI, it would additionally need to implement

• product.get_zero_doppler_time_spacing()¹

So, in theory, it should be sufficient to implement an RRSD product reader class with the above attributes and methods in order to compute histograms and spectra using the existing nisarqa functionality.

Integrating this RRSD class into the existing product reader class hierarchy is another story. The main obstacle to this is that the QA code assumes that NISAR products will contain metadata LUTs, but these are absent in L0B products:

• The base NisarProduct class assumes that the product contains metadata cubes in a ../metadata/geolocationGrid/ or ../metadata/radarGrid/ group. L0B products don't have these.
• Similarly, NonInsarProducts are assumed to have calibration metadata LUTs (noise equivalent backscatter and elevation antenna pattern) under ../metadata/calibrationInformation/, which the L0B products do not contain.

So we'd have to re-conceptualize the product reader class hierarchy a bit to distinguish L1/L2 products that contain metadata LUTs from L0 products that don't. Otherwise, AFAICT there doesn't seem to be any reason that RRSD couldn't inherit from NisarRadarProduct and NonInsarProduct (although it'd have to override some methods from those parent classes in order to handle the differing structure of L0B products).

If I had to estimate the scope of work, I'd guess that it's probably not less than 3-4 weeks of effort to implement an RRSD class within the product reader hierarchy and add a basic L0B QA workflow that generates a PDF report with histograms and spectra.

Footnotes

1. Note that the method name is a bit awkward since L0B product image layers are not defined on a zero-Doppler coordinate grid, but really this method just needs to return the azimuth time spacing of the image grid -- it shouldn't actually matter whether the grid is squinted or in zero-Doppler geometry for computing spectra. ↩

[nemo794]

Thank you for this assessment, @gmgunter !

Couple of thoughts:

• The initial request also mentioned "raster". Does that mean plotting the unfocused image rasters, or..?
• In addition to the PDF, the QA functions mentioned above also populate the computed histograms, etc. into a STATS.h5 file. This is currently non-optional; the HDF5 file handle is required, and it will always be populated within these functions. If L0B wants the STATS.h5 file, great! If not, then that will take extra work. (Or, the HDF5 file could simply be generated and discarded; it adds negligible runtime.)

[rad-eng-59]

Thanks @gmgunter and @nemo794 for your quick and thorough thoughts.
A couple of comments for clarification:

• AZ spectrum is not originally requested by the team and not a priority. The focus is mainly on range line behavior ( fast time direction). If time allows and the work is not trivial, perhaps having simple AZ spectrum plots for fixed PRF would be great. The nominal PRF, ref epoch, and whether PRF is dithering or not can be readily extracted from PulseTime field altogether . Check out Raw parser class in ISCE3.
• Chirp parameters (slope, center frequency) and slant range spacing under each frequency band should give you all needed in range direction. Note that chirp slope( bandwidth) and center frequency can be zero for a QP case with split-spectrum on frequency band "B". I wouldn't worry about it and simply let your script do its job as usual. The respective range spectrum will have noise-only that still shows the full correct spectrum but labeled with zero center frequency.
• requested raster and histograms plots are all for raw data as it is (complex floating point deBFPQed data, of course). SE team prefers histograms of I/Q instead of Amp/phase if it is trivial work on your side.
  Just a reminder that as opposed to other higher level products you would need to use BFPQLUT dataset (stored under the same group as polarimetric datasets HH, HV, ...) to convert compound 16-bit integer into Complex float 64-bit. Note that all BFPQLUT datasets are exacly the same! Alternatively, You may want to use decoder class under NISAR producr readers.
• Having a Stats HDF5 product would be great given such a feature does not currently exists in L0B QA!

Thank you so much guys!

[hfattahi]

Thanks @gmgunter @nemo794 and @rad-eng-59 for the detailed discussion. Clearly, the L0B QA will require reorganization of the product readers in nisarqa which is currently out of scope for us. If the scope of our work for nisarqa will change, we will revisit this topic.