GUVI imaging updates #198

aburrell · 2023-07-20T20:43:52Z

Description

Addresses #197 by adding a custom concat_data method to the TIMED GUVI instrument. This requires the pysat pull request: pysat/pysat#1131 to work.

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Load TIMED GUVI data for the sdr-imaging tag. Check and see that the Auroral data does not have the 'time' dimension, rather it has only the 'time_auroral' dimension (in addition to other appropriate dimensions). Also, the extent of the 'time_auroral' dimension should be limited to the period of time encompassed by 'time'.

import datetime as dt
import pysat
import pysatNASA

guvi = pysat.Instrument(inst_module=pysatNASA.instruments.timed_guvi, tag='sdr-imaging', inst_id='low_res')
guvi.load(date=dt.datetime(2005, 6, 29), use_header=True)
guvi['DISK_RECTIFIED_INTENSITY_DAY_AURORAL']

Yields:

<xarray.DataArray 'DISK_RECTIFIED_INTENSITY_DAY_AURORAL' (nCrossDayAur: 13, time_auroral: 5694, nchan: 5)>
array([[[7209.4736 , 4871.778  ,  590.9235 ,  730.5943 ,  717.60645],
        [6576.525  , 4667.561  ,  621.9355 ,  707.2645 ,  665.0437 ],
        [6809.1646 , 4822.3955 ,  596.235  ,  748.06116,  691.1846 ],
        ...,
        [4474.8916 , 2927.6484 ,  298.7434 ,  364.07367,  367.03128],
        [4596.163  , 3004.093  ,  346.05643,  425.2893 ,  366.12378],
        [4790.0947 , 3107.468  ,  339.42386,  416.37698,  412.0753 ]],

       [[7700.39   , 5367.413  ,  588.6728 ,  689.4741 ,  633.90454],
        [7502.0303 , 5343.9834 ,  563.9284 ,  690.71356,  622.1614 ],
        [7610.072  , 5227.0176 ,  579.8522 ,  672.1343 ,  624.2945 ],
        ...,
        [5129.152  , 3147.917  ,  272.6081 ,  374.8297 ,  332.13535],
        [4943.952  , 3176.0396 ,  301.06866,  390.83847,  325.5044 ],
        [5508.1694 , 3418.333  ,  295.7247 ,  376.09515,  322.656  ]],

       [[8324.27   , 5803.675  ,  538.3639 ,  673.33435,  598.7233 ],
        [8609.587  , 5865.27   ,  522.56   ,  644.01355,  619.9537 ],
        [8638.347  , 5787.5786 ,  531.3132 ,  659.1144 ,  591.5215 ],
        ...,
...
        [3811.3555 , 1985.8969 ,  318.80933,  288.66934,  318.9553 ],
        [3880.6448 , 2146.9849 ,  216.54195,  299.55103,  377.7116 ],
        [4197.2637 , 2367.9148 ,  255.80391,  360.47253,  444.9939 ]],

       [[5934.3174 , 4275.513  ,  729.7753 ,  916.8984 ,  999.6828 ],
        [6192.6875 , 4144.6416 ,  653.86615,  805.297  ,  850.64716],
        [6697.9395 , 4189.3247 ,  737.42444,  809.34186, 1050.4926 ],
        ...,
        [3662.6667 , 2056.6392 ,  273.36102,  287.7209 ,  266.2291 ],
        [3729.201  , 2185.1753 ,  240.62947,  369.76355,  333.78735],
        [4228.927  , 2228.7437 ,  278.9088 ,  358.12225,  396.88416]],

       [[6082.5977 , 4183.329  ,  782.99896,  914.4199 ,  966.175  ],
        [6344.678  , 3610.2083 ,  769.0373 ,  906.5941 ,  961.9961 ],
        [6586.852  , 4194.138  ,  791.0451 ,  772.94763,  916.8184 ],
        ...,
        [3534.3728 , 1967.3213 ,  320.33307,  286.80234,  449.809  ],
        [3753.4165 , 2096.4358 ,  278.61523,  340.76562,  396.22098],
        [3862.314  , 2027.921  ,  258.25385,  369.7472 ,  472.15524]]],
      dtype=float32)
Coordinates:
  * nchan                              (nchan) <U8 '121.6nm' ... 'LBHlong'
    PIERCEPOINT_DAY_LATITUDE_AURORAL   (nCrossDayAur, time_auroral) float32 4...
    PIERCEPOINT_DAY_LONGITUDE_AURORAL  (nCrossDayAur, time_auroral) float32 2...
    PIERCEPOINT_DAY_SZA_AURORAL        (nCrossDayAur, time_auroral) float32 3...
  * time_auroral                       (time_auroral) datetime64[ns] 2005-06-...
  * nCrossDayAur                       (nCrossDayAur) int64 0 1 2 3 ... 10 11 12

Also, there should be no warnings raised when loading the data, as there is meta data for all variables and cleaning applied to these data tags/inst IDs.

Finally, unit tests should be run locally against pysat/pysat#1131 to ensure appropriate behaviour with updated pysat handling.

Test Configuration:

Operating system: OS X Big Sur
Version number: Python 3.9
Any details about your local setup that are relevant: requires pysat branch inst_concat_data_method

Checklist:

If this is a release PR, replace the first item of the above checklist with the release
checklist on the wiki: https://github.com/pysat/pysat/wiki/Checklist-for-Release

Fixed the JHU APL load routines to handle an empty file list.

Updated labels definition to use meta labels attributes instead of strings and keys.

Updated the TIMED GUVI clean method for imaging data.

Fixed the dictionary formatting for `_clean_warn`.

Allow for channels when masking for the clean routine.

Removed extra whitespace and some typo characters in the docstring.

Added a concat_data method to fix issues loading data.

Remove the time sorting in `concat_data`, as data should be concatenated in the desired order.

Updated the changelog with a summary of the changes in this branch.

Added a whitespace after the `if` keyword. Also replaced hardcoded key with the appropriate variable.

Added metadata for the 'time_auroral' and 'nCross' variables.

jklenzing · 2023-10-06T18:51:08Z

What's the status of this pull? Is it waiting on a new pysat release?

aburrell · 2023-10-06T19:48:40Z

This requires the pysat pull request: pysat/pysat#1131 to work.

Yes, it requires a pysat release.

MAINT: update RC tests

jklenzing · 2023-11-30T21:36:02Z

Testing this branch against the pysat 3.2.0 RC, I see the following failures (for guvi):

test_load_multiple_days fails for tag='sdr-spectrograph' (all inst_ids)
- AssertionError, assert len(np.unique(self.test_inst.index.day)) > 1
test_load_w_pad fails for tag='sdr-imaging' (all inst_ids) with combine_times=True
- ValueError: cannot rename 'time_auroral' because it is not found in the dimensions of this dataset ()

Full error output at https://github.com/pysat/pysatNASA/actions/runs/7051616139/job/19194923688
Some of this has been documented in #218

Pull Develop changes into guvi branch

jklenzing · 2023-12-19T19:31:32Z

@aburrell, I think to finish this one we just need to fix the test_load_with_pad test (see above comment). The other guvi error can be handled separately.

Fix time where microseconds can become too big from rounding.

Updated type equalities to use `isinstance`.

Added a `linkcheck_ignore` statement, as the GATS page checks to see if the link request is human or machine.

Ensured that only dimensions with data are renamed. Also updated test date to highlight a time with better data coverage across all instruments.

aburrell · 2024-03-04T16:02:16Z

@jklenzing I fixed both GUVI errors. The first one was a data-availability issue. I picked test days with better coverage and everything looks good.

Update TIMED GUVI test dates, as coverage needs are different for different instruments.

jklenzing · 2024-03-05T15:28:50Z

TODO:

merge BUG/MAINT: cdaweb downloads, ICON IVM-B tests, new flag to skip new tests #223 pending successful run after comment update
merge changes to develop in here for better compliance with new pysat RC
run this branch against latest RC

jklenzing · 2024-03-06T16:55:57Z

Running vs the RC, there is a segmentation fault in the ubuntu branch: https://github.com/pysat/pysatNASA/actions/runs/8163504069

Reran the branch, happened again. The develop branch passes without issue.

jklenzing

Verified that the new lines of code are being run when using pysat 3.2.0rc6 (which accounts for the drop in coverage in this PR). Not sure why the segmentation fault appears in ubuntu but not the other ones.

pysatNASA/instruments/timed_guvi.py

jklenzing · 2024-03-07T15:36:44Z

Fixed the segmentation fault issue. Applying the skip test flags to the high-res data only allows the new concat routines to be tested, while avoiding memory issues on GA. https://github.com/pysat/pysatNASA/actions/runs/8189899989

jklenzing · 2024-03-07T15:47:29Z

From the tests:

Run with	Stmts	Miss	Cover
pysat 3.1.0	78	27	65%
pysat 3.2.0rc6	78	5	94%

aburrell added 9 commits July 20, 2023 10:45

BUG: fixed load without data

30447f0

Fixed the JHU APL load routines to handle an empty file list.

STY: updated labels

2958226

Updated labels definition to use meta labels attributes instead of strings and keys.

ENH: updated clean method

f72e2ab

Updated the TIMED GUVI clean method for imaging data.

BUG: fixed dict format

d5053c2

Fixed the dictionary formatting for `_clean_warn`.

BUG: expanded masked dimensions

44c6a36

Allow for channels when masking for the clean routine.

STY: removed whitespace

fcedfed

Removed extra whitespace and some typo characters in the docstring.

ENH: added concat_data method

068d54a

Added a concat_data method to fix issues loading data.

BUG: remove time sort in concat_data

acc3395

Remove the time sorting in `concat_data`, as data should be concatenated in the desired order.

DOC: update changelog

77668d2

Updated the changelog with a summary of the changes in this branch.

aburrell linked an issue Jul 20, 2023 that may be closed by this pull request

BUG: GUVI sdr-imaging data #197

Closed

aburrell added the bug Something isn't working label Jul 20, 2023

aburrell mentioned this pull request Jul 20, 2023

ENH: allow custom concat_data methods pysat/pysat#1131

Merged

10 tasks

aburrell added 2 commits July 25, 2023 12:12

STY: updated spacing

c2f7233

Added a whitespace after the `if` keyword. Also replaced hardcoded key with the appropriate variable.

ENH: added metadata

819d8e1

Added metadata for the 'time_auroral' and 'nCross' variables.

aburrell requested a review from jklenzing July 25, 2023 17:31

Merge branch 'develop' into guvi_imaging_fix

57228d1

jklenzing and others added 3 commits November 30, 2023 10:36

Merge branch 'develop' into guvi_imaging_fix

8a91717

MAINT: update rc tests

499fe07

Merge pull request #219 from pysat/maint/rc_tests

4f0df46

MAINT: update RC tests

This was referenced Nov 30, 2023

Release Candidate 3.2.0 pysat/pysat#1162

Merged

BUG: test_load_multiple_days failures #218

Open

Merge pull request #220 from pysat/develop

c3638e7

Pull Develop changes into guvi branch

jklenzing marked this pull request as ready for review December 19, 2023 19:28

aburrell added 3 commits February 27, 2024 10:31

BUG: fix JHUAPL time

959481e

Fix time where microseconds can become too big from rounding.

STY: removed type equality

7c80f0d

Updated type equalities to use `isinstance`.

TST: added an ignore list for links

566244e

Added a `linkcheck_ignore` statement, as the GATS page checks to see if the link request is human or machine.

BUG: fixed inners assignment

d175a35

Ensured that only dimensions with data are renamed. Also updated test date to highlight a time with better data coverage across all instruments.

TST: update GUVI test dates

eaaef3b

Update TIMED GUVI test dates, as coverage needs are different for different instruments.

jklenzing and others added 2 commits March 5, 2024 13:30

Merge branch 'develop' into guvi_imaging_fix

236cb98

MAINT: remove new test flag from timed_guvi

33ed75f

jklenzing reviewed Mar 6, 2024

View reviewed changes

pysatNASA/instruments/timed_guvi.py Show resolved Hide resolved

jklenzing added 2 commits March 6, 2024 18:26

Update pysatNASA/instruments/timed_guvi.py

a6a61fb

MAINT: only skip high res data

483c8e1

jklenzing approved these changes Mar 7, 2024

View reviewed changes

aburrell merged commit 2d33371 into develop Mar 7, 2024
25 of 27 checks passed

aburrell deleted the guvi_imaging_fix branch March 7, 2024 18:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GUVI imaging updates #198

GUVI imaging updates #198

aburrell commented Jul 20, 2023 •

edited

Loading

jklenzing commented Oct 6, 2023

aburrell commented Oct 6, 2023

jklenzing commented Nov 30, 2023 •

edited

Loading

jklenzing commented Dec 19, 2023

aburrell commented Mar 4, 2024

jklenzing commented Mar 5, 2024 •

edited by aburrell

Loading

jklenzing commented Mar 6, 2024

jklenzing left a comment •

edited

Loading

jklenzing commented Mar 7, 2024

jklenzing commented Mar 7, 2024

GUVI imaging updates #198

GUVI imaging updates #198

Conversation

aburrell commented Jul 20, 2023 • edited Loading

Description

Type of change

How Has This Been Tested?

Checklist:

jklenzing commented Oct 6, 2023

aburrell commented Oct 6, 2023

jklenzing commented Nov 30, 2023 • edited Loading

jklenzing commented Dec 19, 2023

aburrell commented Mar 4, 2024

jklenzing commented Mar 5, 2024 • edited by aburrell Loading

jklenzing commented Mar 6, 2024

jklenzing left a comment • edited Loading

Choose a reason for hiding this comment

jklenzing commented Mar 7, 2024

jklenzing commented Mar 7, 2024

aburrell commented Jul 20, 2023 •

edited

Loading

jklenzing commented Nov 30, 2023 •

edited

Loading

jklenzing commented Mar 5, 2024 •

edited by aburrell

Loading

jklenzing left a comment •

edited

Loading