-
Notifications
You must be signed in to change notification settings - Fork 6
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'develop' into tst/34_omni
- Loading branch information
Showing
10 changed files
with
135 additions
and
75 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,69 @@ | ||
Building data files for archival at NASA SPDF | ||
============================================= | ||
|
||
The codes and routines at :py:mod:`pysatNASA` are designed for end-users of NASA data | ||
products. However, pysat in general has also been used to build operational | ||
instruments for generating archival data to be uploaded to the Space Physics | ||
Data Facility (SPDF) at NASA. | ||
|
||
In general, such instruments should include separate naming conventions. An | ||
example of this is the REACH data, where netCDF4 files are generated for | ||
archival purposes as part of the :py:mod:`ops_reach` package, but can be accessed by | ||
the end user through :py:mod:`pysatNASA`. | ||
|
||
In general, a :py:class:`pysat.Instrument` object can be constructed for any | ||
dataset. Full instructions and conventions can be found | ||
`here <https://pysat.readthedocs.io/en/latest/new_instrument.html>`_. In the | ||
case of the REACH data, the operational code reads in a series of csv files and | ||
updates the metadata according to user specifications. Once the file is loaded, | ||
it can be exported to a netCDF4 file via pysat. In the simplest case, this is | ||
|
||
:: | ||
|
||
reach = pysat.Instrument(inst_module=aero_reach, tag='l1b', inst_id=inst_id) | ||
pysat.utils.io.inst_to_netcdf(reach, 'output_file.nc', epoch_name='Epoch') | ||
|
||
|
||
However, there are additional options when translating pysat metadata to SPDF | ||
preferred formats. An example of this is | ||
|
||
:: | ||
|
||
# Use meta translation table to include SPDF preferred format. | ||
# Note that multiple names are output for compliance with pysat. | ||
# Using the most generalized form for labels for future compatibility. | ||
meta_dict = {reach.meta.labels.min_val: ['VALIDMIN'], | ||
reach.meta.labels.max_val: ['VALIDMAX'], | ||
reach.meta.labels.units: ['UNITS'], | ||
reach.meta.labels.name: ['CATDESC', 'LABLAXIS', 'FIELDNAM'], | ||
reach.meta.labels.notes: ['VAR_NOTES'], | ||
reach.meta.labels.fill_val: ['_FillValue'], | ||
'Depend_0': ['DEPEND_0'], | ||
'Format': ['FORMAT'], | ||
'Monoton': ['MONOTON'], | ||
'Var_Type': ['VAR_TYPE']} | ||
|
||
pysat.utils.io.inst_to_netcdf(reach, 'output_file.nc', epoch_name='Epoch', | ||
meta_translation=meta_dict, | ||
export_pysat_info=False) | ||
|
||
|
||
In this case, note that the pysat 'name' label is output to three different | ||
metadata values required by the ITSP standards. Additionally, the | ||
:py:attr:`export_pysat_info` option is set to false here. This drops several | ||
internal :py:mod:`pysat` metadata values before writing to file. | ||
|
||
A full guide to SPDF metadata standards can be found | ||
`here <https://spdf.gsfc.nasa.gov/istp_guide/istp_guide.html>`_. | ||
|
||
Other best practices for archival include adding the operational software version | ||
to the metadata header before writing. The pysat version will be automatically | ||
written to the metadata. | ||
|
||
:: | ||
|
||
reach.meta.header.Software_version = ops_reach.__version__ | ||
|
||
|
||
A full example script to generate output files can be found at | ||
https://github.com/jklenzing/ops_reach/blob/main/scripts/netcdf_gen.py |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -28,10 +28,10 @@ | |
import functools | ||
|
||
from pysat.instruments.methods import general as mm_gen | ||
from pysat import logger | ||
|
||
from pysatNASA.instruments.methods import cdaweb as cdw | ||
from pysatNASA.instruments.methods import general as mm_nasa | ||
from pysatNASA.instruments.methods import iss as mm_iss | ||
|
||
# ---------------------------------------------------------------------------- | ||
# Instrument attributes | ||
|
@@ -49,34 +49,7 @@ | |
# ---------------------------------------------------------------------------- | ||
# Instrument methods | ||
|
||
|
||
def init(self): | ||
"""Initialize the Instrument object with instrument specific values. | ||
Runs once upon instantiation. | ||
""" | ||
|
||
ackn_str = ' '.join(('Data provided through NASA CDAWeb. Contact', | ||
'[email protected] for support and use.')) | ||
logger.info(ackn_str) | ||
self.acknowledgements = ackn_str | ||
self.references = ' '.join(('V. N. Coffey et al., "Validation of the', | ||
'Plasma Densities and Temperatures From', | ||
'the ISS Floating Potential Measurement', | ||
'Unit," in IEEE Transactions on Plasma', | ||
'Science, vol. 36, no. 5, pp. 2301-2308,', | ||
'Oct. 2008,', | ||
'doi: 10.1109/TPS.2008.2004271.\n', | ||
'A. Barjatya, C.M. Swenson, D.C.', | ||
'Thompson, and K.H. Wright Jr., Data', | ||
'analysis of the Floating Potential', | ||
'Measurement Unit aboard the', | ||
'International Space Station, Rev. Sci.', | ||
'Instrum. 80, 041301 (2009),', | ||
'https://doi.org/10.1063/1.3116085')) | ||
|
||
return | ||
init = functools.partial(mm_nasa.init, module=mm_iss, name=name) | ||
|
||
|
||
# Use default clean | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
# -*- coding: utf-8 -*- | ||
"""Provides non-instrument specific routines for C/NOFS data.""" | ||
|
||
ackn_str = ' '.join(('Data provided through NASA CDAWeb Key Parameters -', | ||
'Shin-Yi Su (Institute of Space Science, National Central', | ||
'University, Taiwan, R.O.C.)')) | ||
refs = {'ivm': ' '.join(('Yeh, H.C., S.‐Y. Su, Y.C. Yeh, J.M. Wu, R. A.', | ||
'Heelis, and B. J. Holt, Scientific mission of the', | ||
'IPEI payload on board ROCSAT‐1, Terr. Atmos. Ocean.', | ||
'Sci., 9, suppl., 1999a.\n', | ||
'Yeh, H.C., S.‐Y. Su, R.A. Heelis, and J.M. Wu, The', | ||
'ROCSAT‐1 IPEI preliminary results, Vertical ion', | ||
'drift statistics, Terr. Atmos. Ocean. Sci., 10, 805,', | ||
'1999b.'))} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
# -*- coding: utf-8 -*- | ||
"""Provides non-instrument specific routines for C/NOFS data.""" | ||
|
||
ackn_str = ' '.join(("R.M. Suggs, S.L. Koontz, NASA Johnson Space Center", | ||
"Contact Rob Suggs for support and use.", | ||
"[email protected]. Please acknowledge the data", | ||
"providers and CDAWeb when using these data.")) | ||
|
||
refs = {'fpmu': ' '.join(('V. N. Coffey et al., "Validation of the Plasma', | ||
'Densities and Temperatures From the ISS Floating', | ||
'Potential Measurement Unit," in IEEE Transactions', | ||
'on Plasma Science, vol. 36, no. 5, pp. 2301-2308,', | ||
'Oct. 2008, doi: 10.1109/TPS.2008.2004271.', | ||
'\n', | ||
'A. Barjatya, C.M. Swenson, D.C. Thompson, and K.H.', | ||
'Wright Jr., Data analysis of the Floating Potential', | ||
'Measurement Unit aboard the International Space', | ||
'Station, Rev. Sci. Instrum. 80, 041301 (2009),', | ||
'https://doi.org/10.1063/1.3116085', | ||
'\n', | ||
'Debchoudhury, S., Barjatya, A., Minow, J. I.,', | ||
'Coffey, V. N., & Chandler, M. O. (2021).', | ||
'Observations and validation of plasma density,', | ||
'temperature, and O+ abundance from a Langmuir', | ||
'probe onboard the International Space Station.', | ||
'Journal of Geophysical Research: Space', | ||
'Physics, 126, e2021JA029393.', | ||
'https://doi.org/10.1029/2021JA029393')) | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters