MMS archive directory structure

MMS ingests inputs of various types, among them MD files, satellite L1B or L2 data, in-situ data, and auxiliary data. These inputs are to be provided in directories accessible to MMS. This is a proposal for a structure for these directories.

• The only hard requirement imposed by MMS is that MMS can distinguish the type of the data and a temporal organisation by directories and regular expressions on file names.
  • Different types shall be in different directories.
  • Data ingested at different points in time, i.e. new data added later, shall be in different directories.

Rational for this is that MMS reads the data in chunks of e.g. 1 month. For this reason, the optional <month> directory level should be present in the directory tree structure.

As a generic guideline the data should be stored in a directory tree of the structure:

• <archive area>/eodata/<type>/<revision>/<year>(/<month>(/<day>))/<file>
• example: /fs12/mms/eodata/AMSR-E/v05/2010/06/152/20100601-AMSRE-REMSS-L2P-amsr_l2b_v05_r42957.dat-v01.nc
• example: /fs12/mms/eodata/METOP_MD/1.0/2010/06/mdb1_metop02_20100601.nc
• <archive area> may be different for different types of inputs.
• <type> may be composed of <mission>, <sensor> and <code> where reasonable, e.g. avhrr/l1b/noaa-16
• The purpose of <revision> is to allow parallel storage of updated/reprocessed data without overwriting
• For inputs of general use (e.g. ATSR satellite data) the structure may deviate from the proposal.

For the storage on Eddie, we assume that we use the file system on

• /exports/nas/exports/cse/geos/scratch/gc/sst-cci/ (= <archive area>)
• The ATSR and AVHRR satellite data is already provided in subdirectory trees below this.
  • example: aatsr/arc112/aatsr/aatsr/2002/08/12/ats_toa_1p/ATS_TOA_1P....N1.gz
  • example: avhrr/l1b/noaa-15/2010/NSS.GHRR.NK.D10121.S2059.E2253.B6221517.GC.gz
• ATSR is currently organised in two directory trees. The first, for data up to Jan 2010 is a copy of the ARC servers:
  • aatsr/arc1<N>2/aatsr/<sensor>/<year>/<month>/<day>/ats_toa_1p
    • Where <N> is 1,2,3,4 representing the arc server which held the copy. Files are split between arc servers using <N> = day_of_year mod 4 to spread load when processing on the ARC servers.
    • <sensor> is atsr1, atsr2, aatsr
    • Data in the arc structure also includes the 'aux' subdirectory containing GEO location tiepoints, interpolated NWP, and SKiN effect corrections used for the ARC project.
  • Data for Feb 2010 onwards is in a copy of the NEODC directory structure:
    • aatsr/ftpFromNEODC/neodc/aatsr_multimission/aatsr-v2.0/2010/<month>/<day>/ats_toa_1p
• AVHRR is organised in large directories, one per year with more than 5000 entries.
• MD files may go into
  • atsr_md/1.0/2010/atsr_md_201006.nc
  • metop_md/1.0/2010/06/mdb1_metop02_20100601.nc
  • seviri_md/1.0/2010/06/sstmdb1_meteosat09_20100601.nc
• PMW files may go into
  • tmi/v04/2010/06/20100601-TMI-REMSS-L2P-tmi_L2b_v04_071449.dat-v02.nc
  • amsr-e/v05/2010/06/20100601-AMSRE-REMSS-L2P-amsr_l2b_v05_r42957.dat-v01.nc
• Aerosol and sea ice concentration files may go into
  • aerosol-aai/1.0/2010/06/20100601.egr
  • sea-ice/1.0/2010/06/ice_conc_nh_201006011200.hdf
• In-situ history data is identified by WMO ID (and date range) and may go into
  • insitu/1.0/???/insitu_WMOID_14934_20100306_20100622.nc
• Some in-situ data files cover several years. For files added in the future, MMS requires that they are ingested before matchups for the respective month are computed. Therefore, ??? may be composed of <year>/<month> of the end date of the interval. To use the first two digits of the call sign for ??? requires MMS to use the file system creation date to detect new files.
• ECMWF data shall also use a structure of <type>/<revision>/<year>/<month>(/<day>).
• MMS will add directory structures for extracted subscenes and for processing results of subscenes
  • atsr_sub/1.0/2010/06/01/...
  • amsr-e_sub/v05/2010/06/01/...
  • ...