-
Notifications
You must be signed in to change notification settings - Fork 7
MMS system requirements
MM – multi-sensor match-up (coincidence).
MD – match-up dataset (e.g. NetCDF file) comprising multiple records each being a single-sensor match-up. MDs are versioned.
MMD – multi-sensor match-up dataset comprising multiple records each being a multi-sensor match-up.
MMS – multi-sensor match-up system. Includes database, interfaces and tools.
Single-sensor match-up – record of a single reference (usually in situ, sometimes with an invalid measurement value) with a single satellite with matching geolocation and time.
Multi-sensor match-up – record of single in situ, multiple satellite and auxiliary data with matching geolocation and time; primary reference is the satellite data; then add other data which must be subsetted by the MMD system
- SEVIRI (already subsetted) (NetCDF format),
- ATSR L1b (ENVISAT format),
- AVHRR GAC (NOAA format),
- PMW (NetCDF format),
- NWP (ECMWF ERA-interim, GRIB format, single values only),
- aerosol (TOMS (text format), SAGE II (binary, IDL reader available), SDI, ASDI, single values only, action on Gary: clarify if area or single value),
- sea-ice concentration (NetCDF or HDF) action on Gary: clarify if area or single value).
- history of in-situ data (NetCDF format)
Forward model fields and other fields will be added later during the project.
Match-up records shall consist of either
1. in-situ, ATSR, AVHRR GAC (nearest, long-term record 1991-2002 before other satellites where available)
2. in-situ, ATSR, other satellites (2003 – now)
3. in-situ, METOP, other satellites (2007 – 2010).
Thus, the MMD will cover dates as follows:
- ATSR/AVHRR: 1991 – 2010
- ATSR/AVHRR/Metop/SEVIRI/AMSRE/TMI: 2003 – 2010
Single-sensor match-up criteria – Criteria used to generate single-sensor match-ups.
The criteria are
- time threshold +/-2h from in-situ
- in-situ lat,lon has to be within pixel boundaries
- “Gross cloud test” passed
Multi-sensor match-up criteria – Criteria used to generate multi-sensor match-ups.
The criteria are
- must include ATSR or METOP (this is the satellite reference)
- must include at least two single sensor match-ups
- time threshold +/-12h from satellite reference
- spatial citeria is lat,lon of satellite reference pixel has to be within other satellite’s sub-scene image boundaries
Dummy in-situ – Used in hi-lat cases where no real in-situ data exists. The SST value in this case is a fill value.
Reference data – subset of in situ data (marked by flag in ATSR MD)
The following datasets are the resulting extracts of predefined MMS database queries.
DBT2 – an extract from the MMS database. Database for task 2 (project deliverable). Includes all coincidences except ‘reference data’ and all records except forward model fields.
RRDP – DBT2 plus supporting documentation on content and protocol for participation. It includes specification for MMS database contributions.
Algorithm Testing MMD – an extract from the MMS database. Includes all MMs except reference data. Contains test and training records.
Reference MMD – an extract from the MMS database. Includes only MMs for reference data.
MMD data provider – internal to project. Provides baseline MD (ATSR, METOP, SEVIRI) files for populating MMD; also in situ (drifting and moored buoys only) ATSR (Level 1b), AVHRR GAC (Level 1A), PMW (Level 2 AMSR-E and TMI), aerosol (TOMS AAI, SAGE II extiction profiles) and NWP (ECMWF – lots of fields!).
Algorithm selection – internal to project. Provides baseline algorithms for SST retrieval. Initial evaluation done using MMD output; may also use full Level 1A/1B data as well.
RRDP Users – provides back MMD files from their analysis or alternate data (e.g. from MODIS); can be seen by all (needed for ‘final’ algorithm selection); propose to provide format verification tool
CDRP and validation users – internal to project; can query all data in MMD
- M1:, KO+6, end of Jan 2011. Have a first version of MMS database in place covering 2007 – Sep.2010, extraction possible for the training dataset. Full ingestion.
- M2:, end of April 2011. DBT2 extraction possible and done, full data coverage required, may have to wait for (partial) GAC data
- M2’:, re-issue on availability of the full GAC data (optional)
- M3:, KO+13, Sept. 2011. ingestion of additional fields, update of existing fields by RR input, full functionality
- M1: MMS shall support the ingestion of new MD files, in-situ, satellite data files, …
- Opt: MMS (finally) shall ingest in-situ data and satellite data (to generate the MD files, which becomes an internal function) (desirable)
- M3: MMS shall validate input files and check for duplication of existing data
- M3: MMS shall support updates of existing fields (by adding columns with the updates, not by updates within files),
- M1: MMS shall support the addition of new fields (e.g. from RRDP users, from ARC reprocessing) in the MMD Exchange Format. Priorities: MODIS, GOES, MTSAT.
- M3: MMS shall support the ingestion of flags and explanation for invalid MMs, e.g. because in-situ is considered invalid.
- M2: MMS shall support versioning of all data being ingested.
- M1: MMS shall compute coincidences from MD inputs. In addition to the coincidence criteria where there is always an ATSR observation we agreed that triplets of Metop+SEVIRI+PMW should also be collected.
-
MMS shall assign additional satellite and auxiliary data:
- M1: MMS shall interpolate NWP values in space and time, and fill it into the fields of MMD extracts. A possible realisation for this is using the MPI CDO operators.
- M3: MMS shall fill in aerosol values as well. This may be implemented by simple nearest-neighbour resampling. This applies to TOMS and SAGE II.
- M1: MMS shall fill in aerosol values from SEVIRI (SDI (sahara dust index) is provided my in SEVIRI MDs already).
- M1: MMS shall fill in in-situ for 24h (see definition, +/-12 h from satellite observation time).
- M1: MMS shall calculate ASDI (ATSR sahara dust index) from ATSR L1b subsence.
- M1: MMS shall directly call RTTOV-10. The RTROV-10 output shall be automatically ingested back into the MMS database.
- M3: MMS shall assign a unique DOI for each of the datasets extracted from the MMS database.
- M3: MMS shall support the extraction of data using selection criteria (on in-situ data and other fields) and projection to fields
- M1: MMS shall provide MMD files – extracts as NetCDF files using a specified format.
- M2: MMS shall provide a command line interface (+ text files as parameters) for ingestion and extraction, and inspection (developed in several versions)
- Opt: MMS shall provide a GUI for upload for ingestion, ingestion authorisation, extraction and inspection (desirable)
- Opt: MMS shall provide a format validation tool for RRDP inputs to be used externally (desirable)
- Opt: MMS shall provide a visualisation tool that informs about the current DB contents in terms of sensors, spatial and temporal coverage, number of coincidences (desirable)
- M3: MMS shall have clearly defined access policies for its Users (see above)
-
M2: MMS shall provide a complete tracability of all data that is ingested into it. Traceability means:
- Trace back the original data (any file)
- When, who, what has been ingested?