MMS process for ingestion, matchup, MMD generation

This page describes the initial process of ingestion, matchup computation and complete MMD generation. The process shall follow "option 2c" with parallelisation of all three parts on the Eddie cluster, but with a final centralised database.

hosts: thetis, Eddie frontend, Eddie nodes database: mmdb1 on thetis located in /home/v1mbottc/mms/db backups: /exports/nas/exports/cse/geos/scratch/gc/sst-cci/db-backups archive: /exports/nas/exports/cse/geos/scratch/gc/sst-cci/archive

The process is split into several phases:

1. MD ingestion + removal of duplicates
2. matchup computation
3. timeandlocation file extraction + referenceflag updates
4. small datasets ingestion and coincidences
5. staging (per year or month)
6. satellite data ingestion and coincidences (per month)
7. ARC1+2 (+ NWP?) + reingestion (per month)
8. NWP + ARC3 + regingestion (per month)
9. MMD generation (per month)