Home

The NOAO SDM data queue

This document describes the development context for SDM-Q, a data queuing component of the SDMO Ops data management ecosystem.

SDM-Q purpose

The purpose of SDM-Q is to queue data while awaiting another component to service it. These include:

• Waiting to be transferred over the network to another computer/site
• Waiting to be registered with the mass store
• Waiting to be ingested into the archive

[Say something about DECAM, suported and legacy instruments]

Terms

• data file: a file that needs to be curated in the NOAO SDM holdings; these are commonly FITS files (.fits) or header files (.hdr), which are plain text files with just the FITS header part of the main data file.
• summit computer: a computer at one of the telescope sites: Kitt Peak, Cerro Tololo, Cerro Panchon
• mass store: the disk-based data holdings, currently either at La Serena or Tucson
• transport: an operation that requires the data to go over the network
• storage: an operation that results in the data being written to disk, either directly or through a data managemen system such as iRODS
• ingest: the process of loading a partial portion of a file’s metadata to the archive through an NSA-provided service
• raw data: the data as obtained from the summit computers
• pipeline: the data processing system operated by the SDM Pipeline group that processes raw data which is then submitted for storage and ingest
• reduced aka processed data: data provided by pipeline

General requirements

• Robustness: SDM-Q shall not lose track of any data pushed onto it, even in the case of hardware or software faults that terminate the process.
• Useability: Required actions shall be performed via a single command-line request (or, in the case a GUI is added in the future, an equivalent simplicity).

SDM-Ops requirements

• git etc

Functional requirements

• Start
• Stop
• Push
• Configurable pop
• List
• Remove
• Stash
• Queue-jump

Edge case behaviour

• Duplicates