Recorder and Deployment Functionality

The recorder is an abstraction of the filesystem, specialized for recording Smartfin data.

Each deployment is an abstraction of a file, again specialized for recording Smartfin data.

Recorder

Semantically, a data recorder should be able to keep a structured set of data with proper labels and metadata. It should be aware of sessions, packets, and ensembles. Each session should have a name.

Below are the functional requirements for the recorder:

1. The Recorder MUST accept unstructured binary blobs to record sequentially. These blobs MAY NOT correspond to complete packets, and MAY NOT correspond to complete ensembles.
2. The Recorder MAY accept structured binary blobs to record sequentially, however, it MUST treat the blobs as unstructured.
3. The Recorder MUST NOT require the final filename on recording start, and MUST provide a method to set the filename at some point later during the session.
4. The Recorder MUST retain the sequence in which data was recorded across multiple sessions.
5. The Recorder MUST retrieve packets in the reverse sequence (LIFO). If a session does not consist of an integer number of packets, the last packet for that session MUST be a partial packet, and MUST NOT include data from the next session. If a packet is retrieved, then additional data is recorded, that additional data MUST be retrieved first.
6. The Recorder MUST not lose data once the function to commit data has returned.

Below is the functional API for the recorder:

Recorder::Recorder(void);
int Recorder::init(void);

int Recorder::has_data(void);
int Recorder::get_last_packet(void* p_buffer, size_t buffer_len, char* p_name, size_t name_len);
int Recorder::pop_last_packet(size_t len);
int Recorder::set_session_name(const char* const name);
int Recorder::get_num_sessions(void);

int Recorder::open_session(const char* const name);
int Recorder::close_session(void);
int Recorder::put_bytes(const void* const p_data, size_t n_bytes);
int Recorder::put_data(T& data);

In order for this to work, we need to retain a sequential record of sessions, irrespective of name. This is important since there is no guarantee that any session will have a name, since the names are based on GPS time (which in certain cases may never occur). In order to maintain power fail-safe, the naming metadata must be kept in a separate file (most filesystems do not permit renaming a file while it it open). Since the only dynamic metadata required to generate the filename is time, we can retain this as a 32-bit UTC timestamp for each file. If the GPS time is not known, we record it as 0 (1970-01-01T00:00-00:00).

The Recorder will not pad files to be multiples of packet length. Instead, partial packets will be partial.

Data Layout

Since there are other files and directories inside the the Tracker SoM's filesystem, all data for the Smartfin MUST be stored in /data/. The Recorder is responsible for ensuring that this directory exists on initialization.

We use Unix filesystem conventions - any file starting '.' will be considered hidden, and does not count towards committed data.

We will use the following special files for nonvolatile storage:

• .metadata - this file will store the 32-bit timestamp for each dataset

Metadata File

Offset	Description	Type
0x00	Next Data Index	U32
0x04	Table Size (N)	U32
0x08	Timestamp Table	U32[N][2]
0x08 + 8N

Naming

If the start time for the session is known, then the session name generated will be "Sfin-{device_id}-{timestamp}-{packet_num}". The timestamp will be "%y%m%d-%H%M%S", using the Python3 datetime.strftime format behavior, in the UTC timezone.

If the start time for the session is not known, then the timestamp will instead be the sequence number of that session as 08d.