Potential data structures for 3d data

[jgriesfeller]

In preparation for tomorrow's (16 Sept 2022) PointCloudData meeting I thought it's a good idea to note some of my basic ideas.

Data structure:

The most basic data structure is a 2d numpy array of np.float. One row is one time step, the columns hold the data.

Columns of data array: (some are optional):

• (middle) time (seconds since the epoch since the ndarray is of dtype np.float, + 2 bounds)
• metadata index (more on that later)
• longitude (+4 bounds)
• latitude (+4 bounds)
• altitude (or other height coordinate e.g. pressure, sigma, etc.)
• additional data needed (e.g. ground pressure fro hybrid sigma vertical coordinate)
• obs data (can be several columns)
• model data (same # of columns as obs data or separate ndarray)
• additional data needed (e.g. error, carried over reading flags etc)

time bounds are NaN for instantaneous data, altitude is either the station's altitude or the altitude of the measurement (for 3d measurements)

Metadata structure (one entry per station)

• metadata index (to point to the data)
• lat, lon, altitude of the station
• station name
• additional metadata
• possibly pandas.DataFrame with time series for several averaging time sizes for convinience

optimisations for speed
in order to speed up common usage patterns indexes can be used. Common selections are based on:

• time
• location
• height

additional data structures: Indexes

• time_index (simple dict with time as key an a list of all row indexes with a certain start time as values?)
• location index (single float that e.g codes the latitude with a certain precision in the integer part of a float and the longitude in the floating point part of a float avoiding two index searches that way)
• possibly altitude index

Things to remember
ndarray is immutable (np.append / np.resize copy the data to a new grown array), the so avoid growing and shrinking where possible. If needed add data in chunks.
Start with an array size that is likely big enough to hold all data

[lewisblake]

My thoughts:

Data Structure

2d pandas.DataFrame. (Mutable)

Columns of the array

Most basic form (lon, lat, alt, time, obs)

• lon + 2 bounds (min_lon, max_lon) which can be derived from meta data on swath size
• lat + 2 bounds (min_lat, max_lat) " "
• alt (or other height coord) + metadata on what it is. (meters from sea level, hPa, Geopotential height, etc.)
• time : seconds since epoch or Julian
• obs: whatever the data is

A ColocatedData would follow a similar basic form with an additional column for model data and columns for the associated metadata

Metadata

• type of value (active remote sensing instrument, model output, etc.)
• swath size / grid size
• type of altitude
• type of time
• quality flags
• metadata on quality flag interpretation (not consistent across satellite products, for example)
• station name if relevant
• instrument name if relevant

[jgriesfeller]

Personal concern: Pandas is very slow (as one can see in the current implementation)

[lewisblake]

Yes, and memory inefficient. But it is also very convenient, so in that sense it's more of a wishlist. I'd also like to talk about xarray.Dataset and iris.cube.Cube quickly if only for my own edification

[jgriesfeller]

Before we discuss, we should clarify what kinds of data we need to store:

1. single time, one location and height (e.g. Aeronet)
2. time period, one location and height (e.g. precipitation)
3. single time, one location, 10 to 1000 heights (Lidar pointing up, e.g. EARLINET)
4. time period, location and height change over time (e.g. ozone sonde)
5. any of the above with an area it's valid (Aeolus satellite data)