IPF replacement format

[Manangka]

In GitLab by @Huite on Mar 8, 2022, 13:45

Ideally we could replace the IPF format. Reading is quite slow and there are few guarantees with regards to consistencies (delimiter, column names, etc.).

NetCDF is one option, but is difficult to implement with regards to QGIS. QGIS integration is really a key point, as we'd like to use this format to easily visualize. This rather limits options: if QGIS interprets it as raster data, it's fundamentally unsuitable. This basically excludes netCDF, since within QGIS we'd rely on GDAL to read netCDF.

I think a reasonable candidate might be two Tab-Separated-Values files (TSV):

• One with the static data (location, surface, whatever)
• One with ALL timeseries: this guarantees consistency atleast for the column names

Together in a single ZIP file. The logic for this is extremely easy to write, requires zipfile in Python standard library and pandas read csv.

This is also easily workable from QGIS (zipfile, read delimited file).

This might also work from a geopackage (but always requires something like geopandas to write from Python!).
But the problem is that within QGIS, you cannot use the relatively efficient pandas read functions; one has to use the slow Python QGIS API to fetch row by row.

[JoerivanEngelen]

Relevant comment from @erikgeo:

Deltares/imod-qgis#27 (comment)

I was coincidentally discussing this with Joeri yesterday.

Perhaps the parquet or feather file format could be used as a more efficient way to store ipf data? Both are column-based formats, as opposed to csv, which is row-based. I don't know how practical it is to use these formats over traditional ipf's or csv's to store, read and write borehole/well/cpt data, but there are some potential benefits:

• Fast reading and writing, especially feather
• Much smaller file sizes compared to csv (and to IPF, I suppose), especially parquet
• Both are integrated in Pandas (pandas.read_parquet/feather, pandas.write_parquet/feather functions available)
• You can read specific columns to further speed up reading time (note pandas.read_csv with the usecols argument isn't actually faster as it still has to read every column due to the way the data is stored in memory (row-by-row in sequence), whereas parquet and feather have their columns in sequence in memory)

I can imagine there are some drawbacks when trying to use these formats, some of which I'm probably not aware of.

https://databricks.com/glossary/what-is-parquet https://arrow.apache.org/docs/python/feather.html

[Huite]

Ideally, it works with QGIS too. Arrow (which is feather) is supported by GDAL and can be read. GeoArrow conventions prescribe geometry (basically "Well Known Binary" (WKB)).

If I recall correctly, there is also dict encoding for Arrow so you wouldn't have to duplicate geometries, but I'm somewhat sure this isn't supported. QGIS has been getting support for a number of joins however, see also Marnix his research for Ribasim. This is directly supported from geopackages -- I think. Might not be possible through geopandas though.

In that case, an acceptable solution to me would be to have a button just like our current IPF and IDF buttons which loads the geometries and automatically does a join with the associated data.