ZFP_Compressor
See H5Z-ZFP, an HDF5 compression plugin. This plugin addresses many of the issues discussed in the notes below.
Notes from a short teleconference with Quincey Koziol and Peter Lindstrom on implementation of zfp within HDF5 and Silo.
Peter has developed a new floating point compression algorithm named zfp.
- It can hit a target bit-rate which means compressed size is predictable
- This suggests it could be used in a parallel HDF5 setting
- However, there are other implementation details having to do with how HDF5 stores chunk metadata that still preclude the application of ZFP in HDF5 parallel writes.
- It compresses in tiny blocks of 4^d where d is the dimension of the floating point array being compressed
- The whole array doesn’t need to be UNcompressed in order to read just some portion of it
- This facilitates random access to portions of the array
- It can hit a fixed precision or fixed accuracy target
- In some sense, precision represents a relative error tolerance while accuracy represents an absolute error tolerance
- The current algorithm works for single and double precision floating point data but a newer algorithm is under development that will support 32- and 64 bit integer data
- In the new algorithm it is anticiapted that 8-bit integer data can be handled reasonably well by padding to 32 bits and then using a fixed-precision mode
- Currently, zfp does not support lossLESS compression
- The new zfp implementation may support lossLESS
- Current studies show that minimum loss zfp outputs are numerically nearly equivalent to machine precision noise
- zfp can, optionally, store data such that it represents a progression in quality of uncompressed output and asymptotically approaches lossless
- Peter has also implemented a zfp array class for C++ that behaves like a normal array but is compressed in memory
- Note: would be good to talk to Kokkos, Raja, VTK-m folks about using this as an option internally
- Can ZFP handle 4D arrays (space X time for example)
- ZFP does not have intrinsic 4D support, but could trivially be extended. The main stumbling block is maintaining high performance. ZFP currently exploits the fact that in 3D a block contains 4^3 = 64 values, and can be compressed efficiently using 64-bit integer operations. To handle 4D data in the same way, 4^4 = 256-bit integers would be needed. The 4D implementation would likely have to be specialized, and would be slower than zfp’s current throughput of up to 1 GB/s.
- The presence of inf/nan in zfp will cause unspecified failures due to use of frexp()
- The behavior is unspecified, i.e. it depends on the compiler. gcc gives an exponent of 0 for both infinities and NaNs. These numbers are later converted to integers, which can invoke undefined behavior, i.e. anything can happen. In practice, there will be no seg faults or other “severe problems,” but you probably won’t get back what you put in. E.g. a 1D block { inf, 0, 0, 0 } comes back as { -2, 0, 0, 0}, while { sqrt(-1.0), 0, 0, 0 } comes back as { -FLT_MIN, 0, 0, 0 }.
- Its difficult to handle these corner cases, because that would greatly slow down the compressor, and furthermore there’s no clean way of dealing with them. zfp represents all numbers in a block relative to a common exponent, or roughly equivalently relative to the largest (in magnitude) value within the block, and then everything gets converted to integers. If inf is present, everything else would go to zero in such a representation. If a NaN is present, it isn’t clear what this means. Either such non-numbers would corrupt any valid numbers in the block, or the non-numbers themselves would get returned as some valid number, which is equally problematic.
- We could allow the user to enable handling of non-numbers via compile-time macros that are off by default. This will address the performance issues. But even though its possible detect such non-numbers in C99, it isn’t clear what the right thing to do is in the context of lossy compression. Furthermore, would we not assume that any science data that contains non-numbers is invalid in the first place? Any NaN will propagate in a PDE solver and destroy anything it touches. Infinities will similarly drive nearby values to zero or infinity.
Just to give an idea of zfp’s utility, below are a few images that show how zfp compares with other compressors. By comparison, gzip --best gives you 1.04x lossless compression on this data set, while fpzip (a lossless compressor also written by Peter Lindstrom) gives 1.11x lossless compression. Even at 89x zfp compression (i.e. 0.72 bits/value compared to the original 64 bits/value), you’ll be hard pressed to tell that the data has been compressed. Its concievable this is good enough for vis dumps or for expanding the effective memory to allow much larger data sets to be visualized in-core. For instance, a 4K^3 grid fits comfortably on my 8 GB laptop at this compression level. We’re working on adding such “decompression on demand” to VisIt. A similar capability could easily be supported through HDF5 via chunking.
We have developed an HDF5 filter plugin that is highly efficient in storage of metadata and integrates with HDF5 features and supports all of ZFP’s mode.
- General notes on compression in HDF5 are here
- You can git a stand-alone, HDF5 filter plugin on github
- You can get documentation on the HDF5 filter plugin here
- Peter has also implemented a C++ array class using zfp to store the arrays compressed in memory
- This might have application in tools like Kokkos, VTK-m and maybe RAJA
- zfp is being used in MACSio already
- zfp is being integrated with Silo (to replace hzip and fpzip)
- Since zfp is primarily lossy compression, it represents a very different animal from other compressors the HDF5 community is accustomed to and, in particular, The HDF Group has hosted here
- If we do get to the point of hosting zfp as a registered HDF5 filter, we need to be clear about this filter being lossy