Skip to content

Latest commit

 

History

History
688 lines (491 loc) · 17.4 KB

reference.md

File metadata and controls

688 lines (491 loc) · 17.4 KB

Function reference

The kdb+/HDF5 interface is a wrapper for kdb+ around the HDF Groups C API.

.hdf5 HDF5 interface functionality

object use

Create
createFile Create a named HDF5 file
createGroup Create a single or multiple group levels

Write
writeAttr Write a named attribute associated to a group or dataset
writeData Write data to a named dataset

Read
readAttr Read data from named attribute
readData Read data from a named dataset

Link
createExternal Create a hard external link between locations in two files
createHard Create a hard link between two locations in a file
createSoft Create a soft link between two locations in a file
delLink Delete an existing soft or hard link

Utility
copyObject Copy a group or dataset to another file
dataSize Size of a HDF5 dataset in MB
errorOn Turn the HDF5 C API error signalling on
errorOff Turn the HDF5 C API error signalling off (default)
fileSize Size of a HDF5 in MB
getAttrShape Dimensionality of an attribute
getDataShape Dimensionality of a dataset
gc Run garbage collect on free HDF5 lists of all types
isAttr Whether this attribute exists
ishdf5 Whether a file of HDF5 format
isObject Whether object exists
isThreadsafe Whether HDF5 lib is built with multithread support
ls Representation of the structure of a HDF5 file
version Version of the HDF5 C API being used

For simplicity in each of the examples below it should be assumed that unless otherwise specified a file/group/dataset/attribute that is being manipulated exists and is valid and the results displayed coincide with correct execution in that example.

copyObject

Copy a HDF5 dataset or group from one file to another

.hdf5.copyObject[fname;oname;fdest;odest]

Where

  • fname is the name of the file containing the object to be copied as a string
  • oname is the name of the object to be copied as a string
  • fdest is the name of the destination file for the copied object as a string
  • odest it the name of the path in the destination file to which the copied object is associated as a string

return null on successful execution, signals error if there are issues accessing files.

q).hdf5.writeData["test.h5";"dset";100?5]
q).hdf5.createFile["testing.h5"]
// Copy original object to a new file
q).hdf5.copyObject["test.h5";"dset";"testing.h5";"data"]
q).hdf5.readData["testing.h5";"data"]
4 0 2 1 2 1 2 3 2 4 1 0 2 4 1 2 0 1 1 2 1 0 0 1 ..

createExternal

Creates an external link to an object in a different file

.hdf5.createExternal[lfile;lpath;tfile;tpath]

Where

  • lfile is the name as a string of the HDF5 file containing the end point for the new external link
  • lpath is the name as a string of the object acting as the end point for the link
  • tfile is the name as a string of the target HDF5 file containing the target object
  • tpath is the name as a string of the target object which is linked to lpath

returns null on successful execution and creates the external link.

// Produce a new .h5 file and link a path within this to a dataset in another h5 file
// using the create external link functionality within the api
q).hdf5.createFile "test_external.h5"
q).hdf5.createExternal["test_external.h5";"/dset";"test.h5";"/dset"]
// Access the data from this new link
q).hdf5.readData["test_external.h5";"/dset"]
0.4707883 0.6346716
0.9672398 0.2306385
0.949975  0.439081
0.5759051 0.5919004
0.8481567 0.389056

createFile

Create a named HDF5 file

.hdf5.createFile fname

Where fname is the name of a HDF5 file as a string, returns null on successful creation of file. Attempting to create a file that already exists will result in an error

// Create named file
q).hdf5.createFile "test.h5"

// Attempt to create this file again
q).hdf5.createFile "test.h5"
'error creating file

createGroup

Create a group or set of groups within a file

.hdf5.createGroup[fname;gname]

Where

  • fname is the name of a HDF5 file as a string
  • gname is the structure of the groups to be created as a string

returns null on successful execution and creates the desired groups.

// Create a single group
q).hdf5.createGroup["test.h5";"SingleGroup"]
// Create a set of groups
q).hdf5.createGroup["test.h5";"Group1/SubGroup1/SubGroup2"]

createHard

Create a hard link to a physical address in a file

.hdf5.createHard[lfile;lpath;tpath]

Where

  • lfile is the name of a HDF5 file as a string
  • lpath is the name of the object acting as the end point for the link as a string
  • tpath is the name of the target path for the new hard link as a string

returns null on successful execution and creates the hard link.

// Create a Group 'G1' to house the new hard link to the dataset
q).hdf5.createGroup["test.h5";"G1"]
// Create a hard link from existing 'dset' to 'G1/dset'
q).hdf5.createHard["test.h5";"G1/dset";"dset"]
// Access the data using the new link
q)5#.hdf5.readData["test.h5";"/G1/dset"]
0.4707883 0.6346716
0.9672398 0.2306385
0.949975  0.439081
0.5759051 0.5919004
0.8481567 0.389056

createSoft

Create a soft/symbolic link between two locations in a HDF5 file

.hdf5.createSoft[lfile;lpath;tpath]

Where

  • lfile is the name as a string of a HDF5 file
  • lpath is the name as a string of the object acting as the end point for the link
  • tpath is the name as a string of of the target path for the new soft link

returns null on successful execution and creates the specified soft link.

Note File paths

Neither lpath nor tpath need exist before creating the soft link. They can be created afterwards.

But if the links are to be contained within a group, the group must exist.

// Create a group 'G1' within the file 'test.h5'
q).hdf5.createGroup["test.h5";"G1"]

// Create a soft link (symbolic link) between two unpopulated points in the file
q).hdf5.createSoft["test.h5";"/G1/dset1";"/G1/dset2"]

// Write data to 'G1/dset2'
.hdf5.writeData["test.h5";"G1/dset2";4 5#20?10]

// Read the data from the location 'G1/dset1'
q).hdf5.readData["test.h5";"/G1/dset1"]
8 1 9 5 4 6
6 1 8 5 4 9
2 7 0 1 9 2
1 8 8 1 7 2

dataSize

Size of an uncompressed HDF5 dataset in MB

.hdf5.dataSize[fname;dname]

Where

  • fname is the name of a HDF5 file as a string
  • dname is the name of the dataset as a string

returns the size of a HDF5 dataset in MB.

q).hdf5.dataSize["test.h5";"dset"]
8f

delLink

Delete an assigned external hard or soft link

.hdf5.delLink[lfile;lpath]

Where

  • lfile is the name of a HDF5 file as a string
  • lpath is the name of the link to be deleted as a string

returns a message indicating successful deletion of a link.

// Delete a soft link
q).hdf5.delLink["test.h5";"/G1/dset1"]
"Successfully deleted the link"
// Attempt to delete the link again
q).hdf5.delLink["test.h5";"/G1/dset1"]
'could not delete specified link

errorOff

Turn off printing of errors from HDF5 C API

.hdf5.errorOff[]

returns null on successful execution.

Note "The interface is initialized with HDF5 erroring turned off."

// Failing function called with HDF5 error ons
q).hdf5.createFile "test.h5"
HDF5-DIAG: Error detected in HDF5 (1.10.5) thread 0:
  #000: H5F.c line 444 in H5Fcreate(): unable to create file
    major: File accessibilty
    minor: Unable to open file
  #001: H5Fint.c line 1558 in H5F_open(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #002: H5FD.c line 734 in H5FD_open(): open failed
    major: Virtual File Layer
    minor: Unable to initialize object
  #003: H5FDsec2.c line 346 in H5FD_sec2_open(): unable to open file:
    major: File accessibilty
    minor: Unable to open file
'error creating file
  [0]  .hdf5.createFile "test.h5"

// Turn off errors and repeat the above function call
q).hdf5.errorOff[]
q).hdf5.createFile "test.h5"
'error creating file

errorOn

Turns on printing of errors from HDF5 C API

.hdf5.errorOn[]

returns null on successful execution.

Note The interface is initialized with HDF5 erroring turned off.

// Execute an invalid command with error printing off
q).hdf5.createFile "test.h5"
'error creating file

// Turn error printing on and repeat command
q).hdf5.errorOn[]
q).hdf5.createFile "test.h5"
HDF5-DIAG: Error detected in HDF5 (1.10.5) thread 0:
  #000: H5F.c line 444 in H5Fcreate(): unable to create file
    major: File accessibilty
    minor: Unable to open file
  #001: H5Fint.c line 1558 in H5F_open(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #002: H5FD.c line 734 in H5FD_open(): open failed
    major: Virtual File Layer
    minor: Unable to initialize object
  #003: H5FDsec2.c line 346 in H5FD_sec2_open(): unable to open file:
    major: File accessibilty
    minor: Unable to open file
'error creating file

fileSize

Size of a HDF5 file in MB

.hdf5.fileSize fname

Where fname is the name of a HDF5 file as a string returns the size of a HDF5 in MB.

q).hdf5.fileSize "test.h5"
8.002048

gc

Garbage collect on free HDF5 lists of all types

.hdf5.gc[]

returns 0i on successful execution.

q).hdf5.gc[]
0i

getAttrShape

Get the shape of an attribute dataset

.hdf5.getAttrShape[fname;oname;aname]

Where

  • fname is the name as a string of a HDF5 file
  • oname is the name as a string of an object (group/dataset) to which an attribute is associated
  • aname is the name as a string of an attribute

returns an numerical list indicating the attribute shape.

q).hdf5.getAttrShape["test.h5";"dset";"Temperatures"]
1 10

getDataShape

Get the shape of a dataset

.hdf5.getDataShape[fname;dname]

Where

  • fname is the name of a HDF5 file as a string
  • dname is the name of the dataset of interest as a string

returns a numerical list indicating the shape of the dataset.

q).hdf5.getDataShape["test.h5";"dset"]
2 4

isAttr

Does the named attribute exist

.hdf5.isAttr[fname;oname;aname]
  • fname is the name as a string of a HDF5 file
  • oname is the name as a string of an object (group/dataset) to which the attribute is associated
  • aname is the name as a string of the attribute

returns a boolean indicating whether the attribute exists.

q).hdf5.isAttr["test.h5";"dset";"Temperatures"]
1b
q).hdf5.isAttr["test.h5";"dset";"Temps"]
0b

ishdf5

Is the specified file a HDF5 file

.hdf5.ishdf5 fname

Where fname is the name of a HDF5 file as a string, returns a boolean indicating if the file is a HDF5 file or not.

q).hdf5.ishdf5 "test.h5"
1b
q).hdf5.ishdf5 "test.txt"
0b

isObject

Whether an object is a group/dataset

.hdf5.isObject[fname;oname]

Where

  • fname is the name of a HDF5 file as a string
  • oname is the name of an object (group/dataset) as a string

returns a boolean indicating whether the object exists.

// Dataset exists
q).hdf5.isObject["test.h5";"dset"]
1b
// Group exists
q).hdf5.isObject["test.h5";"G1"]
1b
// Object doesn't exist
q).hdf5.isObject["test.h5";"not_obj"]
0b

isThreadsafe

Whether HDF5 lib is build with multi thread support

.hdf5.isThreadsafe[]

Returns a boolean indicating whether the HDF5 library was build with multithread support.

ls

Display the structure of a HDF5 file

.hdf5.ls fname

Where fname is the name of a HDF5 file as a string, returns a dict of the overall structure of the HDF5 file.

q).hdf5.ls "test.h5"
type | group
name | `Group1
value| `type`name`value!(`group`group;(`SG1;`SG2);((`symbol$())!();`type`name`value!(,`group;,`SG3;,(`symbol$())!())))

readAttr

Read the data contained in a HDF5 attribute to kdb+

.hdf5.readAttr[fname;oname;aname]

Where

  • fname is the name of a HDF5 file as a string
  • oname is the name of the object (group/dataset) with which the attribute being read from is associated as a string
  • aname is the name of the attribute from which data is to be read as a string

returns a kdb+ dataset containing the data associated with the named HDF5 attribute

// Read numeric data from attribute
q).hdf5.readAttr["test.h5";"dset";"temperature"]
0.08123546 0.9367503 
0.2782122  0.2392341 
0.1508133  0.1567317 
0.9785     0.7043314 
0.9441671  0.7833686 

// Read string data from attribute 
q).hdf5.readAttr["test.h5";"dset";"Description"]
"This data relates to experiment 1"
"Time difference between values = 1ms"

readData

Read the data contained in a HDF5 dataset to kdb+

.hdf5.readAttr[fname;dname]

Where

  • fname is the name of a HDF5 file as a string
  • dname is the name of the dataset which is to be read from as a string

returns a kdb+ dataset containing the data associated with a HDF5 dataset

// Read numeric data
q)5#.hdf5.readData["test.h5";"dset"]
7.263142  9.216436 
1.809536  6.434637 
2.907093  0.7347808
3.159526  3.410485 
8.617972  5.548864 

// Read string data
q).hdf5.readData["test.h5";"strdset"]
"Value 1 = 0.01"
"Value 2 = 0.02"
"Value 3 = 0.04"

// Read table
q).hdf5.readData["test.h5";"table"]
x          x1         x2 x3           x4  
------------------------------------------
0.2400771  0.7263287  1  08:12:43.345 apah
0.1039355  0.4886361  1  04:42:08.234 mlfb
0.2353326  0.6535973  0  17:54:05.495 fmbf
0.6423479  0.02810674 0  16:58:19.059 iibh
0.5778177  0.4449053  1  12:47:14.478 fpam

// Read dictionary
q).hdf5.readData["test.h5";"dict"]
x1| 0 1 2 3 4 5 6 7 8 9
x2| `nbmb`gbdn`ijca`khjj`gncf
x3| 2003.01.16D01:51:45.083828416 2002.11.25D05:50:34.964843088 2000.09.19D10.

🌐 Definitions of external, hard and soft links as they pertain to the HDF5 data format

version

Display C API major/minor/release versions

.hdf5.version[]

Returns a dictionary with the major/minor and release versions of the HDF group’s C API.

q).hdf5.version[]
Major  | 1
Minor  | 10
Release| 5

writeAttr

Write a kdb+ dataset to a HDF5 attribute

.hdf5.writeAttr[fname;oname;aname;dset]

Where

  • fname is the name of a HDF5 file as a string
  • oname is the name of an object (group/dataset) to which an attribute is to be written to as a string
  • aname is the name of the attribute to be written to as a string
  • dset dataset which is being written to HDF5 format

returns null on successful writing of data to attribute. Failure to write to the attribute will signal an appropriate error.

⚠️ Watch out

To write an attribute to a dataset/group the file and dataset/group specified must already exist.

Attributes can only be written to with data types where the mapping q => HDF5 => q does not require a datatype_kdb attribute associated with the data.

// Writing attribute whe file does not exist
q).hdf5.writeAttr["test.h5";"dset";"temperature";10 2#20?1f]
'error opening file

// Writing attribute when dataset or group does not exist
q).hdf5.writeAttr["test.h5";"dset";"temperature";10 2#20?1f]
'error opening dataset/group

// Writing data with an inappropriate type
q).hdf5.writeAttr["test.h5";"dset";"times";10 2#20?0t]
'kdb+ type can not be mapped to an appropriate attribute

// Successful execution
q).hdf5.writeAttr["test.h5";"dset";"temperature";10 2#20?1f]

writeData

Write a kdb+ dataset to a HDF5 dataset

.hdf5.writeData[fname;dname;dset]

Where

  • fname is the name of a HDF5 file as a string
  • dname is the name as a string of the dataset which is to be written to
  • dset dataset which is being written to HDF5 format

returns null on successful writing of data to dataset. Failure to write to the dataset will signal an appropriate error.

!!! warning "To write a dataset to a file, the file must already exist."

// Writing dataset when file does not exist
q).hdf5.writeData["test.h5";"dset";10 2#20?10f]
'error opening file

// Write numeric dataset
q).hdf5.writeData["test.h5";"dset";10 2#20?10]

// Write string data
q).hdf5.writeData["test.h5";"strdset";enlist "test string"]

// Write tabular data
q).hdf5.writeData["test.h5";"table";([]100?1f;100?1f;100?0b;100?0t;100?`4)]

// Write dictionary data
q).hdf5.writeData["test.h5";"dict";`x1`x2`x3!(til 10;5?`4;100?0p)]