A_Generalized_Read_API

A Generalized Read API

Over the years, the number of variations in the way Silo writers can construct their files and in particular their multiblock objects has grown substantially.

This was not seen as too much of a problem because VisIt was the tool primarily responsible for handling all these variations and VisIt developers have been comfortable handling the majority of complexity in reading Silo files in the VisIt plugin. In addition, VisIt developers are conversant enough with the Silo interface that the level of effort required to handle these variations was minimal.

As Silo is used more and more as a data exchange format for inter-code data exchange, these issues have become problematic for application developers. If a data producer changes the way it writes Silo files, then each data consumer needs to be adjusted to handle the new Silo objects. In addition, if they wish to be able to handle the older way of doing business, these applications are then forced to maintain more and more Silo reader code to handle all the variations. Worse, this effort is replicated throughout application code teams whose codes exchange data via Silo.

Recent discussions with ASQ working groups have revealed a need to develop a higher level read interface that isolates any given reader from having to worry about most or all of the details of how a writer chose to write its Silo files.

• How are meshes decomposed into files?
• How is the mapping of meshes to files encoded?
  • Explicit list of names
  • Some EMPTY blocks
  • Implicit namescheme

These wiki pages are devoted to a discussion of that design.

Generalizing multi-block object reads

void* DBGetMeshblock(
    DBfile *file,     /* [IN] The Silo database file handle */
    DBmultimesh *mm,  /* [IN] The multimesh object in the file */
    int block_idx,    /* [IN] The index (0...nblocks-1) of the desired mesh block */
    int *mesh_type)   /* [OUT] Returned mesh type */

void* DBGetVarblock(
    DBfile *file,     /* [IN] The Silo database file handle */
    DBmultivar *mv,   /* [IN] The multivar object in the file */
    int block_idx,    /* [IN] The index (0...nblocks-1) of the desired var block */
    int *var_type)    /* [OUT] Returned var type */

DBmaterial* DBGetMatblock(
    DBfile *file,     /* [IN] The Silo database file handle */
    DBmultimat *mm,   /* [IN] The multimat object in the file */
    int block_idx)    /* [IN] The index (0...nblocks-1) of the desired mat block */

DBmatspecies* DBGetMatspecblock(
    DBfile *file,         /* [IN] The Silo database file handle */
    DBmultimatspace *mms, /* [IN] The multimatspec object in the file */
    int block_idx)        /* [IN] The index (0...nblocks-1) of the desired matspec block */

For mesh and var calls, the return value is a void* and the caller is responsible for inspecting the returned type and casting to DBucdmesh, DBquadmesh, DBpointmesh or var equivalents. Alternatively, we could implement type-specific DBGetUcdmeshBlock(), DBGetQuadmeshBlock() and DBGetPointmeshBlock() and var equivalents.

It probably makes sense for the implementation to have a generic interface but for the high-level API to be type specific. That results in tiny wrappers to faclitate compile-time type checking that the Silo caller calls but that call the generic implementation underneath.

A challenge is ensuring we can keep to a minimum the amount of file open and file close activity relative to the way applications can currently do it if they do the work themselves. Instead, we may want to construct some intermediate object, a multi-mesh read helper object, in which we can also manage information about currently open silo files.

Here are some of the issues a caller may be interested in optimizing…

• getting multiple blocks in one call
• ordering calls to get blocks to ensure all from the same file are obtained before moving onto a next file
• inquiring blocks presence in files to help order them
• given a string in a multi-block object, return filename and path within file
• given a multi-block object and a list of integers indicating blocks of interest, return name(s) of files
• given this mb object in this file and given this mapping of blocks to processors, load the blocks for this processor
  • file and mb objects identified by char* names or actual handles.
    

    
    
    DBGetBlocks(
    
        char const mb_filename, / [IN] name of file containing the multi-block object /
    
        char const *mb_objname, / [IN] name of multi-block object within the file /
    
        const int *map, / [IN] array of length equal to the number of blocks in the mb object indicating mpi-ranks each block is assigned to /
    
        int myId, / [IN] the mpi-rank of the caller /
    
        void *blocks,/* [OUT] array of block object pointers returned /
    
        int *db_types, /[OUT] array of block object types */
    
        int *nblocks) / [OUT] number of returned objects */
    
    

  • finds occurances of myId in map, if none then done, otherwise, nmatches
  • allocaes pointers for nmatches objects and allocates nmatches db_types
  • opens the mb_filename’d file
  • reads the mb_objname multi-block object
  • finds the nmatch matches and sorts them
    • sort by filename, later upon opening a file, sort by offset within file
  • iterate over filenames
    • open file
    • sort objects by offsets
    • read objects and assign pointers

int DBGetMultiUcdblock(
    char const * const *ucdblock_names,
    int nblocks,
    DBucdmesh **ucdmeshes,
    DBfile **dbfile_cache, int dbfile_cache_size)