Merge pull request #189 from philipwjones/omega/io-time-slice

Adds a time-slice capability to Omega IO and IOStream

components/omega/configs/Default.yml

@@ -94,15 +94,17 @@ Omega:
         - SshCellDefault
     Highfreq:
       UsePointerFile: false
-      Filename: ocn.hifreq.$Y-$M-$D_$h.$m.$s
+      Filename: ocn.hifreq.$Y-$M
       Mode: write
-      IfExists: replace
+      IfExists: append
       Precision: single
       Freq: 10
       FreqUnits: days
+      FileFreq: 1
+      FileFreqUnits: months
       UseStartEnd: true
       StartTime: 0001-06-01_00:00:00
-      EndTime: 0001-06-30_00:00:00
+      EndTime: 0001-07-31_00:00:00
       Contents:
         - Tracers
   ManufacturedSolution:

components/omega/doc/devGuide/Field.md

@@ -40,7 +40,8 @@ Fields are created with standard metadata using
                  FillValue,   ///< [in] scalar used for undefined entries
                  NumDims,     ///< [in] number of dimensions (int)
                  Dimensions,  ///< [in] dim names (vector of strings)
-                 InTimeDependent ///< [in] (opt, default true) if time varying
+                 TimeDependent   ///< [in] (opt, default true) if time varying
+                 RetainPrecision ///< [in] (opt, false) retain full prec in IO
    );
 ```
 This interface enforces a list of required metadata. If a CF standard name does
@@ -55,7 +56,14 @@ to be true by default. Fields with this attribute will be output with the
 unlimited time dimension added. Time should not be added explicitly in the
 dimension list since it will be added during I/O. Fields that do not change
 with time should include this argument with the value false so that the time
-dimension is not added. Actual field data stored in an array is attached in a
+dimension is not added. The argument RetainPrecision is also optional and should
+be used if the full precision of the field needs to be retained during IO even
+if it is a part of an IO stream that requests reduced precision. An example is
+the time field that can require full precision to represent the seconds from a
+reference time far in the past. The default is false and the argument does not
+need to be supplied in most cases. However, if it is needed, then the
+TimeDependent argument must also be explicitly included due to the language
+rules on argument ordering. Actual field data stored in an array is attached in a
 separate call as described below. Scalar fields can be added by setting the
 NumDims to zero (the Dimensions vector is ignored but an empty vector
 must still be supplied in the argument list). Scalar data is attached using

components/omega/doc/devGuide/IO.md

@@ -101,6 +101,14 @@ reading of variable metadata. For writing, a FillValue is supplied to fill
 undefined locations in an array and the variable ID must have been assigned
 in a prior defineVar call prior to the write as described below.
 
+Writing or reading multiple time slices (where there in an unlimited time
+dimension) is also possible and an additional optional Frame argument
+specifies the time index along that dimension that should be read/written:
+```c++
+int Err = IO::readArray (&Array, Size, VariableName, FileID, DecompID, VarID, Frame);
+int Err = IO::writeArray(&Array, Size, &FillValue,   FileID, DecompID, VarID, Frame);
+```
+
 For arrays or scalars that are not distributed, the non-distributed variable
 interface must be used:
 ```c++
@@ -110,6 +118,17 @@ int Err = IO::writeNDVar(&Array, FileID, VarID);
 with arguments similar to the distributed array calls above. Note that
 when defining dimensions for these fields, the dimensions must be
 non-distributed. For scalars, the number of dimensions should be zero.
+Multiple time slices can be also be read/written for non-distributed fields,
+but require two additional arguments. As in the distributed array, the
+Frame (index of the time slice) must be provided. In addition, a vector
+``std::vector<int> DimLengths`` containing the length of the non-time
+dimensions must be provided:
+```c++
+int Err = IO::readNDVar(&Array, VariableName, FileID, VarID, Frame, DimLengths);
+int Err = IO::writeNDVar(&Array, FileID, VarID, Frame, DimLengths);
+```
+Note that the full arrays in this case are written so if any masking or
+pruning of points is required, it should be performed before the call.
 
 The IO subsystem must know how the data is laid out in the parallel
 decomposition. Both the dimensions of the array and the decomposition
@@ -201,7 +220,12 @@ the MetaValue is the value of the MetaData. All supported Omega data types are
 allowed except boolean which must be converted to an integer type. The FileID
 is once again the ID of the open data file and VarID is the variable to which
 this metadata is attached. For global file and simulation metadata not attached
-to a variable, the ID ``IO::GlobalID`` is used to denote global metadata.
+to a variable, the ID ``IO::GlobalID`` is used to denote global metadata. If
+data is being appended to a file (eg for multiple time slices), writeMeta will
+check first to see if metadata already exists with the same value. If the
+entry is absent, new metadata will be written. If it exists but with a
+different value, the writeMeta function will replace the current entry with
+the new value.
 
 For an example of the full read/write process, the IO unit test contains
 the full reading and writing of a data file and associated metadata. We

components/omega/doc/userGuide/IOStreams.md

@@ -46,12 +46,14 @@ Omega:
         - Tracers
     Highfreq:
       UsePointerFile: false
-      Filename: ocn.hifreq.$Y-$M-$D_$h.$m.$s
+      Filename: ocn.hifreq.$Y-$M
       Mode: write
-      IfExists: replace
+      IfExists: append
       Precision: single
       Freq: 10
       FreqUnits: days
+      FileFreq: 1
+      FileFreqUnits: months
       UseStartEnd: true
       StartTime: 0001-06-01_00.00.00
       EndTime: 0001-06-30_00.00.00
@@ -93,7 +95,8 @@ a template can be:
    - Fail if you want the code to exit with an error
    - Replace if you want to replace the existing file with the new file
    - Append if you want to append (eg multiple time slices) to the existing
-     file (this option is not currently supported).
+     file. When re-running an interval, this will also over-write existing
+     slices corresponding to the simulation time if they exist in the file.
 - **Precision:** A field that determines whether floating point numbers are
    written in full (double) precision or reduced (single). Acceptable values
    are double or single. If not present, double is assumed, but a warning
@@ -114,6 +117,16 @@ a template can be:
    - Hours for a frequency every Freq hours (*not* Freq times per hour)
    - Minutes for a frequency every Freq minutes (*not* Freq times per minute)
    - Seconds for a frequency every Freq seconds (*not* Freq times per seconds)
+- **FileFreq:** An optional entry for including multiple time slices in a
+   single file (only supported for output presently)
+- **FileFreqUnits:** A field that, combined with the integer frequency,
+   determines the frequency of new files when multiple time slices are
+   included in a file. The file frequency must be less than or equal to
+   the data frequency. Acceptable values include all of the frequency values
+   listed above except the one-time values (eg OnStartup/Shutdown or AtTime).
+   The filename template should reflect the frequency and the time used for
+   the file will correspond to the template at the time the file is first
+   opened.
 - **UseStartEnd:** A required entry that is true or false and is used if the
    I/O is desired only within a certain time interval. An example might be
    for specifying high-frequency output within a certain period of a simulation.

components/omega/src/CMakeLists.txt

@@ -49,6 +49,7 @@ target_link_libraries(
     metis
     gptl
     pacer
+    stdc++fs
 )
 
 if(GKlib_FOUND)