imageseries dev docs

docs/source/dev/imageseries-load-options.rst

@@ -0,0 +1,98 @@
+.. _keyword-options:
+
+Keyword Options for imageseries
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Each type of imageseries has its own keyword options for loading and saving.
+
+Image Files
++++++++++++++
+
+The format name is ``image-files``.
+
+This is usually written by hand. It is a YAML-based format, so the options are
+in the file, not passed as keyword arguments. The file defines a
+list of image files. It could be a list of single images or a list of
+multi-imagefiles.
+
+YAML keywords are:
+
+``image-files``
+    dictionary defining the image files
+
+    - ``directory``: the directory containing the images
+    - ``files``:  the list of images; it is a space separated list of file
+      names or glob patterns
+
+``empty-frames``
+    (optional) number of frames to skip at the beginning of
+    each multiframe file; this is a commonly used option
+
+``max-total-frames``
+    (optional) the maximum number of frames in the imageseries; this option
+     might be used for testing the data on a small number of frames
+
+``max-file-frames``
+    (optional) the maximum number of frames to read per file; this would
+    be unusual
+
+``metadata``
+    (required) it usually contains array data or string, but it can be empty
+
+There is actually no write function for this type of imageseries. It is
+usually used to load image data to be sparsed and saved in another (usually
+frame-cache) format.
+
+
+
+HDF5
+++++++++++
+
+The format name is ``hdf5``.
+
+This is used at CHESS (Cornell High Energy Synchrotron Source). Raw data from
+the Dexela detectors comes out in HDF5 format. We still will do the dark
+subtraction and flipping.
+
+**On Write.**
+
+``path``
+    (required) path to directory containing data group (data set is named
+    `images`
+
+``shuffle``
+    (default=True) HDF5 write option
+
+``gzip``
+    (default=1) compression level
+
+``chunk_rows``
+    (default=all) sets HDF5 chunk size in terms of number of rows in image
+
+**On Open.**
+
+``path``
+    (required) path to directory containing data group (data set is named
+    `images`)
+
+Frame Cache
+++++++++++++++++++++
+The format name is ``frame-cache``.
+
+A better name might be sparse matrix format because the images are stored as
+sparse matrices in numpy npz file. There are actually two forms of the
+frame-cache. The original is a YAML-based format, which is now deprecated.
+The other format is a single .npz file that includes array data and metadata.
+
+**On Write.**
+
+``threshold``
+    (required) this is the main option; all data below the threshold is ignored;
+    be careful because a too small threshold creates huge files; normally,
+    however, we get a massive savings of file size since the images are
+    usually over 99% sparse.
+
+``output_yaml``
+    (default=False) This is deprecated.
+
+**On Open.**
+No options are available on open.

docs/source/dev/imageseries-old.rst

@@ -0,0 +1,17 @@
+imageseries package
+===============
+The *imageseries* package provides a standard API for accessing image-based data sets.  The primary tool in the package is the ImageSeries class. It's interface is analagous to a list of images with  associated image metadata. The number of images is given by the len() function. Properties are defined for image shape (shape), data type (dtype) and metadata (metadata). Individual images are accessed by standard subscripting (e.g. image[i]).
+
+The package contains interfaces for loading (load) and saving (save) imageseries. Images can be loaded in three formats: 'array', 'hdf5' and 'frame-cache'. The 'array' format takes the images from a 3D numpy array. With 'hdf5', images are stored in hdf5 file and accessed on demand. The 'frame-cache' is a list of sparse matrices, useful for thresholded images. An imageseries can be saved in 'hdf5' or 'frame-cache' format.
+
+The imageseries package also contains a  module for modifying the images (process). The process module provides the ProcessedImageSeries class, which takes a given imageseries and produces a new one by modifying the images. It has certain built-in image operations including transposition, flipping, dark subtraction and restriction to a subset.
+
+
+Metadata
+----------------
+
+The metadata property is generically a dictionary. The actual contents depends on the application. For common hexrd applications in which the specimen is rotated while being exposed to x-rays, the metadata has an 'omega' key with an associated value being an nx2 numpy array where n is the number of frames and the two associated values give the omega (rotation) range for that frame.
+
+Reader Refactor
+-------------
+While the imageseries package is in itself indpendent of hexrd, it was used as the basis of a refactoring of the reader classes originally found in the detector module. The main reader class was ReadGE. In the refactored code, the reader classes are now in their own module, image_io, but imported into detector to preserve the interface. The image_io module contains a generic OmegaImageSeries class for working with imageseries having omega metadata. The refactored ReadGE class simply uses the OmegaImageSeries class to provide the same methods as the old class. New code should use the OmegaImageSeries (or the standard ImageSeries) class directly.

docs/source/dev/imageseries-overview.rst

@@ -33,10 +33,16 @@ format. Possible formats currently are:
   image files and metadata.
 * ``array`` - images are stored as a 3D numpy array; used for testing.
 
-See also [load options](https://github.com/donald-e-boyce/hexrd/wiki/ims-load-options).
+See also :ref:`keyword-options`.
 
 **Processed Imageseries.**
-This is a subclass of imageseries. It has a number of built-in operations, such as flipping, dark subtraction, restriction to a sub-rectangle. It was intended to be further subclassed by adding more operations. It is instantiated with an existing imageseries and a list of operations. When a frame is requested, the processed imageseries gets the frame from the original image series and applies the operations in order. It can then be saved as a regular imageseries and loaded as usual.
+This is a subclass of imageseries. It has a number of built-in operations,
+such as flipping, dark subtraction, restriction to a sub-rectangle, and
+selecting frames. It can be further subclassed by adding more operations. It is
+instantiated with an existing imageseries and a list of operations. When a
+frame is requested, the processed imageseries gets the frame from the original
+image series and applies the operations in order. It can then be saved as a
+regular imageseries and loaded as usual.
 
 For more detail, see [processed imageseries](https://github.com/donald-e-boyce/hexrd/wiki/processed-ims).
 
@@ -66,3 +72,7 @@ For the hexrd work, we usually have a sequence of rotations about the vertical a
 See [omega](https://github.com/donald-e-boyce/hexrd/wiki/imageseries-omega).
 
 .. include:: imageseries-usage.rst
+
+.. include:: imageseries-load-options.rst
+
+.. include:: imageseries-processed.rst

docs/source/dev/imageseries-processed.rst

@@ -0,0 +1,74 @@
+.. _processed-ims:
+
+Processed Image Series
+^^^^^^^^^^^^^^^^^^^^^^^
+This class is intended for image manipulations applied to all images of the
+imageseries. It is instantiated with an existing imageseries and a sequence of
+operations to be performed on each image. The class has built-in operations for
+common transformations and a mechanism for adding new operations. This class is
+typically used for preparing raw detector images for analysis, but other uses
+are possible. The `rectangle` operation is used in ``stats.percentile`` to
+compute percentiles one image section at a time to avoid loading all images at
+once.
+
+Instantiation
+++++++++++++++++++++
+Here is an example:
+
+.. code-block:: python
+
+    oplist = [('dark', darkimg), ('flip', 'v')]
+    frames = range(2, len(ims))
+    pims = ProcessedImageSeries(ims, oplist, frame_list=frames)
+
+Here, `ims` is an existing imageseries with two empty frames. The operation
+list has two operations. First, the a dark (background) image is subtracted.
+Then it is *flipped* about a vertical axis. Order is important here; operations
+do not always commute. Note that the dark image is usually constructed from
+the raw images, so if you flipped first, the dark subtraction would be wrong.
+Finally, the only keyword argument available is `frame_list`; it takes a
+sequence of frames. In the example, the first two frames are skipped.
+
+
+Built-In Operations
+++++++++++++++++++++
+The operation list is a sequence of (key, data) pairs. The key specifies the
+operation, and the data is passed with the image to the requested function.
+Here are the built-in functions by key.
+
+``dark``
+    dark subtraction; it's data is an image
+
+``flip``
+    These are simple image reorientations; the data is a short string;
+    possible values are:
+
+    - ``y`` or ``v``: flip about y-axis (vertical)
+    - ``x`` or ``h``: flip about x-axis (horizontal)
+    - ``vh``, ``hv`` or ``r180``: 180 degree rotation
+    - ``t`` or ``T``: transpose
+    - ``ccw90`` or ``r90``: rotate 90 degrees
+    - ``cw90`` or ``r270``: rotate 270
+
+    Note there are possible image shape changes in the last three.
+
+``rectangle``
+    restriction to a sub-rectangle of the image; data is a 2x2 array with each
+    row giving the range of rows and columns forming the rectangle
+
+Methods
+++++++++++
+In addition to the usual imageseries methods, there are:
+
+.. code-block:: python
+
+    @classmethod
+    def addop(cls, key, func):
+        """Add operation to processing options
+
+        *key* - string to use to specify this op
+        *func* - function to call for this op: f(img, data)
+        """
+    @property
+    def oplist(self):
+        """list of operations to apply"""