-
Notifications
You must be signed in to change notification settings - Fork 34
Future Processing
This page describes a feature planned for a future version of Nion Swift.
The user must be able to configure live processing and analysis between library items.
Computations should allow the user to adjust their parameters via an inspector always and graphically when possible. When graphical, the connection goes both ways: from the graphics to the computation; but also from the computation inspector to the graphic.
Computations are explicitly allowed to use graphics on their associated display to adjust the data. They are not allowed to use their own data output as input (no feedback).
Computations keep track of the provenance as much as possible. All built-in computations should do this; custom computations based on raw NumPy may not.
Computations may have multiple steps and users may need to view intermediate data. Or: computations may have multiple data outputs.
Computations may be lengthy in nature and there must be a mechanism to handle lengthy computations.
Computations may debounce or sample inputs for efficiency.
Computations must handle case of not writing themselves or any dependents (including via regular binding) to disk during live adjustment.
Computations must allow for various data input styles that might include source data, reduced source data, display data, a mask, or a rectangular input.
Computations must allow for various output including data, regions, objects, display parameters, and numbers (statistics).
Computations must be stateless, but may have an associated data structure as input.
Computations allow library items, data arrays, masks, and variables to be inputs as required, optional, or as a dynamic list.
Computations allow library items to be outputs as required or as a dynamic list.
Computations are recomputed when any input changes.
Computations track dependencies between items.
Computations may delete themselves (and effectively be one-use). (???)
Deleting an input to a computation deletes the computation.
Deleting a computation deletes all outputs.
- All built-in processing (e.g. inverse, line profile, crop, resample, change data type, transpose, cross correlate, rgb split, pick region, projection)
- Double gaussian with intermediate data
- Stamp tool which stamps selection onto itself
- 1-d background subtraction.
- EELS UI, Multiple EELS acquire too
- Pick region on SI with pick graphic shown on SI and HAADF
- Pick point on 4d with pick point shown on HAADF
- Map reduce a 4d image to produce a 2d image
- Feature finder (peaks on 1-d, faces on 2-d)
- Operate on a rotated crop
- Operate on a Fourier mask filter
- Operate on a shape filter (rect, ellipse, point)
- Statistics
The algorithm invert is a simple point-by-point operation. It may, however, work on a rectangular region, including one that is rotated.
The algorithm crop has inputs of a rectangular graphic that may be rotated on the original data.
The algorithm resample produces data that is a different size than the source. The algorithm change data type produces a different data type. The algorithm projection produces data that has a different shape. The algorithm transpose produces data that is a view on the original data but with axes switched.
The algorithm line profile has the inputs of src, vector, and integration width. A line profile graphic on the source represents the inputs graphically. The vector and integration width can be represented in multiple coordinate systems and the user must have an inspector that reflects those choices and allows them to set it how they want. In particular, the inspector for the algorithm inputs and the inspector for the line profile graphic are largely the same. The line profile might have extra choices such as the color. The inspector for the algorithm can either refer to the line profile graphic (as it is now) or it can break out the parameters into their own model. The naming system may come into play here too as it needs the graphic to be sensible. So we can probably assume that the graphic 'model' fully represents the parameters and the algorithm will present the same inspector as the graphic.
The algorithm pick region has inputs of a mask region composed of one or more graphics combined to form a mask. The mask may be boolean or floating point. The input is 2d or higher and produces output with two fewer dimensions than the source. The algorithm shouldn't be tied to the graphics; but to the mask data.
The special case of 3d spectrum image reduced to a 1d spectrum may also be used to graphically display the slice of the 3d spectrum image used to convert the 3d to 2d for display.
The algorithm cross correlate takes two inputs and produces a single output. The algorithm RGB split takes a single input and produces multiple outputs.
The algorithm double gaussian has inputs of src, sigma1, sigma2, and weight. Useful intermediate data and/or visualizations include a 1-d line plot of the radial profile and also a filtered FFT with the position of sigma1 and sigma2 indicated by ellipse graphics. Ideally, we could avoid doing double calculation of the filtered FFT.
The algorithm stamp has inputs of the source image, an area graphic on the target, and a destination point on the target. It produces a copy of source with the area graphic copied and stamped onto the destination point of the target. The key point here is that the target data is dependent on graphics on itself.
The algorithm background subtraction has inputs of a 1-d spectrum and outputs a group of spectrum which are the original source, a fit for a region on the source, and a background subtracted version of the source. The fit region is specified by a graphic on the composite output line plot display. The output data items are shown as layers on the source display item.
The algorithm EELS mapping is covered by the other use cases. It differs, however, in where the parameters for the algorithm are stored. Most of the use cases above could be stored on the target. EELS mapping must be stored on either the source or independently.
The algorithm 4d pick is intrinsic to the display of 4d data. The input is the collection index and it is stored with the display. However, there may be a mapping image associated with the 4d image and a pick point or region could be displayed on the associated mapping image and linked to the collection index/map.
Algorithms that reduce the data via a pick/slice/project may have sources with associated map images.
The algorithm map-reduce produces 1d or 2d data from higher dimensional data by iterating over collection dimensions and generating 0d or 1d data for each datum via a secondary algorithm. A 4d data set with two collection dimensions could be reduced to a 2d data set by summing all values in each 2d datum image. Likewise a 3d data set with one collection dimension could be reduced to a 2d data set by projecting each 2d datum image and stacking them into a 2d data set.
The algorithm feature-finder produces a list of positions of features that are displayed using graphics on the source. The output is only graphics. Finding peaks on 1d data, potential edges, or faces in a 2d image are example features.
The algorithm highlight produces display limits and a color table to highlight the peaks in an image.
The algorithm Fourier mask produces a mask region, an FFT with the mask applied, and a filtered image. The mask region is comprised from Fourier mask graphics on the source combined. The algorithm shouldn't be tied to the graphics; but to the mask data.
The algorithm statistics produces 0d data from the source.
The algorithm sum-n sums n line plots into a single output. User can edit the input list.
Other variations are an algorithm with a single source but a fixed number of input regions, and an algorithm with a single source and a dynamic number of inputs.
The algorithm integrate sequence integrates a sequence by summing the sequence dimension. The user may be selected a slice of the sequence.
The algorithm trim sequence trims a sequence by extracting a sequence slice into a new sequence data item. The algorithm extract from sequence extracts a single sequence index into a new data item.
The algorithms insert and concatenate form a new sequence by inserting/concatenating a similar sequence into a position in an existing sequence.
-
uint8
-
uint16
-
uint32
-
int16
-
int32
-
float32
-
float64
-
complex64
-
complex128
-
rgb8
-
rgba8
-
rgb16
-
rgba16
-
rgbfloat
-
rgbafloat
-
TODO: represent a sequence of vectors
-
TODO: represent a vector field
- Spectrum (1 data dimension)
- Raster (2 data dimensions)
- Scan (2 collection dimensions)
- Line scan (1 collection dimension, 1 data dimension)
- 2d scan (1 collection dimension, 2 data dimensions)
- 2d stack (1 collection dimension, 2 data dimensions)
- Spectrum image (2 collection dimensions, 1 data dimension)
- 4d image (2 collection dimensions, 2 data dimensions)
- Sparse XYZ (data points)
- Sparse XY (data points)
- Sparse Y (data points)
- Movie (sequence of 2 data dimensions)
- Scan movie (sequence of 2 collection dimensions)
- Sequence {T} (sequence of {T})
- Fractional
- Pixel
- Calibrated
- Zero
- Center
Ways to reduce data from one form to another
- Sequence slice (range from sequence dimension)
- Sequence index (point from sequence dimension)
- Vector (1d vector of either data or collection dimensions)
- Line Profile (1d vector area of either data or collection dimensions)
- Crop (2d crop of either data or collection dimensions)
- Rotated crop (2d rotated crop of either data or collection dimensions)
- Interval (1d crop of either data or collection dimension)
- Channel (1d point of either data or collection dimension)
- Mask
- Fourier mask
Should 1D windows be applied along rows and columns? Or should a 1D window be applied radially? If radially, should unequal dimensions be stretched to a square or treated as square pixels? How should the overall results be normalized?
Reference: https://holometer.fnal.gov/GH_FFT.pdf
Reference: https://dsp.stackexchange.com/questions/36221/scaling-for-a-2d-hanning-window
Reference: https://en.wikipedia.org/wiki/Two_dimensional_window_design
The script API should allow for setting up live computations. Initially it can be limited to predefined operations; but eventually expanded to arbitrary scripts with arbitrary inputs.
Any processing steps should be recorded.
More than one computation per item?
Computation should also specify how they modify it related coordinate system. Pick relates to display interval. Map 4d image relates to collection index.
Change all scripts to use direct item. Allowed most flexibility. Source you're merely indicates type of listener.
Composite operations. Allow map to take any operation with input of one of its datum and reduced it. Can work in sequence or collection. But start with scalar functions? Also allow regular computations to include multiple steps, like originally thought.
Computations window. Allow listing of all computations. Menu item to show computations associated with a particular item.
Change data panel to list all items. Data items are a row with all display, etc. Same with computations and data structures. Don't list connections. Quickly filter for displays. Show progress or activity for computations.
Item references and item reference lists in data structures. General data structure editor.
Registered reduction functions must allow a UI. For instance dark subtraction requires another image.
Registered functions and things in packages that are persistent should include a URL for their package home page.
Could reduce be implicit in all operations? Doing FFT on collection does it for everything. FFT3d is separate. Sum to scalar. Function must limit itself to 1d or 2d and have a flag that says "is collection compatible". Some registered functions can provide the collection aware version for efficiency.
Computations should register their UI separately? Either way, the handler for the computations could be implicit, so user only writes the UI itself. Maybe. Or standard handler can se be injected into custom handler is necessary.
Reduction computations should specify input requirements and output style. For instance FFT takes 1d, 2d and can produce hermitian or complex. All combinations should be iterated. This allows chaining of functions in one image. Should also define coordinate system transformation.
Some algorithms should be able to pick specific algorithm on the fly - pick a point vs pick sum a mask. User should be able to change the mask on the fly.
User should have a way to pick on the result of a map operation to pick the 2d or 1d item from the source.
Picking and doing an operation should be back computed on every pixel of source. Background subtraction.
spectrum = pick_fn(spectrum_image, mask)
background, signal = subtract_background_fn(spectrum, fit_interval, signal_interval)
map_background, map_signal = subtract_background_fn(spectrum_image, fit_interval, signal_interval)
display = composite(spectrum, background, signal, fit_interval, signal_interval)
Setup:
- create an object (null computation?) to hold the source, fit, signal, electron shell
- create a computation to produce the spectrum
- create a computation to produce the background and signal, dependent on null computation
- construct a composition to display the results of the spectrum + background + signal
- add a task to update composition display when first computation complete
- add UI to switch composition to display other spectrum + background + signal available on source
Requirements for the elemental mapping object:
- track dependents
- track transactions
- allow connections
- cascade deletes
A computation has sources, which can be data, regions, or other, but are not currently individual properties.
A computation has properties.
A computation has results (dependents), which can be data items, regions, or other, but, again, are not currently individual properties.
A computation has methods to do computation (execute) and apply results (commit).
... but maybe the computation needs to be separated into a data storage part and an action part.
How would this work? A crop computation would be 'crop parameters' and 'crop executor'. It's really already this way, but there is no way to follow dependencies, transact or connect to variables in the computation.
Determining dependents:
- library computations list inputs and outputs
- data item computations list inputs (and implicit output of data item)
- composite items list inputs (composite item is implicit output)
- connections (both sides are inputs and outputs, depends how the connection is reached in tree)
- connections have a control direction (dependent to source)
- and a dependency direction (source to dependent)
- only the dependency direction contributes to the dependency graph
For library connections, how do display properties (slice) fit into the scheme? Should display be a proper object accessible by the computations and connections? Or is it a special case of something?
Difference between a computation and a connection? Computation happens an unspecified time later; connections happen immediately. Computations have well defined source and targets; connections are bidirectional.
Maybe connections should be superseded by computations that act immediately and don't track dependencies in one direction?
Generalized definition: Computations watch source objects for changes, run a Python script, and store the results in persistent objects.
The source objects may be data, metadata, graphics, etc. and change notifications may be debounced and/or sampled.
The result objects may be data, metadata, graphics, etc.
The declarative portion is what gets written out to files. The initial configuration of these processes is custom and no attempt is made to make it anything except that. Once configured, the processes will get notifications such as 'loaded' or 'input changed' or 'output deleted'. The processes can decide what to do with those notifications.
A connection or binding is process which is bidirectional.
Any processes that can be saved to a file must be declarative in nature in order to facilitate continued maintainance. The underlying script should carry out the process described declaratively.
Active scripts, which do immediate processing and/or set up persistent processes, do not need to be declarative since they only need to be valid at the time they are run.
Is it possible to make computations work entirely within the API, including dependencies, variables, and outputs? For instance, a computation could define its variables, configure its connections, and do its computation. A computation would be instantiated and attached to specific data items or other objects.
As discussed 2016-11-01, a computation will have parameters to its algorithm. If those parameters are connected to a graphic (for instance, a line parameter could be connected to a line graphic), the computation inspector does NOT need to include secondary graphic parameters such as color. Those can be edited by the user on the graphic if necessary. The computation may initially configure those parameters; but they need to be present past configuration.
Different way of looking at computations: When this data changes or that object changes, run this script [in background] [async].
When a computation reloads, it will have inputs and a script, either of which may be invalid. The computation handler must be able to examine the current inputs and update them as required for the current version, and then update the script to the current version.
For reloading there are several version designations to take into account.
- Computation environment version
- ~1.0 was domain specific and custom scripts are no longer supported.
- ~2.0 is xdata based, requiring explicit inputs.
- 3 may be class based, but isn't in use yet.
- API version
- ~1.0 is the current standard API
- Version of the specific computation (i.e. FFT 1.0)
- This should only be bumped if the inputs change, otherwise this is just a script change to current version.
Dependent data doesn't copy metadata anymore; but user would still like to see metadata associated with source of data when there is one. What mechanism?
If double gaussian parameters are stored on the source, it provides a convenient sharing location. But are more than one set of parameters stored? If not, how would one do comparison computations? If so, then how would one know when they're deleted? To provide comparison, more than one set of parameters must be provided. To delete, dependencies can add themselves to the observers list and the parameters get deleted when there are no more observers. The UI may count as an observer for some types of parameters, but probably not for double gaussian. Graphics are currently stored on the source.
Alternatively, double gaussian parameters can be stored on the result item. If this is done, there is no problem knowing when to delete the parameters since they get deleted when the result gets deleted. On the other hand, showing intermediate images now becomes problematic because the intermediate images would need to have knowledge of their downstream result (the parameters).
Another way to look at that problem, though, is to define the master parameters on the result. But each computation has its own set of parameters. They just get linked via binding. The problem here is that intermediate results don't get shared. If they do get shared, then there is a funny situation of having the intermediate image depend on the parameters stored on the result image; and the result image not directly using the parameters.
Still another option is to store the double gaussian parameters in its own library item. Inspectors for that library item still need to be available for each of the intermediate and result images, though. Now the intermediate images are free to use the parameters and the result can cleanly depend only on the intermediate images. There would need to be a mechanism to display the parameter inspector on each of the intermediate and result images.
Computation inputs can be sources, parameters sets, or individual parameters. Sources provide raw data. The inspector must be able to able to inspect all three types of inputs. In addition, there must be an ability to bind parameters sets together. Are parameters sets part of the computation object or do they exist separately in the data item?
Must a parameter set for 'rectangle' must include the bounds, image size, calibration information, and display choice? Or just simply an inspector for the source graphic? Are the objects such a graphics good enough to count as the parameter set? Is there some way to define standard subsets of objects such as graphics to simplify the inspector portion used in the computation? What happens for extended items such as mask graphic sets? If parameter sets are to be shared, perhaps they should include intrinsic UI widget, perhaps with minor options such as to display only coordinates and not display related items such as titles.
Another thought is that an algorithm doesn't depend on a graphic; the graphic is only a representation of the parameters to an algorithm. So perhaps defining the algorithm in terms of a parameter model object (parameters) and then linking the graphic to the parameter model object is a better way to think about the architecture. On the other hand, the Graphic object in the model is exactly that... the parameters.
- display.slice_interval === interval_region.interval
- elemental_mapping.fit_interval === fit_region.interval
- elemental_mapping.signal_interval === signal_region.interval
- line_profile_graphic1.vector === line_profile_graphic2.vector
- line_profile_graphic1.width === line_proflie_graphic2.width
- the interval list thing: intervals on the data item display === line_profile.interval_list
Three ways to handle computations.
- Computations run synchronously and provide a secondary asynchronous function to evaluate from plug-ins.
- Requires special handling (recording) to make model changes.
- Easiest to understand for the user, but least flexible
- Go full asynchronous and make xdata functions asynchronous.
- Complicates the implementation of xdata and confuses script writers.
- Full asynchronous and make user put anything asynchronous in an inner function. xdata stays synchronous.
- Would require user to always define the async function.
Idea for making the computation self contained and easy to migrate: a computation establishes its object inputs on it's containing data item. When the computation creates a graphic/object on a source, it marks itself as a listener to that graphic/object. At startup, all items are loaded and then connections are established between the data and the graphic. If there are graphics that don't have current references, they're deleted. Then the computations are free to reestablish graphic connections. In this way, the parameters for the computation are stored on the target and the computation is free to establish new graphics on the sources as needed during migration.
Computations are difficult to configure — so it would be nice to use the full power of Python, through the API, to configure them. That means that it must be easy to understand where the 'model' is and also easy to enable connections between the model and the algorithm and the model and the graphics. There must be an easy way to migrate from one version to another and enough information to know whether a particular computation setup is valid for the version of Swift. Ugh.
Rejected idea of declaring inputs (context) using code. There are a number of reasons for the rejection including that it is too wordy, too inflexible, difficult to reconcile the declaration with the execution (i.e. run it once, it generates inputs? then run it again once the items in the inputs have been mapped?), difficulty understanding what happens when the user removes an input in code or the UI, etc.
Furthermore, the inputs are the context. So the way it was before was OK.
Still need to solve problem of how to have multiple outputs for a script. If the script is attached to a data item, I would argue that there should not be multiple outputs. However, scripts attached to the workspace or somewhere else may have multiple outputs. So how to define that?
Perhaps explicitly defining the target "intrinsic" or whatever.
What is the data flow through the data item to display pipeline and how does it allow scripts to be attached?
data = data_fn(data, metadata, regions) display_data = display_data_fn(data, inputs)
Possible inputs are:
- data/calibration
- metadata
- graphics
- mask
- variables
- objects (stored within metadata)
A data item consists of:
- data/calibration
- metadata
- display/graphics
- variables
It is possible to make data/calibration be a function of a graphic attached to the same data item. It is also possible to make a graphic be a function of data/calibration of the same data item. It is not possible to do both at the same time.
Computations probably need to specify finer grain dependencies — data, metadata, graphics, etc.
What's legal to do in a computation:
- DataItem: metadata
- BufferedDataSource: data and metadata, metadata
- Display: most properties except calibration style; add/remove graphics
- Graphics: any property
Computations take time and should update the data item all at once.
Some computations may take much longer than others.
Current computations can only update the data and metadata and must use the data toolkit.
Would be nice to be able to write arbitrary Python code, including numpy.
The script for predefined computations should be for information purposes only; not set in stone.
A computation will use the snapshot of its inputs at the time the computation starts.
The inputs to the computation may change while the computation is occurring; which will mark the computation dirty.
If a computation is dirty, it should be serviced and recomputed.
While a computation is running, several things can happen which invalidate it: closing document, removing data item, and removing computation.
Perhaps it shouldn't be API objects which are passed — but a more limited object?
IDEA: Computations could always apply to a copy of the original object, which would then replace the original in an atomic operation. If multiple computations are attached to an object, then they must be run serially and apply themselves to the same copy. If the user modifies the original object during the computation, we can decide whether to discard the computation completely or whether to perform a new computation. Obviously user modifications cannot be thrown out. It may also be possible to partially update an object if we can keep track of what part of the object changed. For slow parts of the object, for instance the data itself, the new object should be (effectively) copy on write.
User needs to be able to tab or something to autocomplete and be able to search. In addition, clicking in a method should give a description and parameters.
Computations currently calculate data, which is then set into the object on the processing thread. Is this OK? Also, what about long running computations? How would a computation queue the change to the data item to the main thread?
Computations should be forward compatible. Computations don't need to be backwards compatible, but a warning would be nice if possible.
To be forwards compatible, a computation script must specify:
- Input variables and their API level
- Data processing libraries and their version
The data and metadata and computation system:
- Processing should describe how to change the data, descriptor, and calibrations.
- Processing should also include the history of processing, but only resolve references through an external system.
- Computations should track changes to input objects, execute stored procedures, and update library items.
- Replace connections with new computations.
- Recognize the distinction between the descriptions of the procedure and the actual values used in the procedure
- Computations will inherently have targets, inputs, and other parameters
Q: How are 'connections' handled? Explicitly or implicitly (established when the computation loads)?
Q: Can processing do anything meaningful with metadata? Should it be included in the toolkit?
Q: Can processing do anything meaningful with displays? Should it be included?
Q: Could the toolkit work better as procedures working on model data items rather than always returning new data? or at least a mix?
More thoughts on data and metadata naming:
- composite data (comdata)
- data object (data_obj)
- data item (data_item)
- data and metadata (datamet)
TODO: make sure computations reload according to id, not script
TODO: begin versioning to new script format
TODO: provide functions to register new imports/functions for computations
TODO: provide functions to construct new computations
"data toolkit" — dtk
"data and metadata" — mdata, xdata?
- each function should have an id and describe its function call (and version) in the metadata
Several ideas for upgrading computations:
- decorators
- subclassing
- fixed function names Another goal: copy and paste entire computation, including data inputs and identifier