Reconstructed images: Linkage between measurement and reconstruction (and calibration)?

[Neumann-A]

MDF does not seem to specify how measurement and a reconstructed image are linked together. Since one measurement can be reconstructed multiple times copying the measurement into each reconstruction file seems inefficient. Although the linkage can be described by a directory structure (see #45) I would rather have an in file description for that (either use file uuid or somehow allow virtual linkage of the HDF5 measurement file [or both])

If the reconstruction is based on a systemmatrix or calibration the linkage to the systemmatrix/calibration must also be specified.

[hofmannmartin]

Since MDF is for exchange I would not rely on the directory structure.

I think it would be much more convenient to use the /uuid, which is unique.

[tknopp]

@NeumannIMT: Where is this issue? /experiment/uuid describes the UUID from the measurement. Based on this knowledge one can obtain the file used for reconstruction.

(Measurement data should not be stored in a Reco file)

[Neumann-A]

@NeumannIMT: Where is this issue? /experiment/uuid describes the UUID from the measurement. Based on this knowledge one can obtain the file used for reconstruction.

you are right did not see that connection. (Missing explanation in spec?)
After thinking about it I questioned if /experiment/uuid is even needed. Is the combination of /study/uuid and /experiment/number not already an unique identifier? Build /experiment/uuid out of /study/uuid and /experiment/number instead of creating a new one? (For me there seems to be one uuid too many)

Wait! Stepping back to get the bigger picture:
File uuid: Identifier of file (is always different between files)
Study uuid: Identifier of study (will be the same for files within the same study)
Experiment uuid: Identifier of experiment ... hmm is the same for files of the same experiment, will be different between new measurements and will not change between different reconstructions for the same file
My conclusion: Experiment uuid is not an extra generated identifier. For measurements files it is the file uuid (A new experiment will always be a new file according to the current specs). For reconstructions it is the file uuid of the reconstructed measurement.

Interessting.

If /experiment/uuid is extra generated or just the file/uuid is in my opinion up to the implementer and another issue. (Won't raise this one; It could have some benefit in searching if experiment/uuid is in fact a file/uuid)

ToDo:
Add text that explains that experiment/uuid links measurement and reco ?

[hofmannmartin]

That is how we intended the UUIDs to be used.

@NeumannIMT would be great if you update the MDF.

[Neumann-A]

@tknopp Sry to bother you again but the proposed solution does not work in all cases. Here is the case.

If I store a raw measurement it has a fixed exp/uuid. If I then postprocess the data and create a new file out of it it has the same exp/uuid. A reconstruction using one of the files thus becomes ambiguous to which file has really been used.

My solution:
Introduce a group /dependencies/ with a 1D dataset /dependencies/data and another 1D dataset /dependencies/calibration to store those data dependencies. The last entry would always point to the latest used data whereas the first entry would always point to the first data ever created within one experimental line. (Somehow like the bitcoin blockchain ;) )

@NeumannIMT would be great if you update the MDF.

Can do if we have a (working) solution.

[tknopp]

yes I understand. We actually store the path to the calibration file in a private Reco parameter. I am actually not so sure what we should do. A dependency group is IMHO not good. If, then lets put that in /reconstruction/.

We use the folder solution, since it scales extremely well and we then always know what the actual measurement was. This solution does however not contains processings, which might be a problem. Not sure. For measurement data we (HH) will never store anything else but the most Raw step. For calibration data certainly not. So, difficult.

[tknopp]

And by the way, we do reconstructions, where several measurements are combined. We also use reconstructions, where several calibrations are involved.

[Neumann-A]

If, then lets put that in /reconstruction/.

seems to be not general enough if we allow postprocessing on data or calibration files.

A dependency group is IMHO not good.

Why? From viewpoint it seems to be a more general solution.

several measurements

useddata dataset as an array to all uuids used to get the new file?
(Will only link to the most recent used data)

[hofmannmartin]

I think at one point this issue is reaching a little bit too far. Most of the MDF users will not have a complete MDF infrastructure, but merely methods to convert from their respective data format to MDF.

If so it is most likely that the dependencies will often be be non MDF files. Or even worse it could be that my system matrix is stored inside a MDF, but the measurements are not. What to do in such a case? Storing just the MDF part seems useless to me without the non MDF part.

From this perspective I would question the why we need a to specify how to link measurements and reconstructions in the first place. If we have no specifications on this issue the user has the freedom to create his own solutions if he has the need to (like we do here in Hamburg).

[tknopp]

I am marking this post v2.0. Dependencies are an open issue but they are highly related to reconstruction parameters which we also do not specify in 2.0. The main focus now are how to store measurements and calibrations.