Remove flyback

[ercius]

Update from_hdf5 to allow the user to crop the flyback column during load. For large data sets this can reduce the time needed in processing later significantly. The keyword is set to True to be compatible with older code.

Need to fully test once the CI builds are available. Dont merge yet. Will update as needed.

[ercius]

This does not save a lot of time when loading data, but it saves ~2x the memory due to a long standing memory leak in h5py objects. Thus, for large data sets the option to load without the flyback can help with very large datasets and memory constraints.

Ready to be looked at @cjh1, @swelborn!

[ercius]

Another idea is to use fancy indexing. I have not tested the speed yet:

# Remove the flyback data
    to_keep = np.ones((scan_shape[0] * scan_shape[1]), dtype=bool)
    to_keep[scan_shape[0]::scan_shape[1]] = False
    data = frames[to_keep]

[ercius]

There is a minimal effect on the speed of loading with either method. The more recent method would allow any "contiguous load map" to be used rather than just deleting the last (flyback) column. It seems like a useful feature.

[swelborn]

I am not too familiar with sparse array class here, but my two cents in this review. perhaps @cjh1 can give a bit more feedback

[swelborn]

any reason to use ii for indices? instead of i for ex.

[ercius]

I never use single letter variables.

I once spent a full day debugging something where i was already used for sqrt(-1). Ever since, I always use ii, jj, kk, etc. for loops.

Also, when using the pdb debugger, c, q, n, etc. are reserved for interacting with the debugger (continue, quit, next, etc.). That messed me up a few times as well.

[swelborn]

no test?

[ercius]

Looks like I started the test but did not finish it.

[ercius]

test added

[ercius]

I think this is ready for review by someone else.

[cjh1]

LGTM

[cjh1]

@ercius I have cut a new release which contains this change. I have updated the images using by jupyterlab.