Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add .seq format for DE 16 and Celeritas Camera #11

Open
wants to merge 114 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
114 commits
Select commit Hold shift + click to select a range
0cd7aae
New Feature:Added in specification for DE .seq format
CSSFrancis Aug 5, 2022
351f6e3
New Feature:Added in generic methods for reading binary data
CSSFrancis Aug 5, 2022
b5b6dd6
Testing: Added tests using large files from the old streampix softwar…
CSSFrancis Aug 9, 2022
6c36f1c
NewFeatures: Added code for reading full seq files, better metadata h…
CSSFrancis Aug 9, 2022
54b4184
BugFix: Proper accounting for Buffer in Celeritas File Format
CSSFrancis Aug 11, 2022
d7aacdd
Refactor: Created a Class for loading .seq files
CSSFrancis Aug 11, 2022
9e29ad0
Refactor: Fixed Celeritas Reader class and implemented memmaping. In…
CSSFrancis Aug 11, 2022
c16ce87
Testing: Cleaned up testing. Small changes to code fixing errors with…
CSSFrancis Aug 11, 2022
2806124
Testing: Added additional tests for different file formats
CSSFrancis Aug 11, 2022
f0661c3
BugFix: Fixed lazy loading
CSSFrancis Aug 12, 2022
c12d5e6
Testing: Added testing for navshape larger than file
CSSFrancis Aug 12, 2022
29d45da
Documentation: Improved documentation for .seq format
CSSFrancis Aug 15, 2022
cde18b3
Formatting: Black Formatting Applied
CSSFrancis Aug 15, 2022
f118e2d
Formatting: added rsciio header instead of hyperspy header Removed pr…
CSSFrancis Aug 15, 2022
151efa9
Bugfix: Fixed improper reading of axes size in celeritas
CSSFrancis Aug 15, 2022
d4b42cb
Bugfix: XML File is recursively read
CSSFrancis Aug 24, 2022
256745d
NewFeature: Added in distributed Processing
CSSFrancis Aug 29, 2022
af28007
Testing: Added tests for writing extra frames
CSSFrancis Aug 29, 2022
ca9c437
Formatting: Ran black on changed files
CSSFrancis Aug 29, 2022
3df4fdd
Testing: Tests passing
CSSFrancis Aug 29, 2022
50cc879
Bugfix: Added in dtype
CSSFrancis Aug 29, 2022
3ac68ca
Bugfix: Fixed large chunking issue
CSSFrancis Aug 29, 2022
0093950
Bugfix: Fixed issue with memmap being loaded into memory prematurely
CSSFrancis Aug 29, 2022
e3c8be7
Bugfix: Fixed issue scalars in metadata. Stopping values from saving …
CSSFrancis Aug 29, 2022
376eae9
Bugfix: Fixed issue scalars in metadata. Stopping values from saving …
CSSFrancis Aug 29, 2022
5a32772
Bugfix: Fixed issue scalars in metadata. Stopping values from saving …
CSSFrancis Aug 29, 2022
9fec021
Bugfix: Fixed issue scalars in metadata. Stopping values from saving …
CSSFrancis Aug 29, 2022
5b0a14d
Bugfix: Fixed issue scalars in metadata. Stopping values from saving …
CSSFrancis Aug 29, 2022
d934c68
NewFeature: Added a `distributed_memmap` a drop in replacement for `n…
CSSFrancis Nov 5, 2022
326a94c
NewFeature: Replaced reading seq file with generic reader.
CSSFrancis Nov 6, 2022
c8f8cf4
BugFix: Fixed generic .seq reading. Added tests for distributed and n…
CSSFrancis Nov 6, 2022
62d65eb
BugFix: Reduced tests. No longer testing adding blank frames.
CSSFrancis Nov 6, 2022
2006b53
BugFix: Bad Pixel map now saved with metadata.
CSSFrancis Nov 7, 2022
fde0345
BugFix: Bad Pixel map now saved with metadata.
CSSFrancis Nov 7, 2022
340dc7e
New Feature:Added in specification for DE .seq format
CSSFrancis Aug 5, 2022
1e821b6
New Feature:Added in generic methods for reading binary data
CSSFrancis Aug 5, 2022
56d80a4
Testing: Added tests using large files from the old streampix softwar…
CSSFrancis Aug 9, 2022
ed111db
NewFeatures: Added code for reading full seq files, better metadata h…
CSSFrancis Aug 9, 2022
abd0d13
BugFix: Proper accounting for Buffer in Celeritas File Format
CSSFrancis Aug 11, 2022
af95663
Refactor: Created a Class for loading .seq files
CSSFrancis Aug 11, 2022
1a37d72
Refactor: Fixed Celeritas Reader class and implemented memmaping. In…
CSSFrancis Aug 11, 2022
b8bd597
Testing: Cleaned up testing. Small changes to code fixing errors with…
CSSFrancis Aug 11, 2022
1beb154
Testing: Added additional tests for different file formats
CSSFrancis Aug 11, 2022
d9206e2
BugFix: Fixed lazy loading
CSSFrancis Aug 12, 2022
9e78606
Testing: Added testing for navshape larger than file
CSSFrancis Aug 12, 2022
b15fe87
Documentation: Improved documentation for .seq format
CSSFrancis Aug 15, 2022
cadd8bf
Formatting: Black Formatting Applied
CSSFrancis Aug 15, 2022
561eda7
Formatting: added rsciio header instead of hyperspy header Removed pr…
CSSFrancis Aug 15, 2022
8418dfd
Bugfix: Fixed improper reading of axes size in celeritas
CSSFrancis Aug 15, 2022
e4176b4
Bugfix: XML File is recursively read
CSSFrancis Aug 24, 2022
7c1c89c
NewFeature: Added in distributed Processing
CSSFrancis Aug 29, 2022
16786f3
Testing: Added tests for writing extra frames
CSSFrancis Aug 29, 2022
9f12dc2
Formatting: Ran black on changed files
CSSFrancis Aug 29, 2022
318f5c8
Testing: Tests passing
CSSFrancis Aug 29, 2022
cdf3ae6
Bugfix: Added in dtype
CSSFrancis Aug 29, 2022
bcee58b
Bugfix: Fixed large chunking issue
CSSFrancis Aug 29, 2022
36092a1
Bugfix: Fixed issue with memmap being loaded into memory prematurely
CSSFrancis Aug 29, 2022
52d0c56
Bugfix: Fixed issue scalars in metadata. Stopping values from saving …
CSSFrancis Aug 29, 2022
70781b2
NewFeature: Added a `distributed_memmap` a drop in replacement for `n…
CSSFrancis Nov 5, 2022
23160b4
NewFeature: Replaced reading seq file with generic reader.
CSSFrancis Nov 6, 2022
5e33806
BugFix: Fixed generic .seq reading. Added tests for distributed and n…
CSSFrancis Nov 6, 2022
ed2f1a3
BugFix: Reduced tests. No longer testing adding blank frames.
CSSFrancis Nov 6, 2022
db66fb2
BugFix: Bad Pixel map now saved with metadata.
CSSFrancis Nov 7, 2022
a4c02bf
Formatting: Applied `black`
CSSFrancis Nov 8, 2022
78e9746
BugFix: Changed specifications.ymal `name` and `aliases`
CSSFrancis Nov 8, 2022
b968caf
BugFix/Reformatting: used nav_shape is not None vs `=!`. Reformatted…
CSSFrancis Nov 8, 2022
55dd087
Testing: More direct file pointing.
CSSFrancis Nov 8, 2022
4ede1a2
Coverage: Increased test coverage to handle warnings and errors
CSSFrancis Nov 8, 2022
e441715
Documentation: Added documentation for .seq format
CSSFrancis Nov 8, 2022
c776afd
Formatting: Applied black (2022 version)
CSSFrancis Nov 8, 2022
6230207
Merge remote-tracking branch 'origin/add_seq' into add_seq
CSSFrancis Nov 10, 2022
ae1d882
New Feature:Added in specification for DE .seq format
CSSFrancis Aug 5, 2022
f381a9a
New Feature:Added in generic methods for reading binary data
CSSFrancis Aug 5, 2022
6cb2898
Testing: Added tests using large files from the old streampix softwar…
CSSFrancis Aug 9, 2022
5956d4f
NewFeatures: Added code for reading full seq files, better metadata h…
CSSFrancis Aug 9, 2022
dfa05f1
BugFix: Proper accounting for Buffer in Celeritas File Format
CSSFrancis Aug 11, 2022
268290e
Refactor: Created a Class for loading .seq files
CSSFrancis Aug 11, 2022
5299114
Refactor: Fixed Celeritas Reader class and implemented memmaping. In…
CSSFrancis Aug 11, 2022
9c79d58
Testing: Cleaned up testing. Small changes to code fixing errors with…
CSSFrancis Aug 11, 2022
ac076e9
Testing: Added additional tests for different file formats
CSSFrancis Aug 11, 2022
297b25c
BugFix: Fixed lazy loading
CSSFrancis Aug 12, 2022
a713456
Testing: Added testing for navshape larger than file
CSSFrancis Aug 12, 2022
3f60f0d
Documentation: Improved documentation for .seq format
CSSFrancis Aug 15, 2022
c829e1d
Formatting: Black Formatting Applied
CSSFrancis Aug 15, 2022
dfcaa83
Formatting: added rsciio header instead of hyperspy header Removed pr…
CSSFrancis Aug 15, 2022
aa46d3b
Bugfix: Fixed improper reading of axes size in celeritas
CSSFrancis Aug 15, 2022
06bed95
Bugfix: XML File is recursively read
CSSFrancis Aug 24, 2022
f1cf111
NewFeature: Added in distributed Processing
CSSFrancis Aug 29, 2022
74dbe30
Testing: Added tests for writing extra frames
CSSFrancis Aug 29, 2022
e74e1f4
Testing: Tests passing
CSSFrancis Aug 29, 2022
6c42e65
Bugfix: Added in dtype and chunks for file reading.
CSSFrancis Aug 29, 2022
60443bd
Bugfix: Fixed large chunking issue
CSSFrancis Aug 29, 2022
c4f269a
Bugfix: Fixed issue with memmap being loaded into memory prematurely
CSSFrancis Aug 29, 2022
461e1d7
Bugfix: Fixed issue scalars in metadata. Stopping values from saving …
CSSFrancis Aug 29, 2022
908e35d
NewFeature: Added a `distributed_memmap` a drop in replacement for `n…
CSSFrancis Nov 5, 2022
ac5a9bf
NewFeature: Replaced reading seq file with generic reader.
CSSFrancis Nov 6, 2022
7284ee3
BugFix: Fixed generic .seq reading. Added tests for distributed and n…
CSSFrancis Nov 6, 2022
6025b39
BugFix: Reduced tests. No longer testing adding blank frames.
CSSFrancis Nov 6, 2022
e009ad5
BugFix: Bad Pixel map now saved with metadata.
CSSFrancis Nov 7, 2022
c5a4fdd
BugFix: Changed specifications.ymal `name` and `aliases` to fit with …
CSSFrancis Nov 8, 2022
0325daa
BugFix/Reformatting: used nav_shape is not None vs `=!`. Reformatted…
CSSFrancis Nov 8, 2022
98dd0c1
Coverage: Increased test coverage to handle warnings and errors and d…
CSSFrancis Nov 8, 2022
c9a7803
Documentation: Added documentation for .seq format
CSSFrancis Nov 8, 2022
b0a174a
Formatting: Applied black (2022 version)
CSSFrancis Nov 8, 2022
17c8373
BugFix: Changed memmap to be read only with flag "r"
CSSFrancis Nov 16, 2022
d187a5a
BugFix: Fixed Hyperspy loading rsciio.de.read_file returns a list ins…
CSSFrancis Nov 16, 2022
7f25d01
Bugfixes: reduced errors with rewriting data and cleaned up some file…
CSSFrancis Dec 7, 2022
cca3704
Bugfixes: Added support for passing in chunks to memmaped datasets.
CSSFrancis Dec 7, 2022
b5a94e1
Doc Refactor: Cleaned up documentation and refactored `rsciio.de.__in…
CSSFrancis Dec 7, 2022
73c1745
Testing: Skip test if hyperspy isn't installed and improved test cove…
CSSFrancis Dec 7, 2022
a1c8506
Doc: Added to changelog for #11.
CSSFrancis Dec 7, 2022
f16dacf
BugFix: Got rid of rechunking in distributed loading
CSSFrancis Mar 26, 2023
a340725
Documentation: Improved documentation for loading .seq files
CSSFrancis Apr 6, 2023
27096f9
Merge remote-tracking branch 'origin/add_seq' into add_seq
CSSFrancis Sep 19, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions docs/supported_formats/de.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
.. _de-format:

StreamPix .SEQ ()
-----------------

The .seq file format is a binary file format developed by
:ref:`Streampix` <https://www.norpix.com/products/streampix/streampix.php>. Streampix is
a high speed digital recording software.

Due to the flexible nature of the .seq file format we specifically support
the .seq file format associated with the DirectElectron-16 and DirectElectron-Celeritas cameras.

For other cameras support may not be fully realized but if an issue is raised
:ref: `here` <https://github.com/hyperspy/rosettasciio/issues> support can be
considered.

Specifically for the support of the DirectElectron-Celeritas camera there are two
possible ways to load some file.

1: Explicitly list files
.. code-block:: python
>>> from rsciio.de import file_loader
>>>
>>> file_loader(None, top="de_Top.seq", bottom="de_Botom.seq"
>>> metadata="de_Top.metadata",gain="de.gain.mrc",
>>> dark="de.dark.mrc", xml="de.seq.Config.Metadata.xml",
>>> celeritas=True)

2: Automatically detect the files. In this case the program will automatically
look for files with the same naming structure in the same folder.
.. code-block:: python
>>> from rsciio.de import file_loader
>>>
>>> file_loader(top="de_Top.seq", celeritas=True)


All of the file loaders for the cameras also have a special `distributed` keyword which changes how the data is
loaded into memory or into a dask array. This allows for the user to use
:ref: `dask-distributed` <https://distributed.dask.org/en/stable/> as a backend rather
than using the default scheduler.
1 change: 1 addition & 0 deletions docs/supported_formats/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ big datasets is supported.
blockfile
bruker
de5
de
dens
digitalmicrograph
digitalsurf
Expand Down
2 changes: 2 additions & 0 deletions docs/supported_formats/supported_formats.rst
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@
+---------------------------------------------------------------------+-------------------------+--------+--------+--------+
| :ref:`Direct electron EMD <de5-format>` | de5 | Yes | No | Yes |
+---------------------------------------------------------------------+-------------------------+--------+--------+--------+
| :ref:`Direct electron StreamPix <de-format>` | seq | Yes | No | Yes |
+---------------------------------------------------------------------+-------------------------+--------+--------+--------+
| :ref:`DENSsolutions Impulse logfile <dens_impulse-format>` | dens, csv & log | Yes | No | No |
+---------------------------------------------------------------------+-------------------------+--------+--------+--------+
| :ref:`DENSsolutions Digiheater logfile <dens_heater-format>` | dens | Yes | No | No |
Expand Down
10 changes: 10 additions & 0 deletions rsciio/de/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
from ._api import file_reader


__all__ = [
"file_reader",
]


def __dir__():
return sorted(__all__)
Loading