Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

issue with large dcd seeking #4879

Open
qlearn-code opened this issue Jan 2, 2025 · 15 comments
Open

issue with large dcd seeking #4879

qlearn-code opened this issue Jan 2, 2025 · 15 comments

Comments

@qlearn-code
Copy link

Reinstalled the MDanalysis today and run rdf calculation. Got this error message "OSError: DCD seek failed with DCD error=Normal EOF"
Exact same script was successfully run before with previous version of mdanalysis.
Shouldn't be memory issue, cause that gives a different error message.

mdanalysis 2.8.0
on Windows using Jupyter notebook

@qlearn-code
Copy link
Author

forgot to mention the dcd file is 4GB. same file was run before with the same script.

@orbeckst
Copy link
Member

orbeckst commented Jan 7, 2025

If you run

import MDAnalysis as mda
import os
u = mda.Universe(DCD)
print("size:", os.stat(DCD).st_size, "B")
print("length:", u.trajectory.n_frames)
for ts in u.trajectory:
   print(ts.frame, end=" ")
print("\nComplete")

with MDA 2.8.0 and a previous version (2.7.0, ... whichever is the last one working) then what do you get?

@qlearn-code
Copy link
Author

qlearn-code commented Jan 7, 2025 via email

@orbeckst
Copy link
Member

orbeckst commented Jan 7, 2025

@qlearn-code thanks for checking. This output indicates that your DCD is read fully (with MDA 2.8.0), so at least the basic DCD reading code does not seem to be at fault.

@orbeckst
Copy link
Member

orbeckst commented Jan 7, 2025

How are you executing the RDF analysis? Can you show the minimal code example (from setting up the Universe) that displays the failure, together with the exception traceback?

@orbeckst orbeckst added the more information needed Please reply to requests for information or the issue will be closed. label Jan 7, 2025
@qlearn-code
Copy link
Author

qlearn-code commented Jan 7, 2025 via email

@orbeckst orbeckst removed the more information needed Please reply to requests for information or the issue will be closed. label Jan 8, 2025
@orbeckst
Copy link
Member

orbeckst commented Jan 8, 2025

Can you enable the Python debugger (%pdb in ipython/jupyter) and then find out at which frame i the error occurs?

@qlearn-code
Copy link
Author

qlearn-code commented Jan 8, 2025 via email

@orbeckst
Copy link
Member

orbeckst commented Jan 8, 2025

Yes,

p i

or possibly

u
p i

You might have to learn how to use the debugger https://docs.python.org/3/library/pdb.html to be able to navigate the call stack. You can move "up" u and "down" d the last function calls and you can print p the values of variables.

@qlearn-code
Copy link
Author

qlearn-code commented Jan 8, 2025 via email

@orbeckst
Copy link
Member

This looks indeed as if the DCD reading stopped in the middle of the trajectory. I am not sure what's happening. Can you check if you can trigger the error by manually seeking to these frames, i.e.,

u = mda.Universe("psf","dcd")

u.trajectory[0]
print(u.trajectory.ts.frame)

u.trajectory[2180]
print(u.trajectory.ts.frame)

u.trajectory[2181]
print(u.trajectory.ts.frame)

u.trajectory[2182]
print(u.trajectory.ts.frame)

u.trajectory[-1]
print(u.trajectory.ts.frame)

@qlearn-code
Copy link
Author

qlearn-code commented Jan 13, 2025

The dcd reading seems to stop right at frame2181

u.trajectory[0]
print(u.trajectory.ts.frame)
0
u.trajectory[2180]
print(u.trajectory.ts.frame)
2180
u.trajectory[2181]
print(u.trajectory.ts.frame)
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
Cell In[35], line 1
----> 1 u.trajectory[2181]
      2 print(u.trajectory.ts.frame)

File C:\ProgramData\anaconda3\Lib\site-packages\MDAnalysis\coordinates\base.py:836, in ProtoReader.__getitem__(self, frame)
    834 if isinstance(frame, numbers.Integral):
    835     frame = self._apply_limits(frame)
--> 836     return self._read_frame_with_aux(frame)
    837 elif isinstance(frame, (list, np.ndarray)):
    838     if len(frame) != 0 and isinstance(frame[0], (bool, np.bool_)):
    839         # Avoid having list of bools

File C:\ProgramData\anaconda3\Lib\site-packages\MDAnalysis\coordinates\base.py:868, in ProtoReader._read_frame_with_aux(self, frame)
    866 def _read_frame_with_aux(self, frame):
    867     """Move to *frame*, updating ts with trajectory, transformations and auxiliary data."""
--> 868     ts = self._read_frame(frame)  # pylint: disable=assignment-from-no-return
    869     for aux in self.aux_list:
    870         ts = self._auxs[aux].update_ts(ts)

File C:\ProgramData\anaconda3\Lib\site-packages\MDAnalysis\coordinates\DCD.py:198, in DCDReader._read_frame(self, i)
    196 """read frame i"""
    197 self._frame = i - 1
--> 198 self._file.seek(i)
    199 return self._read_next_timestep()

File C:\ProgramData\anaconda3\Lib\site-packages\MDAnalysis\lib\formats\libdcd.pyx:400, in MDAnalysis.lib.formats.libdcd.DCDFile.seek()

OSError: DCD seek failed with DCD error=Normal EOF
u.trajectory[2182]
print(u.trajectory.ts.frame)
u.trajectory[-1]
print(u.trajectory.ts.frame)

gives same error message as 2181

@orbeckst
Copy link
Member

This looks as if there's a real bug related to DCD seek.

  • Which version of Windows are you using?
  • Which version of Python?
  • How did you install MDAnalysis?

@orbeckst orbeckst changed the title issue with large dcd issue with large dcd seeking Jan 13, 2025
@qlearn-code
Copy link
Author

Which version of Windows are you using? Windows 10.0.19045
Which version of Python? 3.12.3
How did you install MDAnalysis? Installed with anaconda prompt, using "conda install mdanalysis"

@orbeckst
Copy link
Member

I don't have an immediate solution (and I don't have a Windows machine to reproduce) so for right now, we are keeping this issue up as a bug.

(We do need to check that the issue does not also happens on Linux/macOS.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants