Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Overwriting files that were lazily loaded results in permission error… #241

Open
wants to merge 12 commits into
base: main
Choose a base branch
from

Conversation

roeldegoede
Copy link
Collaborator

…s. Loading the data first solves that problem.

Issue addressed

When reading in a sfincs model with netcdf files (as grid (future)), subgrid or as forcing), these datasets are lazily loaded. In theory, that increases performance, because data is only loaded when necesssary. However, due to lazy references to the data in the original files, the files cannot be overwritten when you write the model.

Of course, you could choose to only write the files that you changed, but often the SfincsModel.write() is used.

Explanation

Everywhere where we would like to write a netcdf, we first check if the file already exists. If that is the case, we have to load the data into memory first, such that we can overwrite the file after (after loading, the lazy references are gone).

Checklist

  • [x ] Updated tests or added new tests
  • [x ] Branch is up to date with main
  • Updated documentation if needed
  • Updated changelog.rst if needed

Additional Notes (optional)

Question is whether we even want to load and overwrite datasets that are still lazy, because in most cases that will mean that they are unchanged and loading/writing would result in unnecessary computations?

…s. Loading the data first solves that problem.
@roeldegoede roeldegoede marked this pull request as draft January 29, 2025 09:08
@roeldegoede
Copy link
Collaborator Author

Somehow, the tests that I implemented to guarantee this would work (which it perfectly did locally) fail online with he same permission errors as before. However, since everything works on my side, it's very hard to debug.. @DirkEilander or @LuukBlom any suggestions are welcome.

@roeldegoede roeldegoede marked this pull request as ready for review March 6, 2025 14:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants