Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When using use_tmp_file or staging_dir, move/rename files as soon as they are finished #129

Open
gerritholl opened this issue Dec 27, 2021 · 3 comments

Comments

@gerritholl
Copy link
Member

Feature Request

Is your feature request related to a problem? Please describe.

I'm using use_tmp_file or staging_dir to prevent files being shipped out by our file distribution system prematurely. Currently, trollflow2 stores all files to temporary files, then renames/moves them when they are all finished. It would be desirable to move/rename each file as soon as it's finished, so that products are delivered to users sooner.

Describe the solution you'd like

I would like that (optionally?) files are renamed to their final name as soon as they're finished.

Describe any changes to existing user workflow

None if the behaviour is optional. But I think I'm the only user anyway.

Additional context

Make the production so fast that it doesn't matter ;-)

@gerritholl
Copy link
Member Author

This may be easier said than done. The save_datasets plugins first collects all the dask graphs, then computes them all simultaneously. Maybe moving the file could be added as a task to the dask graph in some way?

@gerritholl
Copy link
Member Author

May this also be relevant for the FilePublisher? I think that when using FilePublisher, it also only published when all products have been produced, losing the opportunity to ship out files as soon as they are finished?

@gerritholl
Copy link
Member Author

Maybe #138 can be of help here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant