Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make devalidation less sensible in some cases #53

Closed
davibicudo opened this issue Mar 16, 2021 · 3 comments
Closed

Make devalidation less sensible in some cases #53

davibicudo opened this issue Mar 16, 2021 · 3 comments

Comments

@davibicudo
Copy link
Contributor

Hi !

We've been working with synpp in a project, and it's been quite useful so far. It is basically a MVP of a data processing interface. Particularly it's caching is super helpful in which it helps avoiding re-running time-consuming data processing everytime. It is also quite simple and intuitive in comparison to other pipeline libraries, so also a pleasure to work with :)
Sometimes I note however that synpp's devalidation might be too sensible. Like:

  • If a stage is a class, then module_hash devalidates the stage even if any changes happened outside the stage. An option here would be to hash inspect.getsource(foo) whenever the stage is a class, and rename module_hash with stage_hash. This happens here often since we defined several stages as classes (avoid excess of files and is helpful to set fields in configure(), like e.g. self.foo = context.config('bar')).
  • Some config options don't break the calculations, but may be included to control output or verbosity. It would be very useful to have something like context.config('write_output', ignore_in_devalidation=True).
  • Sometimes asking synpp for a stage doesn't mean necessarily we want to re-run it, i.e. the cache would suffice. Something like synpp.run(descriptors, rerun_cached=False) would also be nice.

What do you think? I could try to implement/test here (although I must admit time is not abundant at the moment...). I'd have an idea how to start, except for the second case where I'm not sure how the hash is checked there.

Best,
Davi

@sebhoerl
Copy link
Contributor

Sure, all of these sound like sensible improvements. If you have time to send PRs, I'm happy to look at them!

@sebhoerl
Copy link
Contributor

The getsource seems interesting, maybe it also just works with modules?

@davibicudo
Copy link
Contributor Author

Implemented in #56 , and the second point is better documented in #15.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants