You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We've been working with synpp in a project, and it's been quite useful so far. It is basically a MVP of a data processing interface. Particularly it's caching is super helpful in which it helps avoiding re-running time-consuming data processing everytime. It is also quite simple and intuitive in comparison to other pipeline libraries, so also a pleasure to work with :)
Sometimes I note however that synpp's devalidation might be too sensible. Like:
If a stage is a class, then module_hash devalidates the stage even if any changes happened outside the stage. An option here would be to hash inspect.getsource(foo) whenever the stage is a class, and rename module_hash with stage_hash. This happens here often since we defined several stages as classes (avoid excess of files and is helpful to set fields in configure(), like e.g. self.foo = context.config('bar')).
Some config options don't break the calculations, but may be included to control output or verbosity. It would be very useful to have something like context.config('write_output', ignore_in_devalidation=True).
Sometimes asking synpp for a stage doesn't mean necessarily we want to re-run it, i.e. the cache would suffice. Something like synpp.run(descriptors, rerun_cached=False) would also be nice.
What do you think? I could try to implement/test here (although I must admit time is not abundant at the moment...). I'd have an idea how to start, except for the second case where I'm not sure how the hash is checked there.
Best,
Davi
The text was updated successfully, but these errors were encountered:
Hi !
We've been working with synpp in a project, and it's been quite useful so far. It is basically a MVP of a data processing interface. Particularly it's caching is super helpful in which it helps avoiding re-running time-consuming data processing everytime. It is also quite simple and intuitive in comparison to other pipeline libraries, so also a pleasure to work with :)
Sometimes I note however that synpp's devalidation might be too sensible. Like:
What do you think? I could try to implement/test here (although I must admit time is not abundant at the moment...). I'd have an idea how to start, except for the second case where I'm not sure how the hash is checked there.
Best,
Davi
The text was updated successfully, but these errors were encountered: