Feature/stageclasses #56
Conversation
|
I was also able to use this when running arbitrary code in Jupyter. Caching worked well and devalidation as well, when the code changed (inspect.getsource seems to habe done it's job). Using "global" make it no longer robust, but may be helpful for interactive tests and caching (although IPython also has other solutions for that): |
| # 0) Construct pipeline config | ||
| pipeline_config = {} | ||
| if "processes" in config: pipeline_config["processes"] = config["processes"] | ||
| if "progress_interval" in config: pipeline_config["progress_interval"] = config["progress_interval"] | ||
|
|
||
| if not working_directory is None: | ||
| if not os.path.isdir(working_directory): | ||
| raise PipelineError("Working directory does not exist: %s" % working_directory) | ||
| logger.warning("Working directory does not exist, it will be created: %s" % working_directory) | ||
| os.mkdir(working_directory) |
There was a problem hiding this comment.
I see the point, but so far I was deliberately avoiding this, to make sure that the user is 100% aware where all the caching files will go. Maybe we can add a config flag, which is usually "false" like "auto_generate_working_directory" ?
| # 4.1) Devalidate if they are required | ||
| stale_hashes = set(required_hashes) | ||
| # 4.1) Devalidate if they are required (optional, otherwise will reload from cache) | ||
| if rerun_required: |
There was a problem hiding this comment.
Is this covered by a unit test ? If not, can you add one ?
| return run(definitions, self.config, self.working_directory, flowchart_path=flowchart_path, | ||
| dryrun=dryrun, verbose=verbose, logger=self.logger, rerun_required=rerun_required) | ||
|
|
||
| def run_single(self, descriptor, config={}, rerun_if_cached=False, dryrun=False, verbose=False): |
There was a problem hiding this comment.
Really nice! If it had a unit test, would be even better :)
|
Really nice, I like the jupyter integration! It would be nice if you could add some unit tests so this is robust for the future. Thanks! |
|
Sounds good, I'll increment the PR with a couple tests then. |
|
Please update the |
update CHANGELOG
This PR adresses issues #52 and #53. It doesn't cover the issues entirely however.
@sebhoerl let me know what you think, perhaps you have some other ideas in mind on how to solve the same issues.