Plugin task sublcasses #59
Conversation
- add a Plugin metaclass to register Task subclasses - make core a folder - add core._tasks folder to host all plugin tasks - split core.py into graph_items.py and workflow.py to solve cyclic import issues
| __all__ = [ | ||
| p.name[:-3] for p in Path(__file__).parent.glob("*.py") if p.is_file() and not p.name.endswith("__init__.py") | ||
| ] |
There was a problem hiding this comment.
I wouldn't do this. Shouldn't be too much effort to just write the actual list here, and update it when needed. (as written in the comment above)
| command_option: str | None = None | ||
| input_arg_options: dict[str, str] | None = None |
There was a problem hiding this comment.
I don't find this nomenclature very intuitive. I'd vouch for (in YAML format):
cli_arguments:
positional: [data1, data2]
keyword:
--input: obs_data
flags:
- --restartIn addition, having a source_file key that allows sourcing a file to load environment variables, should cover the vast majority of executables that anyone would ever want to call, e.g. preproc, postproc, etc.
There was a problem hiding this comment.
Yes, that was the conclusion from the in person meeting
| space = ({} if date is None else {"date": [date]}) | {k: parameters[k] for k in param_refs} | ||
| yield from (dict(zip(space.keys(), x)) for x in product(*space.values())) | ||
|
|
||
| class Plugin(type): |
There was a problem hiding this comment.
When I hear Plugin, I typically think of AiiDA plugins, such as aiida-icon, but I see the wording here refers to specific derived classes from core Tasks ^^
Actually, just learned about this use case of metaclasses to register specific subclasses, interesting! Not sure how many plugins we will have an actual need (and time and effort) for, to be discussed tomorrow.
| ) -> Task: | ||
| inputs = list( | ||
| chain(*(datastore.iter_from_cycle_spec(input_spec, coordinates) for input_spec in graph_spec.inputs)) | ||
| ) | ||
| outputs = [datastore[output_spec.name, coordinates] for output_spec in graph_spec.outputs] | ||
| # use the fact that pydantic models can be turned into dicts easily | ||
| cls_config = dict(config) | ||
| del cls_config["parameters"] | ||
| del cls_config["plugin"] | ||
| new = Plugin.classes[config.plugin]( | ||
| coordinates=coordinates, | ||
| inputs=inputs, | ||
| outputs=outputs, | ||
| **cls_config, | ||
| ) # this works because dataclass has generated this init for us |
There was a problem hiding this comment.
Not sure I fully understand the changes here. Possibly the change of behavior of the from_config method could be a separate PR?
| @@ -186,7 +171,7 @@ def iter_from_cycle_spec(self, spec: TargetNodesBaseModel, reference: dict) -> I | |||
| if "date" not in self._dims and (spec.lag or spec.date): | |||
| msg = f"Array {self._name} has no date dimension, cannot be referenced by dates" | |||
| raise ValueError(msg) | |||
| if "date" in self._dims and reference.get("date") is None and spec.date is []: | |||
| if "date" in self._dims and reference.get("date") is None and spec.date == []: | |||
There was a problem hiding this comment.
| if "date" in self._dims and reference.get("date") is None and spec.date == []: | |
| if "date" in self._dims and reference.get("date") is None and len(spec.date) == 0: |
?
| @@ -61,27 +61,25 @@ tasks: | |||
| host: santis | |||
| account: g110 | |||
| - extpar: | |||
| plugin: extpar | |||
| plugin: shell # no extpar plugin available yet | |||
There was a problem hiding this comment.
If we are talking about AiiDA plugins, writing an aiida-extpar plugin will be too much work, I think, for some time to come. I expect us to handle other executables with aiida-shell in the same manner for quite a while, as aiida-shell can be used to run any executable (not just shell scripts), and we have enough work here and in aiida-icon ^^
Host plugin specific behavior/logic in subclasses of
core.Task