Allow path-only datasets in manifest schema
Summary
Allow a manifest's datasets to be specified as paths, allowing metadata to be gathered dynamically from these datasets instead of hard-coded in a manifest serialisation.
Contents (#101)
IMPORTANT: There are 2 breaking changes.
Enhancements
- BREAKING CHANGE: Make
Twine.available_strands
a set instead of a tuple - Update manifest schema to allow path-only datasets
- Raise error if a dataset specified in a manifest strand of the twine is missing from the corresponding manifest
- Add
Twine.available_manifest_strands
property
Refactoring
- BREAKING CHANGE: Move dataset files tag checking into
octue-sdk-python
atoctue.runner.Runner._validate_dataset_file_tags
Refactoring
- Move manifest and twine migrations into
migrations
subpackage