ExternalJobs, DictEntryLoadingJobs and fork safety
There is a fancy documentation now: https://tyberiusprime.github.io/pypipegraph2/
This release brings three new jobs:
DictEntryLoadingJob and CachedDictEntryLoadingJob),
which are essentially 'reskins' of AttributeLoadingJobs that store in ['key'] instead of .key.
and ExternalJob which is used to run external CLI programs.
This release also makes numba use TBB (if numba can be imported), which is supposedly thread and more importantly fork safe. In addition, all forking now happens on the main thread, which should help with overall fork safety.
(Unfortunately, forking is never safe. If you hold locks in different threads across forks, you're going to deadlock.
But then, forking is the only way to get decent multi-core out of python, when spawn() + importing python modules runs in the second-range).
There's also a fix to job-deduplication which was handing out too many 'new' objects.
A CLI to hash files just like the ppg2 was added.
A signal storm job when a well connected upstream job failed has been fixed.
There is an experimental 'signal when a job was done' pipe included that can be used by pypipegraph2-interactive.
(pandas) DataFrame hashing will now work correctly, even if DeepDiff is older than version 8.0
The history is now (transparently) converted to zstd, making it much smaller on disk.
Minimum Python version has increased to 3.9.
History dumping can no longer be aborted by KeyboardInterrupt.