Utils for https://github.com/emerald-geomodelling/poltergust and https://github.com/spotify/luigi
poltergust_luigi_utils.caching.CachingOpenerTarget
like luigi.contrib.opener.OpenerTarget
but caches the file locally (forever, cleaning the cache directory must be done by a cron job).
poltergust_luigi_utils.gcs_opener
adds support for Google Cloud Storage targets (gs://bucket/file
style url:s) to luigi.contrib.opener.OpenerTarget
poltergust_luigi_utils.logging_task.LoggingTask
mixin class for luigi.Task
that integrates with the python logging framework to log task activity to a luigi target.
poltergust_luigi_utils.luigi_client.LuigiTaskClient
client to check status and progress of a task with the central scheduler.
poltergust_luigi_utils.poltergust_pipeline_client.PoltergustPipelineClient
client library to submit tasks to Poltergust (by writing the config files) and to retrieve status, progress and logs from tasks.