Skip to content

emerald-geomodelling/poltergust-luigi-utils

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

25 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Luigi utils

Utils for https://github.com/emerald-geomodelling/poltergust and https://github.com/spotify/luigi

Filesystem access

poltergust_luigi_utils.caching.CachingOpenerTarget like luigi.contrib.opener.OpenerTarget but caches the file locally (forever, cleaning the cache directory must be done by a cron job).

poltergust_luigi_utils.gcs_opener adds support for Google Cloud Storage targets (gs://bucket/file style url:s) to luigi.contrib.opener.OpenerTarget

Logging

poltergust_luigi_utils.logging_task.LoggingTask mixin class for luigi.Task that integrates with the python logging framework to log task activity to a luigi target.

Scheduler communication

poltergust_luigi_utils.luigi_client.LuigiTaskClient client to check status and progress of a task with the central scheduler.

Poltergust client

poltergust_luigi_utils.poltergust_pipeline_client.PoltergustPipelineClient client library to submit tasks to Poltergust (by writing the config files) and to retrieve status, progress and logs from tasks.

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages