This repository is the home of application code used by NPG to manage data and metadata in WSI iRODS.
It includes:
-
iRODS CLI utilities
- Metadata verification and repair.
- Checksum verification and repair.
- Replica verification and repair.
- Safe bulk copy.
- Safe bulk deletion.
-
General purpose API
- Managing standard WSI iRODS metadata.
-
Analysis platform-specific API and CLI utilities
- Managing Oxford Nanopore metadata and permissions.
See Building and testing.
The easiest way to get the CLI scripts is to use the pre-built Docker image (the image includes the necessary iRODS clients):
# Latest release
docker pull ghcr.io/wtsi-npg/npg-irods-python:latest
# Specific version
docker pull ghcr.io/wtsi-npg/npg-irods-python:1.1.0
The tests require some services to be running (iRODS and possibly MySQL). See the
docker-compose.yml
file for an example of how to achieve this using our Docker images.
pip install -r requirements.txt
pip install -r test-requirements.txt
pytest --it
Most of the scripts have a CLI option --json
to enable structured logging in JSON.
This is preferred when the scripts are run as a service, particularly when forwarding
logs to an aggregator (such as ELK) because it allows more effective filtering than
unstructured messages.
This package uses the standard Python logging library to deliver log messages. When a
script has the option --log-config
, the user can specify a configuration file
to modify logging behaviour e.g. to set log levels and add new log destinations.
The configuration file must be JSON, in the form of a standard logging configuration dictionary.
An example configuration is provided in the file logging.json
:
{
"version": 1,
"disable_existing_loggers": false,
"formatters": {
"stderr": {
"format": "%(message)s"
},
"syslog": {
"format": "%(message)s"
}
},
"handlers": {
"stderr": {
"class": "logging.StreamHandler",
"level": "INFO",
"formatter": "stderr",
"stream": "ext://sys.stderr"
},
"syslog": {
"class": "logging.handlers.SysLogHandler",
"level": "ERROR",
"formatter": "syslog",
"address": "/dev/log"
}
},
"root": {
"level": "ERROR",
"handlers": [
"stderr",
"syslog"
]
}
}
In the stderr
handler, the level
option refers to the starting priority to consider.
Its priority of INFO means that it will log all messages to STDERR starting from INFO,
including WARNING, ERROR and FATAL. Whereas the syslog
handler will log ERROR and
FATAL to the syslog.
A formatter can be specified for each handler with different variable. However, as we
rely on structlog
to pre-format the messages, we simply forward the pre-formatted
string.