Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add batchset for rendering a hierarchy of containers (single file) #52

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

Add batchset for rendering a hierarchy of containers (single file) #52

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8f55263
Single renderdef file for multiple named projects/datasets
manics Dec 16, 2020
d3ba8c9
Add jsonschema for validating schemas
manics Jan 28, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
137 changes: 130 additions & 7 deletions src/omero_cli_render.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,51 @@
# be updated individually.
"""

BATCHSET_HELP = """Set rendering settings from a hierachy of rendering
definitions corresponding to an OMERO container hierarchy.

'projects' and 'datasets' are supported as top level containers, 'name'
must be provided.
If multiple containers have the same name they will all be processed.
Top level keys beginning with '_' are ignored, so they could for example
be used as YAML anchors.

Definitions can be applied to projects, datasets, or datasets in projects.

'renderdef' is a rendering definition as passed to 'set', except that
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new hierarchy makes sense and the renderdef key has the advantage of containerizing the render metadata. Looking at the examples, I was wondering whether a scheme where each node would be either a containers key (projects, datasets, ...) OR directly a render element had been tested/would be an option e.g.

projects:
  - name: Project
    channels:
    ...
  - name: Project B
    channels:
    ...
projects:
  datasets:
    - name: Dataset A
      channels:
      ...
    - name: Dataset B
      channels:
      ...

Copy link
Member Author

@manics manics Jan 25, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think keeping the renderdef self-contained will make this spec easier to update, otherwise when adding a key to either the container definition or the renderdef spec you'll need to consider future conflicts or potential confusion.

In addition keeping the renderdef under it's own key means this could potentially be extended to handle other metadata e.g. for setting tags or other annotations.

'version' is omitted.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this because the version is considered as global? Is it defined at the bottom of the YML file?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought it would simplify things if we avoided mixed versioning though that's probably not a major problem to support. You're right that it should be specified somewhere though, so the question is should we require the version

  • as part of every renderdef
  • as part of the top-level version for this file i.e. version indicates the version of the renderdef spec + the version of this hierarchical definition
  • in a dedicated top-level renderdef_version key
    And if renderdef has it's own version indicator how do we version this hierarchy spec?


Example:

projects:
# Apply renderdef to all Images in Project A
- name: Project A
renderdef:
channels:
...

# Apply renderdef to all Images in Dataset 1 in Project B
- name: Project B
datasets:
- name: Dataset 1
renderdef:
channels:
...

datasets:
# Apply the channel_group1 renderdef anchor to all Images in Dataset 2
- name: Dataset 2
renderdef:
channels:
*channel_group1

_anchors: &channel_group1
1:
active: true
label: Plk1

"""

TEST_HELP = """Test that underlying pixel data is available

The syntax for specifying objects is: <object>:<id>
Expand Down Expand Up @@ -372,6 +417,7 @@ def _configure(self, parser):
info = parser.add(sub, self.info, INFO_HELP)
copy = parser.add(sub, self.copy, COPY_HELP)
set_cmd = parser.add(sub, self.set, SET_HELP)
batchset = parser.add(sub, self.batchset, BATCHSET_HELP)
edit = parser.add(sub, self.edit, EDIT_HELP)
test = parser.add(sub, self.test, TEST_HELP)

Expand All @@ -386,7 +432,10 @@ def _configure(self, parser):
x.add_argument("object", type=render_type, help=tgt_help,
nargs="+")

for x in (copy, set_cmd, edit):
batchset.add_argument("path", help=(
"Directory containing subdirectories with renderdefs."))

for x in (copy, set_cmd, edit, batchset):
x.add_argument(
"--skipthumbs", help="Do not regenerate thumbnails "
"immediately", action="store_true")
Expand All @@ -400,13 +449,14 @@ def _configure(self, parser):
copy.add_argument("target", type=render_type, help=tgt_help,
nargs="+")

for x in (set_cmd, edit):
for x in (set_cmd, edit, batchset):
x.add_argument(
"--disable", help="Disable non specified channels ",
action="store_true")
x.add_argument(
"--ignore-errors", help="Do not error on mismatching"
" rendering settings", action="store_true")
for x in (set_cmd, edit):
x.add_argument(
"channels",
help="Local file or OriginalFile:ID which specifies the "
Expand All @@ -430,6 +480,11 @@ def _lookup(self, gateway, type, oid):
self.ctx.die(110, "No such %s: %s" % (type, oid))
return obj

def _lookup_name(self, gateway, type, name):
gateway.SERVICE_OPTS.setOmeroGroup('-1')
objs = gateway.getObjects(type, attributes={'name': name})
return list(objs)

def render_images(self, gateway, object, batch=100):
"""
Get the images.
Expand Down Expand Up @@ -667,21 +722,89 @@ def set(self, args):
""" Implements the 'set' command """
data = self._load_rendering_settings(
args.channels, session=self.client.getSession())
self._set(data, args.object,
args.ignore_errors, args.disable, args.skipthumbs)

def _get_batchset_targets(self, otype, descriptor):
""" Gets targets and renderdefs for the 'batchset' command """
def require_one(required, keys):
if sum((r in keys) for r in required) != 1:
raise ValueError(
'Exactly one of {} required'.format(required))

descriptor_keys = list(descriptor.keys())

require_one(['name'], descriptor_keys)
if otype == 'Project':
require_one(['datasets', 'renderdef'], descriptor_keys)
else:
require_one(['renderdef'], descriptor_keys)

parents = self._lookup_name(self.gateway, otype, descriptor['name'])
if not parents:
raise Exception(f"No match for Project: {descriptor['name']}")
for parent in parents:
if 'renderdef' in descriptor_keys:
yield parent, descriptor['renderdef']
else:
dataset_renderdefs = dict(
(d['name'], d) for d in descriptor['datasets'])
expected_names = set(dataset_renderdefs.keys())
for dataset in parent.listChildren():
if dataset.name in dataset_renderdefs:
for target in self._get_batchset_targets(
'Dataset', dataset_renderdefs[dataset.name]):
yield target
try:
expected_names.remove(dataset.name)
except KeyError:
# Multiple containers have the same name
pass
if expected_names:
raise Exception(
'No match for datasets: {}'.format(expected_names))

@gateway_required
def batchset(self, args):
""" Implements the 'batchset' command """
try:
batch_data = pydict_text_io.load(
args.path, session=self.client.getSession())
except Exception as e:
self.ctx.dbg(e)
self.ctx.die(103, "Could not read %s" % args.path)
for key, containers in batch_data.items():
if key.startswith('_'):
continue
if key not in {'projects', 'datasets'}:
raise NotImplementedError(
'Invalid batchset container: {}'.format(key))
for container in containers:
for c, renderdef in self._get_batchset_targets(
f'{key[0].upper()}{key[1:-1]}', container):
self.ctx.out('Applying settings to '
f'{c.OMERO_CLASS}:{c.id} {c.name}')
self._set(
renderdef, c._obj,
args.ignore_errors, args.disable, args.skipthumbs)

def _set(self, data, target, ignore_errors, disable, skipthumbs):
""" Utility method to apply rendering settings """
(namedict, cindices, rangelist, colourlist, minmaxlist) = \
self._read_channels(data)
greyscale = data.get('greyscale', None)
if greyscale is not None:
self.ctx.dbg('greyscale=%s' % greyscale)

iids = []
for img in self.render_images(self.gateway, args.object, batch=1):
for img in self.render_images(self.gateway, target, batch=1):
iids.append(img.id)

(def_z, def_t) = self._read_default_planes(
img, data, ignore_errors=args.ignore_errors)
img, data, ignore_errors=ignore_errors)

active_channels = []
if not args.disable:
if not disable:
# Calling set_active_channels will disable channels which
# are not specified.
# Need to reset ALL active channels after set_active_channels()
Expand Down Expand Up @@ -730,7 +853,7 @@ def set(self, args):
img.saveDefaults()
self.ctx.dbg(
"Updated rendering settings for Image:%s" % img.id)
if not args.skipthumbs:
if not skipthumbs:
self._generate_thumbs([img])
except Exception as e:
self.ctx.err('ERROR: %s' % e)
Expand All @@ -739,7 +862,7 @@ def set(self, args):

if not iids:
self.ctx.die(113, "ERROR: No images found for %s %d" %
(args.object.__class__.__name__, args.object.id._val))
(target.__class__.__name__, target.id._val))

if namedict:
self._update_channel_names(self.gateway, iids, namedict)
Expand Down