image format order of preferences

[eroux]

For engineering reasons, I want my iiif viewer to request the images in png (not jpg) in certain circumstances. I want to be able to indicate the desired behavior (png or jpg):

• in the image annotation of the canvas in my manifests
• in the info.json of my images (if possible, although that's less important)

and reading the spec, it seems that for v2 it's ambiguous and for v3 impossible.

This relates to:

• Couldn't force OpenSeaDragon to ask png format from IIIF tileSource openseadragon/openseadragon#1343
• Support webp when requesting images UniversalViewer/universalviewer#640
• PNG files being changed to jpg ProjectMirador/mirador#1535

Presentation API

For the presentation API (v2 and v3), it seems to me there is an ambiguity when the format property is used with a iiif image. For instance if I have something like this in V3:

  "body": {
    "id": "https://example.org/iiif/book1/res/page1.png",
    "type": "Image",
    "format": "image/png",
    "service": [
      {
        "id": "https://example.org/images/book1-page1",
        "type": "ImageService3",
        "profile": "level2"
      }
    ]
  }

It's not clear if the information .png and "format": "image/png" should have an impact on a iiif compliant viewer. In this case, is the use of the png version of the file by the iiif viewer a MAY, SHOULD, MUST or nothing at all? If this should not be used as an indication by the viewer, could there be another way to indicate the desired extension?

Image API

In the image v3 the only way to provide format indication is through the extraFormats property. But it only seems to be an indication of possible formats, not an order of preferences.

Is there a way to indicate a preferred format, or maybe an ordered array of format preferences?

If not, I propose to add the formatPreferences property, having an ordered array as its value, listing formats in descending order of preference. Not all possible formats would have to be listed. If a viewer cannot display any of the indicated formats, it can pick another one. The viewer behavior could be a SHOULD.

Perhaps the same property could be used in the presentation API too...

what do you think?

[tomcrane]

For the first question, the answer is "nothing at all".

Given this image, which happens to be a png...

{
    "id": "https://example.org/iiif/book1/res/page1.png",
    "type": "Image",
    "format": "image/png"
}

...a client can use that image as-is.

I could set up an image server and use it to provide an image service for this image (or for some unseen master image from which that png was derived). I can then attach the image service to the image, asserting in the Presentation API resource that an Image API endpoint is available...

{
    "id": "https://example.org/iiif/book1/res/page1.png",
    "type": "Image",
    "format": "image/png",
    "service": [
      {
        "id": "https://example.org/images/book1-page1",
        "type": "ImageService3",
        "profile": "level2"
      }
    ]
  }

But then once I step into the service I'm in Image API land; I can tell from the info.json that this service supports png, but nothing tells me that (for example) the image is a born-digital scatter-plot that's much better as a png than a jpeg, so in this case I should request png.

I don't believe that this concern lives in the Presentation API at all - the manifest publisher has provided a png (rather than a jpeg) with an image service; any further decisions lie in the client's handling of the Image API. But as you point out, once in the Image API there's no way to tell that for this particular image, png would be better than jpeg. The Image service must support jpeg, so it can't just not provide that format.

@irv's example (UniversalViewer/universalviewer#640) is similar, but not exactly the same, as the client isn't making any assumptions about what kind of image it is. If I can use webp, and the service supports it, then as a client I'd always ask for it in preference to jpg, I don't need the server to hint that to me for each image. The client would be configured to "prefer webp over jpg when available". But for jpg vs png, I'd need a hint from the publisher, i.e., in the image service description.

I can see the need. Another reason to provide a format hint would be if the image had alpha-transparency and didn't make sense as a JPEG.

The OpenSeadragon issue is not a spec issue, it just needs to be implemented - perhaps when someone updates the OSD IIIFTileSource for Image API 3 support?

[tomcrane]

...although if there were such a spec feature, OSD could process it in IIIFTileSource. But it still needs the ability to select format, regardless of the above.

[eroux]

I understand yes, I'll try to work on a PR for OSD. Something I'm not sure I get right though is that if we have in the image API say, "formatHints": ["png"], then we would be able to have in the manifest

{
    "id": "https://example.org/iiif/book1/res/page1.png",
    "type": "Image",
    "format": "image/png",
    "service": [
      {
        "id": "https://example.org/images/book1-page1",
        "type": "ImageService3",
        "profile": "level2",
        "formatHints": ["png"]
      }
    ]
  }

right? (I'm just thinking in terms of the viewer not having to fetch the info.json to know that...) that's probably an obvious question but I just wanted to be sure.

Also, I'm not really familiar with the workflow, is this going to be discussed in a meeting?

[tomcrane]

If that property were part of the Image API v3 then yes, you could include it in the manifest like that.

See #1217 (although it's a long story!) - a JSON-LD Scoped Context would make the property available within the service of type ImageService3.

Typical workflow (see also https://iiif.io/api/annex/notes/editors/)

Editors discuss; issue gets a "discuss" label and can be raised in a community call (tech focus).

[eroux]

Ok thanks!

[azaroth42]

To distill the thread, the request is a method by which the image API publisher can, on an image by image basis, declare that there is a preferred order of image formats, similar to q values in HTTP content negotiation. Then a client could compare it's preferences with the server preferences for which format to request. The most convincing-to-me use case is when there are transparent layers, when a PNG is essential but the default format is JPG which does not support transparency.

As the 3.0 property is extraFormats, if JPG is the preferred format, it wouldn't be at the front of the list, or indeed in the list at all, even if the list were ordered. Having a duplication of the format list seems redundant.

An alternative would be to have an extensible features list, that includes at least transparency. Then the client can decide which format that manages transparency in an acceptable way it wants to request. The implications would need to be described in the spec, or at least in a recipe, but it would avoid tying the functionality to formats but instead the features those formats can support.

[eroux]

I think the analysis in your first two paragraph is right yes.

I don't really understand the reluctance of having preferred formats (independently of their features)? For instance one could want to save bandwidth (it's our case, we pay the bandwidth) and, because in some cases png are smaller (for bitonal images for instance), we want to indicate the viewer to prefer png (or webp, or anything else that's dependent on the server). What's wrong with that?

[eroux]

To give an example, we have a few million bitonal images that we store as tiffs. When they are converted to png (example) they stay about the same size (30k), while when converted to jpg with quality 0.75 (example), the size is almost 10 times larger (223k in this case). This makes a huge difference to our users in India where the bandwidth is often very small.

[tomcrane]

Editors discussion:

Would a single preferred format meet all use cases, or would an ordered list of formats be required for some use cases?

[eroux]

well... I think in the case of a transparency use case, if the native images are webp you want the viewer to prefer webp, then png if webp is not handled in the browser, so something like

"formatHints": ["webp", "png"]

in that case would be necessary. I think having an array is generally safer for cases where some formats might not be handled by the browser...

[azaroth42]

I agree that's a theoretical use case ... but are there two implementations that actually deliver webp and it would be better to have webp than png, and there are in-use browsers that don't support it? Edge, Firefox and Chrome all support it natively.

For it to be a list, there would need to be implementations that makes use of the list-ness.

[eroux]

Safari doesn't support WebP, I believe it is considered to be an in-use browser...

(Edited) I'm not sure exactly what you mean by implementation?

[azaroth42]

Right, I left Safari off as it doesn't yet. However, that's only one part of the requirement -- there would also need to be IIIF publishing implementations that need webp and png. Otherwise we add the functionality to the specification for no reason, compared to an application specific extension. Or ... other than you, who else needs this?

[eroux]

I don't know, I just know about our system... having the formatHint feature with just one value instead of an array would already be a consequential improvement for us, and the persons who reported the 3 issues I cited in the initial description. None of them seem to need an array.

[eroux]

This might not be the right place or time to discuss this idea, but I'm thinking it would be a good idea to have a default format returned by the iiif url with no format at the end: for instance:

{identifier}/full/max/default

would return whatever default format of the server. This would pose problems with servers serving webp by default, but it might be another solution... Maybe this has already been discussed though

[irv]

extraFormats not appropriate because png could be provided by you being a level 2 implementation

i think the usecase for either transparency or low colour is compelling enough to warrant this, though. i'd implement it. i'd vote for a list purely because webp is quicker to encode than png, so i'd prefer to server that (and for bandwidth reasons, both better than jpg in the scenario under discussion)

example image endpoint if you need it that provides webp and png: https://api.bl.uk/image/beta/ark:/81055/vdc_100022589179.0x000003/info.json

[eroux]

@ogloke @Kevin-Darty any thought?

[ogloke]

I was able to force mirador to display png rather than jpg. I needed to preserve transparency in overlay layers. In the mirador.js I replaced all of the "default.jpg" with "default.png" and it works rather nicely now, transparency is preserved just as I needed.
http://trin-sites-pub.trin.cam.ac.uk/manuscripts/mirador/mirador.php?n=NQ.10.145
See the layer on p056 for example

[azaroth42]

Editors agree: Preference functionality is useful, list of formats is warranted, suggest preferredFormats as the name of the property, to mirror the construction of extraFormats. Type should be a @list as the order is important to maintain.

[eroux]

excellent, thanks a lot!

[azaroth42]

Further: the items in preferredFormats MUST be listed in either extraFormats or in the referenced compliance level document, to enable validation of the information. The property is OPTIONAL.

[zimeon]

IIIF/trc#21 approved, can move ahead with merge of #1816

[azaroth42]

Closed by #1816