New Image API property: preferredFormats

[mikeapp]

Original Issue
IIIF/api#1786

Pull Request
IIIF/api#1816

Preview
https://preview.iiif.io/api/1786_image_preferredFormats_prop/api/image/3.0/#55-preferred-formats

Summary
The publisher of an image may have one or more preferred formats that they would like to encourage clients to use. The reasons for this preference may be aesthetic or technical. Examples given include:

• line drawings compressed to 20k as png vs 220k as jpg
• webp is faster to encode than png
• formats that support transparency (gif, png) might be preferred

Proposed Solution
Add a preferredFormats property to the Image Information Document (info.json). This will be an array of format parameters (e.g., "preferredFormats": [ “png”, “gif” ]) indicating the publisher’s preferred formats for the image.

[cubap]

Honestly, I'm probably never checking this, but I'm certainly not against a publisher offering the information.

[scossu]

What is the reason for the down vote?

Other than that, I'm not against the feature but I am doubtful about the utility in a real-world scenario, where the "preferred format" would either be applied wholesale to a collection or need some complex image analysis to determine whether a PNG is better than a JPG (the 20K vs. 220K example seems an edge case of born-digital graphics). I'm a bit concerned about feature clutter.

[tomcrane]

(see also the discussion on IIIF/api#1838)

As IIIF features go, this one would have ~1 billion examples if @irv implements it, and is already implemented in OpenSeadragon - which means that...

I'm probably never checking this

...in the context of a deep zoom viewer at least, you probably never have to, as a lower level component will take care of it unless you intervene.

This is how the British Library ensures that most visitors are making webp image requests rather than jpegs.

[cubap]

@tomcrane That's right—you've read it how I meant it. I think this will be something used deeper in the machinery than many of the other properties. I'm not a downvote and I'm not abstaining because I am sure it will be used on resources which I use.

[beaudet]

I'm struggling a little with a separation of duties concern with this request. I'm wondering if there are other cases where a IIIF Image Server is responsible for knowing anything about how an image is being used in a rendering. This almost 100% ensures that implementations wishing to use this feature must implement a proxy to intercept / process the request, either with a plugin to an image server or an actual proxied request, and that a mechanism needs to be created for each image server implementing this feature to receive from the proxy the list of preferred formats and to then render it accordingly. We're doing something along those lines for other reasons, so it's not a huge deal, but it certainly does complicate implementations.

I'm not sure I understand the rationale for why this can't live in the presentation API. Are there not use cases for displaying images without their transparency mask as well as with it? If one of the prime motivations for this is to support transparencies in images, it seems like that warrants a larger discussion about support for transparencies. As for the original request, it seems possible to implement the performance benefits with a simple redirect of jpg->png for the relevant images.

I'm with @mixterj on this one just because I think it warrants further discussion, although I admit not having listened to the recording of the call yet so maybe I will change my mind after doing so.

[irv]

i think this is in the same class as sizes; it's a suggestion, not a requirement that the the client respects it.if you want to request jpg you can still have it

[beaudet]

Yes, but size is a dimension easily measured regardless of the source and target image formats whereas preferredFormats is a curated property. It's difficult (if not impossible?) for an image server to determine whether an image should have a preferred format of PNG rather than JPG. That means it must be derived from some other piece of data that is likely not normally available in the source image itself and almost certainly doesn't conform to a standard. Thus the requirement to measure it with a proxied request and then instruct the image server which value to use. That doesn't generally happen with size unless a custom solution already exists to handle things like rights restrictions.

[irv]

2 different cases i think. an image server can tell if there's an alpha channel and therefore recommend the caller use png or webp, whereas in the other case yes it would be a blanket setting (however you implemented it)

[beaudet]

Yes, it does seem to be two cases.

For the alpha channel case, why not provide robust image metadata in the info.json response that is image standards scoped and let the client determine an appropriate format based on all available data and the context in which the image is being displayed and on the underlying client's support or lack of support for particular formats, e.g. Safari w/ WEBP. Color channels and color depth come to mind as natural starting points for that metadata. I don't think an image server should be put in a position to determine a preferredFormat in this case because it doesn't necessarily have all the information it needs to make that determination and it allows client responsibilities to bleed into the server space in my opinion.

In cases where PNG is preferable to JPEG due to the subject of the image, e.g. an architectural blueprint, something like preferredFormats would be necessary for the client to know which format to request to ensure the best possible reproduction. However, the image server still won't actually determine that, it will just report it back in the info.json response, either as a hard-coded configuration short-cut or by a proxy sitting in front of it.

I think preferredFormats would have a more natural home in the manifest and it might be beneficial to break down the reasons why certain formats are preferred and convey those reasons in the metadata rather than an actual format recommendation.

[eroux]

I'm not opposed to having the preferred format set in the manifest as long as there is a standard way to:

• save bandwidth by suggesting the client to use png or webp in some cases (not all)
• suggest the client to use webp / png when there's an alpha channel

I think the idea of indicating the native image mode (bitonal / grayscale / bit depth / number of channels, etc.) is a good idea.

But I have doubts that the client is the best place to have the logic of format selection. The server is the only one to know:

• what library it's using and how efficient they are
• some information about the original images

Here's a use case that would be problematic for us if the client is to pick the format: when users request the full image in the same format as the original (something like /full/max/0/default.jpg in the case where the original is a jpg), we just serve the original image bit directly without converting it to an image object. It speeds up things a little bit and saves some processing time. Now, if you indicate to the client that you can serve webp and jpg, then the client might choose webp because of its own internal logic. But in that case webp will be slower, and the client cannot know that. So in this kind of cases the server knows things and needs to make decisions. Now you could think of something like:

• "originalformat": "jpg"
• "preferOriginalWhen": ["jpg"]

but that looks like a slippery slope to me where each specific optimization of each iiif server instance will need to be encoded, and the logic of it integrated in the client, which doesn't seem like a good direction to me...

I much prefer the idea where the client is given a list of formats that the server likes and picks the first it supports. It's simple for both the client and the server and works in all cases.

[eroux]

(oh, and if you're concerned about real-world use case, we would apply it to the ~17M images of BDRC's collection. In fact we already do use the property in our info.json and use a patch in mirador to handle it)

[jpstroop]

@eroux Thanks for getting in touch. This raises some interesting ideas.

• First: "originalformat" is a no-go because the Image API very intentionally avoids the notion of an 'original' or source image: images could be generated dynamically on the fly, or composed of several parts. All of the properties and parameters in info.json are, semantically speaking, constraints of the service, and not any particular source material.
• Servers should probably always optimize for default.jpg to whatever extent they can as it's the level 0 assumption that all servers support this (and furthermore, I believe that OpenSeadragon and Leaflet at least default to default.jpg).
• It's the alpha channel use case that seems to be coming up over and over is most compelling. I wonder if we should refine to a property that is more specific to that, e.g. alphaChannelFormats : [ webp, png ] (in order of preference)? I feel like that's something we could even do at level 0 if we wanted, which is probably the only way in which it would really be useful (not sure what compliance level we were thinking for preferredFormats).

[eroux]

thanks for the insight! I agree the alpha channel is interesting, but we have a few million bitonal scans that we make available (example of a Tibetan Canonical collection). Part of our mission is to make these texts available to Tibetans, even in areas where bandwidth is small. So we simply cannot use jpg in these cases, it's an absolute no-go. There is no way to optimize for default.jpg because of the limitations of the jpg format (no bitonal mode).

I understand this is not the most common use case in the iiif world, but it's ours... I don't have the exact number of bitonal images we have, but it's probably around 4 to 5 millions.

[jpstroop]

Compression is an ages-old (in the IIIF universe) topic: IIIF/api#294. Curious.

So bitonal pngs are your preferred format, correct? And bitonal pngs (which must essentially be bitmaps?) are smaller than bitonal (still 8-bit, so I guess technically grayscale) jpgs? I guess I hadn't thought of that but it makes sense. It's an interesting use case.

Does your default return a bitonal image? Apologies if I missed this. In your case, would you prefer to not support jpg at all? I don't think that's something we could pull off supporting in the spec, just curious.

I wonder what the other @IIIF/editors think. Maybe we should require preferredFormats at level 0 for this?

[eroux]

(I'll pretend that our system works as I expected despite a bug I just noticed) Yes the preferred format for these bitonal images is png.

To take a real life example,

http://iiif.bdrc.io/bdr:V22084_I0887::08870003.tif/full/max/0/default.png

is bitonal (5kB), the corresponding jpg:

http://iiif.bdrc.io/bdr:V22084_I0887::08870003.tif/full/max/0/default.jpg

is grayscale (46kB). I'm fine supporting jpg, it's not an issue... as long as I can make sure clients prefer png.

I'm not entirely sure what you meant by "Does your default return a bitonal image?", did I answer your question?

Thanks!

[jpstroop]

I'm not entirely sure what you meant by "Does your default return a bitonal image?", did I answer your question?

Yes, you did. I wasn't sure if you had color available (and default returned it), or if you were expecting clients to explicitly request bitonal as well as png.

At the end of your first post you said:

I much prefer the idea where the client is given a list of formats that the server likes and picks the first it supports. It's simple for both the client and the server and works in all cases.

Isn't that what preferredFormats, as proposed, would do?

[eroux]

Sure, I'm very much in favor of preferredFormats, my remark was aimed at the idea proposed in this thread to allow the client to decide according to more complete information provided in the info.json (edited: actually it might be on another thread, sorry for mixing things up!).

Thanks!

[beaudet]

There are several suggestions going on here and I think (now), 3 distinct use cases. I'm not sure the same solution is ideal for addressing each of them:

1. handling alpha channels when they exist
2. conveying that one format is preferable to another for a particular image for image "quality" reasons (not quality in the IIIF sense)
3. indicating that one format is preferable to another due to significant size differences irrespective of any potential quality differences

I maintain that image quality comparisons of one image v.s. another are out of scope to a IIIF image server. With a slight performance penalty, a IIIF server could, however, encode a medium sized version of an image on the backend in all available formats and report the ratio of those sizes back to a client. Rendering performance could also be measured and reported back at the same time. It could also detect the presence of all color channels and color depth and report that. So, it seems like (1) and (3) could be handled relatively easily by a IIIF server. (2) seems to me to be a curation issue associated with particular collections if not individual images within a collection and is a subjective measure. It seems more appropriate to locate preferredFormat in a manifest while adding a few data elements to info.json to equip a client with the information it needs to make a better decision about which image format to request.

[eroux]

I won't make a proposal for all the properties needed in the info.json for the client to decide what format to take. So what would be the next steps if it's decided things should go in that direction?

[beaudet]

@hadro - How does the TRC process accommodate (or not) discussions that take place here in GitHub after the call where the issue being voted on was introduced but prior to the finalization of voting?

[hadro]

@beaudet I don't have a firm answer for you, but let me raise it with TRC ex officio folks. We've had clarifying conversations in TRC issue threads before, but this seems a bit different/more extensive. Stay tuned

[aisaac]

I'm abstaining here. I see the case and I think the method proposed is alright. But this is beginning to err on the side of content negotiation. And I'm not sure it's good for a web standard to begin replicate what content negotiation does, by other means. Sure, one could say this argument applies to the very fact of offering format as a parameter of the Image API, but so far it didn't look too much like content negotiation. So I'm genuinely puzzled.

[azaroth42]

Issue 21 (New Image API property: preferredFormats)

+1: 24 [Siani81 ahankinson andrewgunther awead azaroth42 cubap gigamorph glenrobson hadro irv jbhoward-dublin joshuago78 jronallo jtweed julsraemy jwd mattmcgrattan mcwhitaker mejackreed mikeapp regisrob tomcrane tpendragon zimeon]
0: 2 [aisaac scossu]
-1: 2 [beaudet mixterj]

Result: 24 / 28 = 0.86

Super majority is in favor, issue is approved