API proposal for color outlines #48

dfrg · 2023-04-26T19:21:31Z

Proposed interface for implementation of color outlines in skrifa.

Implementation of color outlines in skrifa.

drott

Thanks for this well-written proposal. I read it and left a few comments below.

drott · 2023-05-02T08:56:49Z

text/2023-04-26-color-outlines.md

+> `canvas->restore();`<br>
+
+```rust
+fn push_clip(&mut self, glyph_id: Glyph_id, path: &Path)


Is path a fully built path? This creates some redundancy with the scaler's path pen / callbacks. In the existing Skia COLRv1 implementation we branch out to the regular path drawing/retrieval with FreeType. Perhaps this callback should just provide the glyph id that should be fetched for the path to clip?
This would also allow other settings such as hinting mode etc. to still be in principle applicable when fetching those paths as a separate stage.

Yes, the path has been fully loaded at this point. I settled on this design for a couple reasons.

First, while inside the ColorPen callbacks, the Scaler object (and it’s dependencies) have been mutably borrowed and Rust requires that access to be exclusive so you can’t effectively use it to load an outline without resorting to unsafe code.

Second, we need to load the paths internally anyway to compute the bounding box when clip boxes are missing from the table. This design allows us to do that only once and also cache outlines so that we’re not doing redundant work when one is used multiple times in a paint graph.

The Path type does have a method that takes a Pen so code for conversion to SkPath should be reusable.

I missed the point about hinting. Skia doesn’t currently hint these outlines. If that requirement changes, we can also handle that internally using the hinting mode that is set when configuring the scaler.

In Fontations my impression is we generally follow an approach of providing independent, functional building blocks. I suggest we try to do the same here and decouple the color glyph processing and drawing from other independent parts at useful interfaces.

I am in favor of creating a separate GlyphPainter (name TBD) interface, that's independent of Scaler operations. Where the context of the Scaler is useful for outline caching, the client can use it and pass it in.

The GlyphPainter for example, regarding the need for glyph outline caching, the GlyphPainter can provide a bounds() method that accepts a Scaler as an argument, as it needs glyph contour access with glyph rasterisation settings configured. So, I'd suggest to remove the bounds callback from the ColorPen trait . Not only for the reason of decoupling this from the Scaler but because I think the bounds method does not really semantically belong in the ColorPen as that's not a typical Canvas-like operation.

First, while inside the ColorPen callbacks, the Scaler object (and it’s dependencies) have been mutably borrowed and Rust requires that access to be exclusive so you can’t effectively use it to load an outline without resorting to unsafe code.

See above, are there obstacles to decoupling this from Scaler?

With this decoupled, we can remove the path: argument and have the client use Scaler for the purpose.

Second, we need to load the paths internally anyway to compute the bounding box when clip boxes are missing from the table. This design allows us to do that only once and also cache outlines so that we’re not doing redundant work when one is used multiple times in a paint graph.

If outlines are needed, they can be cached in a passed-in Scaler's context, but for the purpose of bounding box calculation in the simple TrueType case, contour retrieval is not needed, is it? For the bounding box, then union of each glyphs xMin, xMax, yMin, yMax should be sufficient, which does not require path retrieval? - May be different for COLRv0/v1 + CFF.

In Fontations my impression is we generally follow an approach of providing independent, functional building blocks. I suggest we try to do the same here and decouple the color glyph processing and drawing from other independent parts at useful interfaces.

I think there is a slightly blurry line when discussing fontations— read-fonts is intended to provide low level building blocks but skrifa is meant to be a more complete solution. That said, I agree with decoupling this in some way. Finding the correct boundary is a bit of a challenge while taking constraints into account.

If outlines are needed, they can be cached in a passed-in Scaler's context, but for the purpose of bounding box calculation in the simple TrueType case, contour retrieval is not needed, is it?

In addition to CFF, variable fonts also need to load outlines for this.

If outlines are needed, they can be cached in a passed-in Scaler's context, but for the purpose of bounding box calculation in the simple TrueType case, contour retrieval is not needed, is it? For the bounding box, then union of each glyphs xMin, xMax, yMin, yMax should be sufficient, which does not require path retrieval? - May be different for COLRv0/v1 + CFF.

Depends on whether you want tight or loose bounds for transformed glyphs.

Ah, this is a good point. We didn’t consider transforms in the graph.

drott · 2023-05-02T08:58:54Z

text/2023-04-26-color-outlines.md

+/// Enumeration containing data for solid or gradient fills.
+pub enum Brush<'a>;
+
+/// Contains a glyph path for clips or fills. Offers a method that can feed


See below, I think we can branch out for the existing scaler API for that?

drott · 2023-05-02T09:01:15Z

text/2023-04-26-color-outlines.md

+pub struct Color;
+
+/// Enumeration containing data for solid or gradient fills.
+pub enum Brush<'a>;


As discussed, it would be useful to see a bit more detail here on the value type and coordinates used here. Making this work well and be easily adaptable with the current Skia implementation would mean that ideally they'd stay quite close to what's in the font.

I think the adaptations to Cairo or Direct2D patterns/shaders and their constraints are different to the ones for Skia, for example.

I’ll add some definitions for this type. In the current prototype, the structure and values should exactly match those returned by FreeType with two exceptions: we do the 16.16 to f32 conversion for all values and the line projection for linear gradients (using the same code as Skia but without the y flip).

drott · 2023-05-02T09:02:54Z

text/2023-04-26-color-outlines.md

+```rust
+fn push_layer(&mut self, mode: Option<CompositeMode>)
+```
+Push a new layer to the state stack, optionally with a composite mode.


Might make sense to describe here which side of the COLRv1 graph this traverses / which layer becomes which: I.e. what's sourcePaint what's backdrop paint and this usually maps a PaintComposite operation.

Great point! I’ll add some notes about exactly how PaintComposite maps to a sequence of callbacks.

drott · 2023-05-02T09:05:15Z

text/2023-04-26-color-outlines.md

+```
+Fill the specified path with the given brush. Note that this is an optimization
+for the pattern `Clip Transform* Fill` in the graph. The intermediate
+transforms are captured in the optional `brush_transform` parameter.


Just FYI, the Skia version of this optimization does not support intermediate transforms, and would just drop the optimization and follow to the standard sequence of operations if there is a clip + transform + Paint*Gradient. So this latter parameter we might not need, or at least I haven't evaluated the performance benefit of that case.

Our own renderer does support this fast path so it would be nice to have it available if it’s not too cumbersome for Skia to special case here based on the presence of the brush transform.

drott · 2023-05-02T09:10:55Z

text/2023-04-26-color-outlines.md

+Skia does a significant amount of
+[processing](https://skia.googlesource.com/skia/+/265a074c910a0a4ac29176bca0a34615247f10f7/src/ports/SkFontHost_FreeType_common.cpp#597)
+on gradient fills to match the expectations of the relevant shaders. The
+applied transformations appear to produce equivalent brushes with color stops


As discussed, my suspicion here is that other 2D libs have different expectations, in particular for the sweep type of gradients. And while the output is hopefully really equivalent, it may then introduce two layers of value conversion (first in Skrifa, then on the client side) when using this functionality with other 2D libraries. Perhaps we can at least look at Cairo and Direct2D to get a sense for that.

Yes, I need to do more investigation on this. My suspicion is that the work Skia does here is generally useful for most renderers but I don’t have the data to back that up. In the meantime, it seems reasonable to return gradient data as contained in the font and revisit when I have a better understanding of the API landscape.

khaledhosny · 2023-05-02T14:56:34Z

FWIW, the corresponding HarfBuzz API https://harfbuzz.github.io/harfbuzz-hb-paint.html

dfrg · 2023-05-02T15:23:22Z

FWIW, the corresponding HarfBuzz API https://harfbuzz.github.io/harfbuzz-hb-paint.html

Thanks for this! It looks like there are a lot of similarities, particularly around how composite nodes are handled.

drott · 2023-10-09T12:07:22Z

text/2023-04-26-color-outlines.md

+pub struct Transform;
+
+/// 32-bit RGBA unpremultiplied color.
+pub struct Color;


If we don't do any gradient stop normalization or preparation for Skia specific shader behavior (which I find preferable due to differences), we don't need to interpolate any colors and we can decouple CPAL handling as well, and return color indices, and leave the rest to the client. I think this would be consistent with returning more values raw and as from the font.

As a side note, blackrenderer backends may be a good reference, but beware of gradient bugs in it, at least sweep is still broken (BlackFoundryCom/black-renderer#17).

Returning palette indices is definitely an option. If we do so, I would like skrifa to have a convenience method for resolving them.

drott · 2023-10-09T12:23:52Z

Separately, as discussed in b/301562314 and as the HarfBuzz API does it, it is probably useful to add a bitmap drawing / placement method to the ColorPen interface - but that can be a separate step.

dfrg · 2023-10-10T12:18:13Z

Separately, as discussed in b/301562314 and as the HarfBuzz API does it, it is probably useful to add a bitmap drawing / placement method to the ColorPen interface - but that can be a separate step.

Since landing googlefonts/fontations#609 and googlefonts/fontations#637 I have some ideas about how to support this. Let’s discuss f2f.

drott · 2023-10-10T16:05:29Z

@dfrg please feel free to amend, quick summary for this particular topic off the top of my head from our f2f:

We'll add a separate interface to Skrifa as a COLRv0 and COLRv1 driver that flattens the graph, drives callbacks.
- We'll keep the optimization for fill_path, but TBD whether we call it for PaintGlyph + PaintTransform + PaintSweep - maybe yes?
- We'll do some normalizations where it seems useful for other 2D APIs (things that come to mind, stop normalization, 2 point linear gradients instead of 3 point.)
- We don't deal with Path objects or actual color values on this interface, but pass glyph ids and color indices back to the client.
We'll separate out bounds computation to a separate function, likely with passing in Scaler or a successor to Scaler, such as ScalerInstance (Skia has a stage where it does bounding box calculation before, then only later calls into drawing.)
We don't include bitmap drawing right now, but keep it in mind to be potentially added to this interface.
@drott will draft a PR for such a new interface and test and develop locally based on that.
For decision on which parts we abstract behind this interface, @dfrg is offering take a look at some other 2D APIs such as Direct2D to decide what works well, and what's too Skia specific.

Among the reasons for keeping the callback interface restricted to COLR:

In this way, we are likely able to avoid extra allocations and caching - we have a driver for COLRv0 and COLRv1 that has low memory overhead and only needs to do circularity checks in a Hashset.
We provide more control to the application on glyph format selection, background on that: For monochromatic vs. color presentation emoji (in a hybrid font), or potentially in future CSS use cases where CSS allows glyph format selection, it's useful to have control over which glyphs are drawn, see also https://g-issues.skia.org/issues/40044044 and https://gitlab.freedesktop.org/freetype/freetype/-/issues/1142 where lack of glyph format selection in FreeType is a problem for Skia.

dfrg · 2023-10-11T04:12:43Z

@drott thanks for the summary. This all sounds good to me.

Use COLRv0/v1 test fonts from [1] to aid in the work of developing a COLRv1/COLRv0 painter callback interface, as outlined in [2]. [1] https://github.com/googlefonts/color-fonts [2] googlefonts/oxidize#48 (comment)

* Add COLR test fonts Use COLRv0/v1 test fonts from [1] to aid in the work of developing a COLRv1/COLRv0 painter callback interface, as outlined in [2]. [1] https://github.com/googlefonts/color-fonts [2] googlefonts/oxidize#48 (comment)

drott · 2023-11-21T14:10:57Z

Regarding the stop normalization, it turns out we can only do that level of stop normalization and preparation that does not involve color interpolation.

For sweep: We can normalize stops to start at 0 and end at 1 by normalizing angles, changing the meaning of angles from counter-clockwise to clockwise, and "flipping" the color line if start angle is greater than end angle. The rest would be handled client side.
For linear: We compute a new P3, so that the gradient is running from P0 to P3, and we can scale the vector from P0 to P3 so that stops are ranging from exactly 0 to 1.
For radial: We can only do the stop normalization, but with the result that we may send a negative angle to the client side. The negative angle may involve the need to truncate a color line and interpolate a color between two color stops. Details in the discussion here: Radial gradients and negative radii with variations or non-normal color stops colr-gradients-spec#367 (comment) - I think it's good to keep the separation of leaving color interpolation to the client, as working only with color indexes here is compatible with a potential future improvement of the CPAL table.

dfrg · 2023-11-22T01:47:42Z

Per f2f, this sounds like a great compromise to provide as many useful transformations as possible without doing color interpolation.

Discussion also touched on using a SmallVec like container to avoid some allocations for the color stop array. Skrifa already has one that might be useful, possibly with modifications.

COLRv1 implementation based on a `ColorPainter` trait object that a client implements. When calling the `paint()` method on a `ColorPaintable` the COLRv1 graph is traversed. Callbacks are sent to the client to execute typical COLRv1 transform, clip and fill operations. Implements googlefonts/oxidize#48

[skrifa] Callback-based COLRv1 implementation COLRv1 implementation based on a `ColorPainter` trait object that a client implements. When calling the `paint()` method on a `ColorPaintable` the COLRv1 graph is traversed. Callbacks are sent to the client to execute typical COLRv1 transform, clip and fill operations. Implements googlefonts/oxidize#48

dfrg added 2 commits April 26, 2023 15:19

API proposal for color outlines

c3f0a8e

Implementation of color outlines in skrifa.

there are many fonts

73a4251

drott reviewed May 2, 2023

View reviewed changes

drott reviewed Oct 9, 2023

View reviewed changes

yisibl mentioned this pull request Oct 12, 2023

Support COLR v1 table. harfbuzz/ttf-parser#133

Closed

6 tasks

drott mentioned this pull request Oct 25, 2023

Add COLR test fonts googlefonts/fontations#674

Merged

drott mentioned this pull request Nov 16, 2023

[skrifa] public color API googlefonts/fontations#702

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

API proposal for color outlines #48

API proposal for color outlines #48

dfrg commented Apr 26, 2023

drott left a comment

drott May 2, 2023

dfrg May 2, 2023

dfrg May 2, 2023

drott Oct 9, 2023

drott Oct 9, 2023

drott Oct 9, 2023

dfrg Oct 10, 2023

behdad Oct 10, 2023

dfrg Oct 11, 2023

drott May 2, 2023

drott May 2, 2023

dfrg May 2, 2023

drott May 2, 2023

dfrg May 2, 2023

drott May 2, 2023

dfrg May 2, 2023

drott May 2, 2023

dfrg May 2, 2023

khaledhosny commented May 2, 2023

dfrg commented May 2, 2023

drott Oct 9, 2023

dfrg Oct 10, 2023

drott commented Oct 9, 2023

dfrg commented Oct 10, 2023 •

edited

Loading

drott commented Oct 10, 2023

dfrg commented Oct 11, 2023

drott commented Nov 21, 2023

dfrg commented Nov 22, 2023

API proposal for color outlines #48

Are you sure you want to change the base?

API proposal for color outlines #48

Conversation

dfrg commented Apr 26, 2023

drott left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

khaledhosny commented May 2, 2023

dfrg commented May 2, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drott commented Oct 9, 2023

dfrg commented Oct 10, 2023 • edited Loading

drott commented Oct 10, 2023

dfrg commented Oct 11, 2023

drott commented Nov 21, 2023

dfrg commented Nov 22, 2023

dfrg commented Oct 10, 2023 •

edited

Loading