Grapheme clusters width miscalculation

[haliphax]

It seems as though grapheme clusters (e.g. anything using skin tone modifiers, emoji variant selector, flags, etc.) are not properly accounted for when calculating the width of things. In the example program below, the scroll bar on initial load looks like a jagged mess. While it will smooth out after scrolling around a bit, a periodic refresh will break it again. I have seen some similar issues with placement of widgets on top of widgets containing grapheme clusters, as well (e.g. modal screens).

from textual.app import App, ComposeResult
from textual.containers import VerticalScroll
from textual.widgets import Static


class ExampleApp(App):
    def compose(self) -> ComposeResult:
        yield VerticalScroll(Static())

    def on_mount(self) -> None:
        widget = self.query_one(Static)
        widget.update("Grapheme clusters ✌️ 🧑‍🚒\n\n" * self.size.height)


if __name__ == "__main__":
    ExampleApp().run()

[github-actions[bot]]

We found the following entries in the FAQ which you may find helpful:

• How do I center a widget in a screen?
• How do I pass arguments to an app?

Feel free to close this issue if you found an answer in the FAQ. Otherwise, please give us a little time to review.

This is an automated reply, generated by FAQtory

[haliphax]

Example with overlaying widgets:

[TomJGooding]

Unfortunately there's nothing Textual can do about it, based on a reply by @willmcgugan to this similar issue #3040 about emojis:

I'm assuming this is just a classic emoji issue. Terminals don't agree on how wide they should be. Not much we can do about it alas.

[willmcgugan]

Yes, terminals render multi-codepoint emojis differently. And there is no way to know exactly how they will render

[haliphax]

While I do understand the morass that is modern terminal emulation with regard to grapheme clusters, various double-width characters, and the like, I guess my point is this: Much like how there are flags in the rich Console for dealing with legacy Windows compatibility and adapting to different color systems, it would be nice if textual had a way for me to specify that the application I'm running should make a particular set of decisions about how to render those characters rather than that decision being hard-coded into the library.

I understand that is not necessarily a trivial amount of work; I'm more hoping to broach a discussion on the possibility of such a feature. I'm not sure if that would make more sense being handled by the textual application or by the underlying rich console, however.

[haliphax]

Here is some work I began in an attempt to tackle this problem outside of my TUI dependencies:

Grapheme class - stores (potentially clustered) graphemes and their attributes, allows for "forced double-width" override of problematic emoji

GraphemeBuffer class (though mislabeled in the docstring) - represents a string-like structure composed of a list of Grapheme instances for displaying to the terminal, has actual display implementation of "forced double-width" override and can simplify unsupported variations to maintain consistent display width by sacrificing modifiers/ZWJ sequences

I had a demo script of these in action that I would be happy to reproduce (since I deleted it a little while back when I abandoned building my own TUI library to use textual).

These are not complete by any means, and have yet to tackle things like detecting "tag" sequences (e.g. flags), and the assumptions about terminal behavior are stored in constants rather than as function arguments, but I felt like I was at least beginning to get somewhere.

Edit: Here are the constants used in both scripts, some of which represent the options mentioned in the paragraph above.

[haliphax]

To clarify further: My terminal is capable of handling these grapheme clusters normally and displaying them at their correct width. It seems as though textual (or perhaps the rich Console beneath it?) is making an assumption that my terminal does not support zero-width-joined emoji.

The work I mentioned in my previous reply was mostly structured around dynamically enabling/disabling support for these features at runtime regardless of my host terminal, as I am using all of this to display to clients within a custom SSH server project. It will ultimately display on the client's terminal, not mine. Sorry if that's a red herring for this discussion, but I wanted to put everything up here that I felt might be useful. (There's a bit of code in there about forcing double-width on characters whose wcwidth() result is incorrect that would apply in either case, though.)