image integration and sixel support

[johnnychen94]

Currently, we have two image encoding strategy in Julia:

• lookup table: converts an RGB color to its closed 256 colors, and encodes into a magic terminal control sequence. https://www.ditig.com/256-colors-cheat-sheet
• Sixel format: an advanced encoding format that only a few terminals support

using TestImages, ImageInTerminal, Sixel
ImageInTerminal.disable_encoding()
basic_image_string(img) = (io = IOBuffer(); ImageInTerminal.imshow(io, img); String(take!(io)))
sixel_image_string(img) = (io = IOBuffer(); sixel_encode(io, img); String(take!(io)))

img = testimage("lighthouse")

Which prints the encoded string like this -- UnicodePlots also take the direct lookup table method

It seems that Term's Panel supports the "basic" encoding very well, while not working well with the Sixel sequence:

julia> print(Panel(sixel_image_string(img)))
ERROR: KeyError: key "9" not found
Stacktrace:
  [1] getindex
    @ .\dict.jl:481 [inlined]
  [2] #ANSICode#3
    @ D:\TongYuan\.julia\packages\Term\eBIZX\src\color.jl:58 [inlined]
  [3] get_style_codes(style::Term.style.MarkupStyle)
    @ Term.style D:\TongYuan\.julia\packages\Term\eBIZX\src\style.jl:104
  [4] apply_style(text::String)
    @ Term.style D:\TongYuan\.julia\packages\Term\eBIZX\src\style.jl:147
  [5] Segment
    @ D:\TongYuan\.julia\packages\Term\eBIZX\src\segment.jl:31 [inlined]
  [6] (::Term.panel.var"#makecontent_line#6"{Symbol, Term.measure.Measure, Int64, Term.layout.Padding, String, String})(cline::String)
    @ Term.panel D:\TongYuan\

cc: @t-bltg @soumitradev

[FedeClaudi]

Hey,

I turned this into a discussion (it's easier for me to keep track).

Thanks for explain the different ways to represent an image in the terminal. I wasn't aware about sixel, I will do some reading to understand how it works and get back to you on it.

For the basic string representation, that should just work since it is a string. Not sure if it's something worth looking into, but one thing to improve the image visualization is to use the half-block unicode character: ▄ instead of █. This allows you to set the color of character and the background differently so that you have to shades in the same line. That's what I did to make the colors in the image in the README, see here. Also Term and most terminal support a lot more colors now so you could use that to create something more faithful to the original image.

Either way, any string representation should just work, I need to look into sixel. I'll get in touch in a bit!

[t-bltg]

I'll try to have a look into this too.
As a side note @FedeClaudi, when you are working with sixels, make sure to use a compatible terminal (see https://github.com/JuliaIO/Sixel.jl#terminals-that-support-sixel).

[FedeClaudi]

Btw, I don't do C, but apparently there's a latex2sixel: https://github.com/nilqed/sixelSPAD/blob/master/bin/latex2sixel

[johnnychen94]

It does almost the same thing to https://github.com/korsbo/Latexify.jl, the idea is to compile latex text to image, and then encode into sixel sequences.

I also have an experiment branch for it: JuliaImages/ImageInTerminal.jl#49 (comment)

[FedeClaudi]

cool!

[johnnychen94]

For xterm encoding/decoding improvement, maybe the best way to push the progress is to steal JuliaImages/ImageInTerminal.jl#62 and make a new repo, expose a simple encoding/decoding interface (such as Sixel.jl does) without interweaving with the REPL, show, or display thing.

I really hope I can do it myself but it seems that I'm always busier than I thought.

[FedeClaudi]

That's fair. I don't think it's something that I would be happy to take on either (at least in the near future), but I just thought I'd give my two cents on the topic.

[FedeClaudi]

Okay, I've done my homework including reading this detailed explanation about sixel syntax: https://vt100.net/shuford/terminal/all_about_sixels.txt

My understanding is that sixel is rendered through some terminal "magic", but it's different from the way text is handled in the terminal. E.g. when I try to select the image:

compared to the basic image encoding:

as such, I don't think it can work with Term, since I can't figure out a way to manipulate it as you would a sequence of characters and ANSI escape codes. Do you have any idea of how that would work?

---

So image integration should work out of the box, but I can't see how sixel would fit with all this unfortunately?

[jerch]

For terminals, sixel is meant to directly place pixels into the graphics output canvas (print on top), thus sidepaths any of the typical foreground (FG) / background (BG) terminal buffer content. Therefore most terminal sequences addressing FG/BG stuff dont work with sixel anymore, as one would expect.

The terminal magic happening is:

• enter DCS mode (different terminal operation mode, capable for binary like payload ingestion)
• switch to sixel subparser (annotated by DCS params)
• eval and decode 6-bit encoded sixel data
• place pixel information into output canvas
• end DCS (dropping back into normal TEXT mode)

On modern TEs with scrollbuffer things are slightly more complicated, as ppl expect image data not to be tagged to a fixed canvas position, but to scroll along with the scrollbuffer relative to the initial cusor position.

[t-bltg]

I can't figure out a way to manipulate it as you would a sequence of characters and ANSI escape codes. Do you have any idea of how that would work?

I could nest a sixel image into a frame in JuliaPlots/UnicodePlots.jl#196, but it feels hacky.

[FedeClaudi]

that's cool.

If I understand correctly, you're moving the cursor around before printing the sixel image right? That's something that one could do, and it would work, but it does feel a bit hacky.

More importantly, it gets hard to handle (e.g. if you combine multple panels each with it's own image etc...). That's partly Term's fault since when you for instance stack to Panel objects you are left with a single Renderable and loose all information about where each panel starts/ends - so you can't do this trick anymore. If there was a way to keep track of where each panel ends up being then one can move the cursor to the right position and print the sixel image.

[jerch]

If I understand correctly, you're moving the cursor around before printing the sixel image right? That's something that one could do, and it would work, but it does feel a bit hacky.

Thats actually the official and only reliable way to do it.

[t-bltg]

In UnicodePlots, we work on the plots lines (rows), so for each row, I print the left border character, then the sixel encoded 'line', then I move the cursor up with \e[A then offset chars forward with \e[$(offset)C, I can then write my right border character and end with a newline, and proceed to the next row.

[FedeClaudi]

I see, that should work then, very clever.

Term uses segments which are basically lines of text with applied styling (ANSI code). So a Panel is just a collection of segments : https://fedeclaudi.github.io/Term.jl/dev/basics/renderables/#AbstractRenderable

So one way to do it would be to split each line of the sixel into a "segment, and then edit the segment printing such that if the segment include sixel code it does the magic printing discussed above.
There's two obstacles

1. Each segment has a measure which says how wide the segment is going to be when printed to the terminal, this should include the correct width of the image
2. You would have to figure out a way to know which segments include sixel information such that you can move the cursor accordingly. Perhaps each segment with sixel info could also include the initiation sequence DCS p1; p2; p3; q and then a simple regex can detect it.

So I guess the first thing would be figure out: how to split sixel into lines, how the width of the lines and if each sixel line can have DCS p1; p2; p3; q in it

[johnnychen94]

I'm not sure if this helps:

https://vt100.net/shuford/terminal/all_about_sixels.txt

Although I've never seen it described in such terms, one can think of a
sixel "cursor" - the position where the NEXT sixel will be displayed,
analogous to the usual text cursor. What I've described so far is
equivalent to a text stream containing no Carriage Return or Linefeed
characters: the "cursor" simply keeps moving "linearly" from left to right.
But the Sixel format includes two "cursor repositioning" control
characters, one which behaves analogous to a carriage return, and another
which behaves like a "carriage return, linefeed" pair.

The "-" (hyphen or minus sign) character moves the sixel "cursor" to
the "beginning of the next line (or band)" of image data. In short, it
acts like a "new line" (*NIX lingo) character or, more precisely, like a
CR/LF pair would in text mode.

The "$" (dollar sign) character moves the sixel "cursor" to the
"beginning of the current (same) line (or band)," analogous to a plain CR
in text mode. This allows new sixel data to "overstrike" what has already
been displayed, a necessity for rendering multiple-color imagery. (Data of
a single color is laid down, then a $ character returns the "cursor" to the
left, then data for a new color is laid down "on top of" what's already
there.)

A trick that might be work is to send a test image with known size to the terminal,
and measure the line width in terms of text positions. Is this doable?

The DCS control sequence DCS p1; p2; p3; q uses p1 to indicate the height-width ratio,
which is 2:1 by default -- the \ePq thing.

[jerch]

To my knowledge none of the modern terminal emulators implements a custom sixel-pixel size handling, instead treats it always as 1:1 to screen pixels.

So while a fully spec conform implementation would also address the pixel ratio, it is currently not really supported by terminals, thus can be left out of an implementation to get things started (just hardcode it to 1:1).

[t-bltg]

A trick that might be work is to send a test image with known size to the terminal,
and measure the line width in terms of text positions. Is this doable?

Looks like libsixel encodes the image size in the header i.e. for a 50x50 image: \ePq"1;1;50;50[...]
The caret size (in pixels) in a terminal can be queried with \e[16t e.g. in xterm I get ;15;7t.

Thus, it looks like the surface occupied by the sixel sequence is approximately 8 columns x 4 rows:

julia> using TestImages
julia> img = TestImages.shepp_logan(50)
[...]
julia> cols = ceil(Int, 50 / 7)
8
julia> rows = ceil(Int, 50 / 15)
4

[FedeClaudi]

This all looks doable, I guess the only question is who's going to do it? :)
Is it going to be for the gsoc project?

[FedeClaudi]

Just a small update on this, Term.jl will support Markdown rendering: https://twitter.com/Federico_claudi/status/1535034402018934784?s=20&t=MrkwTIrk35NTE33r0MdQaA

though of course it wouldn't handle images yet:

sixel support would fix that.

[t-bltg]

For sixels in ImageInTerminal, I think we need to finalize reworking the package structure (JuliaImages/ImageInTerminal.jl#58, JuliaImages/ImageInTerminal.jl#61).

@johnnychen94, these PRs seem to have stalled: do you know what is preventing going forward ?

[johnnychen94]

It's mainly because of my laziness 🙈 (and the fact that I probably took compatibility issues too seriously) Too many things flood in, and with my limited time, I just can't handle them all. I'm really sorry about that.

The technical part is: ImageInTerminal currently is used for two purposes, the encoding of the image and the display. One side-effect of using ImageInTerminal for encoding purposes is that images are unconditionally encoded in REPL and this is unwanted for many scenarios.

After wrapping the Sixel.jl package, I believe it's better to separate the encoding/decoding part from the image display part. Something like the following:

JuliaImages/ImageInTerminal.jl#58 and JuliaImages/ImageInTerminal.jl#61 are a good code refactoring PRs by separating the encoding and display apart. But it is strongly affected by the legacy codebase design. I planned to make it
a "pure" encoding/decoding library with none or few terminal handling codes and JuliaImages/ImageInTerminal.jl#62 is the immediate output for it. The old codebase has a few smart auto strategies on the running terminal, which is good for interactive usage but isn't good for programming. -- Some packages depend on ImageInTerminal to encode the image into text sequences and then test against them (ReferenceTests.jl), but that doesn't work very well if the encoding strategy is terminal-relevant.

@t-bltg If you want to revive it, maybe you can start a new empty repo and move the pure encoding/decoding things there? I wish I'm not the one that blocks the development.

Another thing that blocks the development is image dithering -- we have a fantastic DitherPunk.jl library, and I want to make a room for it so that we can add dither support in the future.

---

For Term.jl and UnicodePlots.jl, a smart (heavy) adapter package TerminalImages can be convenient, but they can just depend on the pure encoder packages. I know Term.jl is trying to make everything self-complete and to depend on as few dependencies as possible.

[t-bltg]

@t-bltg If you want to revive it, maybe you can start a new empty repo and move the pure encoding/decoding things there? I wish I'm not the one that blocks the development.

I can start working on that in the next few days (I want this to go fast, so it won't stale again: it won't be perfect at first shot, but it shall be polished under way) ... I think we can place these repos in the JuliaImages org. Can you can grant me write access to this organization / membership, so that we can work collectively on these (I feel I might not be qualified enough to handle the whole transition) ?

[johnnychen94]

Invitation sent. Feel free to create a new repository for it! I really hope I'm not blocking the development.

[t-bltg]

Thanks. Since Sixel.jl is part of JuliaIO, I think XTermColors.jl should go there as well ? If I could join this org as well, I can start with that.

[johnnychen94]

I don't think we have a standard format name for xterm colors -- we could name it as .jxc (julia xterm color) file but I doubt people will use it in practice.
Let's start with JuliaImages/XtermColors.jl first, or it can be a subfolder(sub-package) in ImageInTerminal.jl (just like SnoopCompile or Makie)

[jerch]

Hi there (comming from Textualize/rich#384)

I commented on a few things above, maybe this helps to get sixel support rolling. What I currently wonder about - how do you create the sixel data in the first place, with which encoding lib?

[t-bltg]

@jerch, thanks for your valuable comments (and pointing the rich discussion) !

how do you create the sixel data in the first place, with which encoding lib

We use https://github.com/JuliaIO/Sixel.jl, a julia wrapper for https://github.com/libsixel/libsixel.

[jerch]

Ah ok, thx for the pointer. I have never worked with libsixel API myself, so cannot tell, if it provides enough flexibility for more advanced low level usage pattern like partial image output (not doing the quantization/dithering steps over and over). If Term.jl aims to be as versatile as Rich/Texual, embedding a sixel image into a styled container with proper (pixel-perfect) alignment and sub scrolling caps will need fast redrawing of image subparts, ideally not reencoding every time from scratch to avoid sluggish performance.

This can be the point, where your own sixel encoder may outperform libsixel, just because you can fine-tune re-calc steps (e.g. do quantization + dithering just once).

In terms of performance, sixel encoding is dominated by the quantization step by several multitudes, while dithering and 6-bit encoding + run length compression are cheapo. So ideally you want to do the quantization only once for a given image source, simply for perf reasons.

But there is a second, maybe even more important reason, when outputting image slices: a full requantization + dithering on image slices is not stable in found palette colors + applied dithering pattern, thus might lead to perceivable color flips and weird moire effects / dithering pattern breaks. (Btw, you already might have spotted this from line-wise sixel-image composition, when doing a full libsixel encoding for the line parts, where colors show slight value shifts from one to the next line.)

[johnnychen94]

This is a very good observation! It's not supported by Sixel.jl but I believe this is somehow doable by caching a dither context object and reuse it.
I've opened an issue for this JuliaIO/Sixel.jl#16 and will try to address it when Term.jl actually reaches this performance/stability issue.

But maybe I'll just reimplement the sixel protocol in pure Julia when DitherPunk.jl's performance is fixed -- it's more flexible and more functional than libsixel's dither option.

[jerch]

If you want to do the encoding yourself, maybe have a look at chafa's sixel encoder, which is currently the fastest one implmented (https://github.com/hpjansson/chafa).