Embedded raster graphics images

[peterbrittain]

Some terminals support embedding graphical images in the terminal. For example:

• iTerm2: https://www.iterm2.com/documentation-images.html
• Sixel graphics: https://github.com/saitoha/PySixel

It would be nice to be able to make use of this in a cross platform manner.

[peterbrittain]

Not enough standard support across terminals yet... Deferring for some time.

[garfieldnate]

blessed supports this using w3mimgdisplay (packaged with Ranger).

[peterbrittain]

Looks like that w3m library is drawing directly to the screen in some cases, which will never work on a true terminal, so I'm not sure I'd want it in by default. Possibly an optional extra?

[peterjschroeder]

Here is my semi-working version of sixel support. It only seems to work correctly when the terminal is full screen. I believe it may have something to do with using _width and _height, but it may also be the use of TextBox for it. Any pointers would be appreciated.

`def sixel(image, x, y, max_w, max_h):
# Translate character count for max width and height to pixels.
# This may be the wrong way to go about this
max_w = int(max_w * (shutil.get_terminal_size()[0] / 12.7))
max_h = int(max_h * (shutil.get_terminal_size()[1] / 2.11))

# Convert to sixel
with Image(filename=image) as file:
    # Resize image keeping ratio
    width = file.width
    height = file.height

    if width > max_w:
        height = (height * max_w) // width
        width = max_w
    if height > max_h or height < max_h and width < max_w:
        width = (width * max_h) // height
        height = max_h

    file.sample(width, height)

    # Save as sixel
    if not os.path.exists("/tmp/sixel"):
        os.makedirs("/tmp/sixel")
    file.format = 'six'
    file.save(filename='/tmp/sixel/%s' % os.path.basename(os.path.splitext(image)[0]+'.six'))

# Display sixel
with open('/tmp/sixel/%s' % os.path.basename(os.path.splitext(image)[0]+'.six'), 'r') as file:
    image = file.read()

    # Center the image
    x += ((max_w - width) / 2) / (shutil.get_terminal_size()[0] / 12.7)
    y +- ((max_h - height) / 2) / (shutil.get_terminal_size()[1] / 2.11)

    sys.stdout.write("\x1b7\x1b[%d;%df%s\x1b8" % (y, x, image))`

Example function call.

sixel("/home/peter/.local/share/Cockatrice/Cockatrice/pics/downloadedPics/OP1/%s.jpg" % (cards[self._list.value]['name']), self._cardimage.get_location()[0]+1, self._cardimage.get_location()[1]+1, self._cardimage._w, self._cardimage._h)

[peterbrittain]

That's a good start! The worry I have is how to decide whether to allow an application to use something like that API or not. How do you detect the safety of that feature on Linux (all terminal types), MacOS and Windows (native, CygWin, or even cases like PuTTY connecting to a Linux server)?

[peterjschroeder]

As far as I know, it would have to be on a case by case basis.

if "mlterm" in os.environ['TERM'] or "xterm" in os.environ['TERM']:

I would have it fallback to a text image using libcaca or something similar.

[peterbrittain]

Asciimatics already supports image display using text images (without falling back to libcaca). Check out the renderer docs, or images.py sample.

My big worry with this feature is that until it is easy for an average user to set up and use on all basic systems, it is going to cause more problems than it solves. We can't insist on an average user doing a custom build of terminals or install non-standard terminals for their OS. And if most users just see the existing feature, is that worth the time and effort?

There will also need to be some extra logic added to the Screen objects to recognise when sixel is in use, so that it properly integrates the image with the terminal double-buffering.

[peterjschroeder]

Nice, I hadn't seen the ImageFile function.

A universal Image function would make it easy for the user.
In this Image function, it could check for sixel compatible terminals. If found it uses the sixel function, if not it falls back to ColourImageFile. Than check if the terminal supports color, if not fall back to ImageFile.

I did try to use Screen to draw the sixel, but for some reason it strips the escape codes so it just shoots out text to the screen.

[peterbrittain]

Yeah - you can't print escape codes in the Screen as it is already handling that for you in a cross-platform manner.

I need to think about this some more, but I suspect there would need to be some logic for the Screen to be aware which parts are reserved for sixel (and a way to link to the data stream for the image while it should be drawn). There also needs to be some way to spot when the sixel image needs removing (as it gets overwritten) or moving (as the screen scrolls).

[peterjschroeder]

I used textbox for the reserved space, but a dedicated widget for images would be preferable.
Is _w and _h suppose to store the current width and height of the widget. Or am I misusing these? Should there be something like get_location but for width and height?
I haven't tested scrolling yet, but if you cat a sixel on a supported terminal it scrolls properly.
As you said, there should still be a way to clear it.

[peterbrittain]

I don't quite understand how you got your custom widget to draw the image at the right time. Could you explain, or share the code?

[peterjschroeder]

It actually tries to draw it before it starts, but that is most likely a simple fix as it didn't do it before I started adjusting some things. Attached is my code.

ccg.zip

[peterbrittain]

Yeah - I get it now. You're bypassing the refresh logic entirely and relying on the fact that asciimatics isn't redrawing in that box because the contents in the double-buffer haven't changed. While it works for this specific case, there are several ways to break that assumption and so this isn't ready for general consumption yet. The Screen object needs to be aware of all the sixel content in the affected text characters and be able to redraw and crop them as needed. Given that this is the most performance critical part of the code, it also needs to be done in such a way that the double-buffering doesn't grind to a halt for large images.

[peterjschroeder]

Here is an updated version that is more integrated (based off TextBox). I made more use of external libraries to shorten the code, but I removed the wand dependency. New dependencies are libsixel-python and python-resize-image.
The positioning (and therefore the width and height) is still off by 1 and I'm not sure why. The math is sound. I am thinking _w and _h may be including separators and/or scrollbars? Or maybe I biffed somewhere.

`class ImageBox(Widget):
"""
An ImageBox is a widget containing an image.

It consists of a framed box with option label.
"""

__slots__ = ["_label", "_required_height"]

def __init__(self, height, label=None, name=None, on_change=None, **kwargs):
    """
    :param height: The required number of input lines for this ImageBox.
    :param label: An optional label for the widget.
    :param name: The name for the ImageBox.

    Also see the common keyword arguments in :py:obj:`.Widget`.
    """
    super(ImageBox, self).__init__(name, **kwargs)
    self._label = label
    self._required_height = height
    self._on_change = on_change

def update(self, frame_no):
    self._draw_label()

    x = self.get_location()[0] + self._offset
    y = self.get_location()[1]
    width = self.width
    height = self._h

    # If the image doesn't exist than clear the image area and return
    if not os.path.exists(self.value):
        for i in range(y, y+height):
            sys.stdout.write("\x1b7\x1b[%d;%df%s\x1b8" % (i, x, '\033[12m '*width))
        return

    # Translate characters to pixels
    import fcntl, struct, termios
    farg = struct.pack("HHHH", 0, 0, 0, 0)
    fd_stdout = sys.stdout.fileno()
    fretint = fcntl.ioctl(fd_stdout, termios.TIOCGWINSZ, farg)
    rows, cols, xpixels, ypixels = struct.unpack("HHHH", fretint)
    fontw = xpixels // cols
    fonth = ypixels // rows
    max_width_pixels = width * fontw
    max_height_pixels = height * fonth

    # Resize image keeping ratio
    with Image.open(self.value) as f:
        image = resizeimage.resize_thumbnail(f, [max_width_pixels, max_height_pixels])

    # Center the image
    x += ((max_width_pixels - image.width) // fontw) / 2
    y +- ((max_height_pixels - image.height) // fonth) / 2

    # Convert image to sixel and display
    sixel = BytesIO()
    dither = sixel_dither_new(256)
    sixel_dither_initialize(dither, image.tobytes(), image.width, image.height, SIXEL_PIXELFORMAT_RGB888)
    sixel_encode(image.tobytes(), image.width, image.height, 1, dither,
            sixel_output_new(lambda imgdata, sixel: sixel.write(imgdata), sixel))
    sys.stdout.write("\x1b7\x1b[%d;%df%s\x1b8" % (y, x, sixel.getvalue().decode('ascii')))

def reset(self):
    pass

def required_height(self, offset, width):
    return self._required_height

def process_event(self, event):
    pass

@property
def value(self):
    if self._value is None:
        self._value = [""]
    return self._value

@value.setter
def value(self, new_value):
    # Only trigger the notification after we've changed the value.
    old_value = self._value
    if new_value is None:
        self._value = [""]
    else:
        self._value = new_value
    self.reset()
    if old_value != self._value and self._on_change:
        self._on_change()

@property
def frame_update_count(self):
    return 0`

[peterbrittain]

We're getting there... This will work on a single Frame UI, but will not work for one with multiple Frames (or Canvases) as it is drawing over the top of whatever is there without looking at what else is on the Screen.

I'm not sure what the best way forwards from here is. Options that spring to mind are:

1. Break the image down into character sized blocks and use sixel to draw each block independently. Then we could extend the Screen to draw each sixel block.
2. Provide a way to reserve some parts of the Screen for image drawing and leave it up to the application to solve.
3. Provide some way of reserving parts of the Screen and an associated image that the Screen will render separately as a sixel image, having masked it for what is visible.

None of these sound easy and/or make a poor API for the user.

[peterjschroeder]

I may be misunderstanding what your are saying, but it is currently reserving a part of the screen to draw the sixel. The sixel drawing is restricted by the coordinates of the ImageBox.

[peterbrittain]

Looks great, but try putting a second Frame in the Scene that overlaps your image.

[peterjschroeder]

Got it, it eats up part of the image. The sixel would need to be refreshed after the obstructing object that is covering it closes. I assume this is normally handled in the canvas drawing, which can't be used because it strips out escape codes (and maybe other reasons).
Something else that I need to fix, is the cursor needs to be put back where it was after drawing a sixel, or it leaves artifacts.

[peterbrittain]

Yep - asciimatics maintains a double buffer for the Screen and any Canvas. It uses this to update any area it thinks has changed when refresh is called. This mechanism needs extending for sixels in some way.

You can probably solve the artifact problem by setting the Screen internal cursor state to invalid values. That would result in it just moving the cursor next time it needs to print something.

[peterjschroeder]

I'll have to give some thought on the first part. It would be preferable to keep the widget self-reliant.

I tried setting it to invalid values, but it had no effect. I am thinking that the selected item on the ListBox is color-bleeding. If I scroll too fast on the ListBox, I get artifacts in it's scrollbar. When I added a popup box for testing, the artifacts became red which seems to verify this.
I haven't really dug into the color code that much yet, but to me it seems like the colors are not being reset at the end of their use.

[peterbrittain]

Asciimatics keeps track of the current colour and only changes it when needed. If sixels mess around with the text colour, you'll need to set the current fg, bg and attribute to invalid values to force the next character to be drawn in the right colour.

[peterbrittain]

I could be wrong, but I suspect that kitty solves the remaining issues with unicode placeholders - see https://sw.kovidgoyal.net/kitty/graphics-protocol/