Aspect ratio, cropping and resizing

[SigmaRichards]

As mentioned in #22, the current implementation forces images to align with the grid. I don't have any experience with ueberzug, however, with kitty I don't believe this is strictly necessary.

The current implementation uses in 'renderer.lua' uses width and height as the target bounding box for the image. When resizing using magick, this bounding box is used as the exact size for resize. I've written a script to change the width/height on the fly to set a bounding box. Only one value is set in the image so the other is calculated by 'renderer.lua'. Below is the result from changing the size with the implementation as-is (with crop turned off in kitty features):

asis.mp4

Unsurprisingly, the aspect ratio changes when the image is resized. Compare that to below:

preserve_aspect.mp4

The change I made was during resize (code on the right, pls ignore how bad it is for now). It assumes that forcing the aspect ratio is always desired. Assuming the image will not by default conform to the grid, there are 2 options for resizing, enforcing pixel-width or height. Only one of these will fit entirely within the bounding box. After determining which to enforce, the other value can be calculated as being the resolution which best preserves the aspect ratio. I haven't setup a PR because I'm hoping to get some feedback first about what the most appropriate solution is.

Additionally, I've been experimenting with ways to efficiently scale images on the fly, which got me looking into the kitty protocol more heavily. Using control keys c and r when displaying, shrinks or expands the displayed image to fit within that rectangle. Since these values do have to conform to the terminal grid, this would still break the aspect ratios and resizing as above, however, may mean we don't strictly need to use temp files.

I'm curious if anyone has thoughts on how this should be handled. Any thoughts on this would be great!

[SigmaRichards]

Oh, and I forgot to mention. I've been looking into the animation frames in the kitty documentation. Since it allows for frames to made by composing smaller images, I believe there is a way to use control keys c and r to perform all the resizing necessary (i.e. without the need for creating tempfiles with magick) while still preserving aspect-ratio. I haven't managed to get this working but I think it should be possible.

[3rd]

Thank you so much for all the effort you've put into this, this is amazing research!
Ueberzug is not a priority at all, Kitty is the focus for most of us (I hope) and this would greatly improve the experience.
I think your solution is super ok, and we can only enable it for Kitty if it won't work as it is with ueberzugpp.

Not having to write the resized variants to disk would also be super amazing, I try not to think about how many files we create as it makes me cringe and be ashamed. I'll look into c and r as well!

Thanks again for all the work!

[SigmaRichards]

I've done some testing on ways the kitty protocol can be used directly to provide scaling (both suggested in #22 ) which retains aspect ratio. These both utilize the calls to display the image so only require 1 transmit call to provide all the scaling. I wanted to look into ways to minimize the temp files that get created.

The first way uses the padding values X and Y to offset where in the starting cell the image is displayed from. Scaling is then achieved by using (appropriately paired) r and c values, as these ignore the offsets. This is by far the easiest implementation which deals with both of these issues. The main issue with this approach is that the image is (often) offset from the starting cell and if resizing rapidly, is very erratic (this is less noticeable at smaller cell sizes). This method can be seen in the first half of the video below.

The second method uses transparent padding on the input image which is cropped as required to maintain the aspect ratio of the image. Scaling is also achieved by using r and c. This has the benefit that the top left of the image is aligned, so resizing is significantly smoother, however, we need the padded images prior. This can be seen at the second half of the video.

aspect_and_scaling.mp4

What is the best way to approach this? I think the second is more desireable since the consistent location obviously makes it smoother, however, in the first one, there is no padding requirements, so will be more scalable.