Support for graphics in the terminal #4763
Conversation
ayosec
force-pushed
the
graphics
branch
2 times, most recently
from
d5a1c08
to
f11ce16
Compare
This sounds like it might have a significant performance impact. Have you actually tested the performance of this PR?
That sounds extremely hacky, which I am not a fan of. This will likely just cause an endless heap of issues with things like selection, so it's not something we can just throw in and forget about.
This seems to just pile on a heap of code to support a bunch of image protocols. I don't like the idea of adding thousands of lines to Alacritty for something with so little use. If we support any protocol it should be the simplest one without any performance impact. I see no benefit in supporting multiple formats.
What about support for devices below OpenGL 3.3? This is something we would like to look into for the future so adding more code that requires at least OpenGL 3.3+ does not seem ideal.
I see little reason for adding memoffset/lazy_static usually, but I haven't looked at the code yet. |
Same |
I launched vtebench three times, in the current
Running $ base64 ./target/release/alacritty > /tmp/data
$ wc -l /tmp/data
635257 /tmp/data
# graphics branch
$ perf stat -e cycles,instructions,branches,branch-misses ./target/release/alacritty -e cat /tmp/data
Performance counter stats for './target/release/alacritty -e cat /tmp/data':
4,590,935,968 cycles
10,125,002,121 instructions # 2.21 insn per cycle
2,038,270,623 branches
4,651,069 branch-misses # 0.23% of all branches
0.854657772 seconds time elapsed
0.774462000 seconds user
0.466697000 seconds sys
# master branch
$ perf stat -e cycles,instructions,branches,branch-misses ./target/release/alacritty -e cat /tmp/data
Performance counter stats for './target/release/alacritty -e cat /tmp/data':
4,692,460,970 cycles
10,053,624,431 instructions # 2.14 insn per cycle
2,035,973,894 branches
4,795,961 branch-misses # 0.24% of all branches
0.855561156 seconds time elapsed
0.762745000 seconds user
0.493408000 seconds sys
I changed the NBSP character with a
Without comments and tests, the support for the iTerm2 protocol is less than 100 lines of code.
The only performance impact of this protocol is an extra branch in the
The fallback uses |
This issue is now fixed. |
That doesn't justify adding it. If Alacritty supports any graphics protocol, there should be one and it should be the simplest one available. |
|
I'm also getting the following error:
|
What is the output of |
|
96 or something like that iirc. |
|
I uploaded a fix in 387a224 to check if this fixes the problem. If this works, we may need to generate the code of the fragment shader dynamically, in order to use the 32 units if they are available in the hardware. |
|
That works and shows a consistent performance decrease:
|
|
@ayosec did you test the iTerm2 protocol with ranger? previewed images are kept on screen and not cleared, so new images are overlayed on top of the old ones, and images are kept in the background when text is rendered in the preview pane. Another issue is that the iterm2 graphics is working with tmux as long as the image file is small, but with large files, it looks like I'm hitting this, which is not an Alacritty issue but Alaciritty is spamming with warning messages like this:
You should be able replicate with:
Note that this issue doesn't happen when running ranger directly without going through a tmux session, and not when running in an ssh session. |
Ranger relies on overwriting images with spaces. The discussion in #910 rejected my first approach to be able to have this functionality. However, if we can change the implementation in this patch to use As a work-around, images can be replaced with the █ character (reversed with
This error happens because the base64 stream is incomplete. Did you use iTerm2 imgcat? It seems that tmux requires specific sequences to send images. Anyways, since iTerm2 will not be accepted, maybe it's not worth it to spend time debugging the issue. |
I can replicate the issue with the linked imgcat script and an image. I won't attach the image here because I'm not sure about its license but I can send it via email.
That's unfortunate, it would have been nice to have it. |
|
I guess it's better to wait for sixel to support 24-bit colors? |
|
Sixel isn't likely to change. |
|
I have updated the patch with the following changes:
|
|
Can your implementation of sixel convert 24bit colors into 256 colors and display a video efficiently? |
ayosec
force-pushed
the
graphics
branch
2 times, most recently
from
18fd66e
to
0a41635
Compare
|
Thanks for thinking about it and taking the time to reply, Chris. Too bad it doesn't make any sense. |
|
public service: copy-pastable reduced version of INSTALL.md to get @ayosec 's fork with sixel support: |
|
My 2c on the discussion:
|
* Add Sixel support Fixes alacritty#910 * Implementation of the XTSMGRAPHICS sequence. * Don't clear cells after the right side of the graphic. * Don’t erase text behind a sixel image; the image might be transparent We still add a reference to the graphic in the first cell of every line under the image, but we don’t erase any of the text in any of the cells. * remove an unncessary variable from insert_graphic * Avoid unnecessary clone when set graphic data. * Define MAX_GRAPHIC_DIMENSIONS as a 2-elements array. * support DECSET/DECRST (CSI ? Pm h) to change where the cursor ends up after printing a sixel image. The default is for the cursor to be moved to the first column of the line after the image. When we receive CSI ? 8452 h, we will instead leave the cursor on the last line of the image, on the next column past the end of the image. * put the TermMode back to just u32; I had miscounted the bits * move_forward takes a relative column count, not an absolute column number * Interprets mode 80 as Sixel Display Mode. This is reverse of the *sixel scrolling* option, which should match the actual behaviour of DEC terminals. For reference: alacritty#4763 (comment) * Fill all cells under a graphic with the template. With the template we can create hyperlinks attached to the graphic. To avoid reflow issues when a row is shrank, wrapped rows that only contain graphic cells are discarded. With this approach we loss some info, like the hyperlink, but the image is always properly positioned in the grid. * Allow hue values up to 360 in the Sixel parser. * Allow replacing part of a graphic with text. When text is added to a cell with a reference to a graphic, an operation is sent to the OpenGL thread to replace a subregion of the cell with a transparent area. If the OpenGL driver supports the GL_ARB_clear_texture extension, the region is updated with glClearTexSubImage. If the extension is not available, the texture is updated with glTexSubImage2D. * Highlight graphics to show hints. Similar to the underline line rendered when the cursor is over an hyperlink, for graphics we now render a border around the graphic. * Allow overlapping graphics. If a graphic is added over another one, the implementation now checks if new graphic has transparent pixels in every cell. If so, the graphic is appended to the cell, instead of replacing the previous one. SmallVec is used to prevent heap allocation when the cell only contains a single graphic. This should be the most common scenario. The GPU will store up to 100 textures. If another texture is added when there are 100, the oldest one is deleted. * Optimize graphics replacement. A simple optimization for inserting graphics is to detect when a new graphic is replacing completely an existing one. If both graphics have the same size, and the new one is opaque, we can assume that the previous graphic will not be displayed anymore, so it is not considered when update the graphics list in a single cell. This commit also adds serde implementation for GraphicsCell. This was used to debug the new logic. * Fix clippy warnings. * Use hls_to_rgb implementation from libsixel. * Changes in sixel module to be compatible with oldstable. - Reimplement abs_diff(). - Use positional arguments to format the error message in assert_color!(). * Initialize cell dimensions when create a Graphics instance. This fixes a bug that crashes the terminal when a graphic is added before resizing the window. * Support GLES2 Renderer in sixel * Set graphics limit per cell. The limit per grid is increased to `1000`, and a new limit per cell is added, set to `20`. * Add Eq derive to ClearSubregion. Suggested by clippy. * CSI XTSMGRAPHICS Pi=2, Pa=1 should return dimensions that fit in text area * Remove entry about Sixel support in the changelog. * Update damage when a graphics is inserted. * Apply rustfmt to term/mod.rs. * Apply rustfmt and clippy to the current branch. * Serialize/Deserialize only if the `serde` feature is set. * Apply clippy suggestions. * Include Sixel support in Device Attributes response. The response for `\e[c` (Send Device Attributes) now returns level 2 with the Sixel extension. The other extensions are 6 (Selectively Erasable Characters) and 22 (Color Text). The values are documented in page 04-19 of DEC-STD-070. * Upgrade vte fork. With `$ cargo update vte`. * Use the published vte-graphics crate, instead of a Git repository. * Fix sixels position when padding>0 --------- Co-authored-by: Ayose <[email protected]> Co-authored-by: Daniel Brooks <[email protected]> Co-authored-by: kumattau <[email protected]>
Fix sixels position when padding>0
|
Thank you so much for the brilliant work on the graphics support! I do most of my research via ssh in environments where X11 forwarding is sometimes disabled and I can't really send files around. So, it's quite painful to see that this functionality might never get merged. But, hopefully this branch can be maintained in the far future to keep alacritty a relevant option for my type of work as well! |
have you tried wezterm tho? |
|
Rio will also have image support in the very near future: raphamorim/rio#625 (WIP) |
|
Windows Terminal now supports sixel: |
Yeah, I am testing it now on the side, but Alacritty has worked absolutely perfectly since day one, so I'm not inclined to change. I am also using Zellij (not going back to tmux) to multiplex, but it doesn't support Kitty's graphics protocol, which would allow for clean in-terminal pdf viewing. So, for now I prefer Alacritty with sixels. If Zellij implements Kittys graphics protocol, I will have an another look. |
|
I would recommend foot (https://codeberg.org/dnkl/foot) in this case, works like a charm and supports sixel without the multiplexing wezterm comes with Edit: Forget foot, it's Wayland only and I assume you have to use Windows |
Use `end` of the cursor to draw a `HollowBlock` from `start` to `end`. When cursor covers only a single character, use `Beam` cursor instead of `HollowBlock`. Fixes alacritty#8238. Fixes alacritty#7849.
This changes the behavior of inline search from only accepting direct key inputs, to also accepting IME and paste. The additional characters are still being discarded, matching the existing behavior. This also fixes an issue where inline search wouldn't work for characters requiring modifiers, since the modifier press was interpreted as the search target instead. Closes alacritty#8208.
This patch adds a daemon mode to Alacritty which allows starting the Alacritty process without spawning an initial window. While this does not provide any significant advantage over the existing behavior of always spawning a window, it does integrate nicer with some setups and is a pretty trivial addition.
|
This kind of stubbornness is what drive good-willed people into hostility, just saying. I think I'll be giving Wezterm a try. |
The pipe was not using O_CLOEXEC, so it was leaked into the child. Fixes alacritty#8249.
This reverts commit 5dca7a8.
alacritty_terminal was pulling `serde` via vte even though serde feature was disabled.
The binary for Linux is also added to the release. The `contents: write` permission is needed to create releases and upload the assets.
|
Gentoo users can now install this fork from my overlay as gui-apps/alacritty-graphics (0.14.0 and live ebuild available) |
This patch adds support for graphics in the terminal. Graphics can be sent as Sixel images.
New features
Sixel parser
The Sixel parser is based on the SIXEL GRAPHICS EXTENSION chapter in a DEC manual.
The support is complete except for the pixel aspect ratio parameters. According to the manual, a Sixel image can specify that it expects a specific shape for the pixels in the device, but in none of terminals that I checked these parameters had any effect: they always assume a 1:1 ratio. Also, I didn't find any Sixel generator that emits a different ratio. To avoid extra complexity in the parser, it always assume 1:1 when the image is built.
There are two new terminal modes:
SixelScrolling(80)If enabled, new graphics are inserted at the cursor position, and they can scroll the grid.
If disabled, new graphics are inserted at top-left, and they are limited by the height of the window.
SixelPrivateColorRegisters(1070)If enabled, every Sixel parser has its own color palette.
If disabled, Sixel images can share a color palette.
Initially I didn't plan to support this mode, since it seems to be specific to xterm, but when I was testing applications using Sixel I found that mpv uses it to reuse a palette between video frames. Since mpv is based on libsixel, I guess that this feature could be used by more applications.
Both modes are enabled by default.
The function to convert HLS colors to RGB is a direct port of the implementation of the same function in xterm. I verified that the function emits the same results in all combinations of values 0, 30, 60, 90, and 100 in every color component. Only a few of these combinations were left in the tests to reduce the noise in the code.
To test the parser there are Sixel images generated with 3 different applications. For each one, there is a
.rgbafile in the same directory with the expected RGBA values. The commands to produce these files are inalacritty_terminal/tests/sixel/README.md.Byte
90as DCSSixel images using byte
90as DCS are not supported. DCS can be eitherESC Por90, but thevtecrate only recognizesESC P. I guess that this is because90can be a continuation byte in a UTF-8 sequence (two most significant bits are10), so it can be a valid input from users.Xterm has the same limitation. I don't expect that many applications depends on it.
ppmtosixelis an exception. It uses90to start the Sixel data, and it has to be replaced if we want to see an image generated by it.It is still interesting to test
ppmtosixelbecause it was written in 1991, long before Sixel was added to most (if not all) terminal emulators.