Dear Bram

I second this request.

--
Erik Falor
Registered Linux User #445632 http://counter.li.org

Send me a patch and I'll look into it.

If you want a more useful response, use your real name and include more
information about what you actually want, what system, etc.

--
hundred-and-one symptoms of being an internet addict:
195. Your cat has its own home page.

/// Bram Moolenaar -- [email protected] -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

Hi Bram,

I would like this. For all systems. Everywhere.

This is a large undertaking. This isn't a "send me a patch" request.

I am talking about ripping out the byte-queue input system and replacing
it with a structured keypress queue. The lot. Everything. Remove the
"stuck in the 1970s all the world is a DEC-style glass virtual terminal"
input queue, and replace it with one that has structures; remembers
modifiers, etc.. With a structure we can properly represent "Ctrl-I" as
being the letter I with a control key, totally separate from the Tab.

Having done this, we can feed in the input events from a terminal in
EXACTLY the same way it currently works. We can take events out to match
them up to map/etc.. much neater. We can feed events in from GTK or
similar GUIs, without having to go near legacy 1970s terminal
technology, meaning that GTK (et.al) users can map Tab, Ctrl-I and
Ctrl-Shift-I, all independently of each other.

Having done this, I can then quite easily send you a patch to use
libtermkey if that is wanted, because then it will be able to recognise
Tab, Ctrl-I and Ctrl-Shift-I even over terminal from a
properly-configured xterm. But that is PURELY an extension mechanism. My
primary request is to fix the underlying queue mechanism.

This is an issue that has been dragging on for years now - 7 or 8 years
by my count. 7 or 8 years I have been sitting in #vim on Freenode; in
all that time almost every week we'll get someone or other asking how
they can map some key or other, that we have to keep explaining "No,
sorry, you can't. Not even in GTK.".

I am happy to sit and discuss it with you for potentially hours if
necessary. What I cannot do is "send a patch" - it's not a small simple
few-lines fix like that.

I think it is about time we ended this unfortunate situation, and
properly fixed the real underlying mechanism.

Yours, with my real name,

--
Paul "LeoNerd" Evans

[email protected]
ICQ# 4135350 | Registered Linux# 179460
http://www.leonerd.org.uk/

I would love to have this, too, if only because after so many happy years with
Vim (a heartfelt thanks to Bram and all developers!), I'm running out of keys
for my custom mappings :-)

Though I'm too ignorant of the internals (and can therefore only offer to do
some testing), I agree that this probably cannot be done in a single patch. But,
with the Mercurial repository, it should be easy to create a feature branch that
allows devs to collaborate and test this, and only have this merged in after
it's done (hopefully in time for Vim 7.4 / 8.0?).

-- regards, ingo

You can just keep the existing queue and make this work. It actually
already works for most keys, using a modifier sequence. Using
structures only makes it bigger.

Main problem is that this won't be backwards compatible, currently
CTRL-I is the same as Tab. I don't know how to make this work without
breaking backwards compatibilty.

Actually, in the GUI many key combinations do work. Not all though.

--
A cow comes flying over the battlements, lowing aggressively. The cow
lands on GALAHAD'S PAGE, squashing him completely.
"Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD

OK, well which ever way works.. I'm not fussy about the internals of
implementation. So long as it is capable of knowing that Enter is not
Ctrl-M is not Ctrl-Shift-M, I am happy.

:set oldterminalmode

:map <Ctrl-I> <Tab>
:map <Ctrl-M> <CR>

For that matter, make it default to this 1970s-like legacy behaviour and
if people want the ability to map newly-recognised keys,

:set newterminalmode

Well, then lets make them all work. Please?

Seriously - I am not the only person here. There is a constant stream of
users on #vim, every week you can guarantee someone will want to map
Ctrl-I or Shift-Space or Shift-Enter or some other key combination that
seems perfectly logical to any outside observer, to want to map. Every
week we have to tell them "no". I'm getting tired of it, when there
isn't any technical reason why it can't be done, only "it hasn't been
done this way so far".

So can we please have a plan? If you care deeply about retaining
1970s-like semantics, I'd be happy for a compromise solution. Some
boolean setting that defaults to a value whereby these aliases do occur,
where Ctrl-I and Ctrl-Shift-I -are- synonymous with Tab. But implemented
in a way where I or whichever knowledgable user can say "Yes vim, I know
that Ctrl-I used to mean Tab in the 1970s but I would like an extra
Ctrl key now and 'I' looks a good choice - let me have it". And then I
could toggle setting, and map it. Yes, I'm aware that not -every-
terminal can represent that. Yes I'm aware if I ever go use a real DEC
VT220 glass teletype I've now lost the ability to type it. But that
should be -my- problem - right now vim pointlessly restricts a whole
selection of possible key mappings from GTK or Mac OS X or Win32 or
proper xterms or all sorts of other places, simply for the
backward-looking legacy of retaining compatibility with what the
behaviour used to be in a time before many vim users were even born.

I think it's about time we opened this up a bit, and allowed users to
map the keypresses they want to.

Oh, and I'm not sure I follow your logic here.

Yes, it would make it bigger. By maybe bytes. Consider

struct keypress {
uint16_t flags; /* Unicode or special */
uint16_t modifiers; /* A bitmask */
uint32_t codepoint;
};

that's 8 bytes. As compared a single byte keypress that's 7 bytes
bigger. Less so, compared to e.g. a 3 or 4 byte modifier prefixed
sequence. Not a massive growth here. Lets be really pessimistic and say
we're using 7 bytes extra. Lets now say we're really memory-constrained,
and can only afford a single extra kilobyte of memory. That's still 146
keypresses of queue we can afford. Is vim's queue that big? Can we
really afford no more than a single kilobyte?

Ofcourse, this was a simplistic structure. If we care more about saving
memory at a slight CPU overhead we could

struct keypress {
unsigned int is_special : 1;
unsigned int modifiers : 10;
unsigned int codepoint : 21;
};

that's 4 bytes, to store the same information, noting that Unicode is
only a 21bit code space, a Unicode/special flag, and storage space for
10 modifiers.

4 bytes.

Those variable-length-encoded prefixed keypresses are getting on for
that long anyway. Plus now we're in a fixed-length encoding, so perhaps
we've saved CPU time and associated code space to store the program used
to encode and decode them.

I'm not sure an argument of memory consumption can be made against using
this encoding scheme.

Plus, it's only the queue of incoming keypresses - that queue isn't
going to stay very big for very long.

It's not just the input queue that's in question here, it's everywhere
in Vim where keypresses are represented. For instance, the right hand
sides of mappings are not primarily characters, but lists of keypresses.
They need the same amount of expressive power to work properly.

When macros are recorded, registers, which usually are primarily lists
of characters, are used to store keypresses. Likewise, for feedkeys() to
work, its input, a string, needs to be able to represent keypresses.

And I'm sure there are plenty more subtleties.

The bottom line, though, is that changing to a struct-based approach
could make the job absolutely huge, requiring reworking and/or
redesigning how maps, registers, etc. all work. And it might not even be
possible since, e.g. registers need to be able to do both characters and
keys. It would be much simpler to extend the current approach which uses
an 'escape mechanism' for representing special keys. Just hit qq and
type some arrow keys, q to end recording, and then inspect register q
and you'll see what's going on here. Vim uses a byte <80> followed by
two more bytes to represent special keys.

IMHO, what needs to be done is for this to be tidied up and made
consistent and documented so that these codes can be more readily
generated, interpreted, and even viewed and understood by users (e.g.
when I do :registers, it would be nice to see <S-Up> rather than
<80><fd>^D, when appropriate). Included in this is a clear specification
of what control characters mean, e.g. does ^I (<09>) mean tab or
control-I? And which of those keys can or does use the escape mechanism?
Then the behaviour of mappings needs to be defined--if there is a
mapping for ^I (<09>) and I push tab, will it be triggered? If in a
terminal which can't distinguish control-I and tab, and a ^I is
received, should the mapping for Tab or control-I be triggered? If
there's a mapping for ^I as well as Tab, which has precedence? All these
kinds of questions need clear answers, and sensible specifications and
design need to address them, avoid ambiguity, and take care to require
as little as possible work for users, plugin authors, etc. to update
their code and mappings.

Then the input code needs to be reworked in all the GUIs and in the
terminal handling to generate the appropriate internal codes,
consistently across all the different GUIs, etc., in line with that
specification/design.

Ben.

> And I'm sure there are plenty more subtleties.

What I see here, if done, is a potential new major version of vim.
Such a new version may trade off backward compatibility for up-to-date
technology and/or design constraints.

Since the changes may be huge, I concur with the suggestion of
creating a branch, and having discussions *and* documentation on its
evolution.
You raised some very interesting questions, and I hope that other
people who know about the internals give their point of view too.

As far as memory usage is concerned, I am now very egoistic and
believe that I have enough RAM to run vim. If I don't, then I'll use
vi, or an old version of vim.
I used to collect and use old hardware, but I'm now getting my job
done and bringing home the bacon thanks to vim on machines with more
memory that I could think of not so long ago…

Anyway,

Cheers,

P!

--
Français, English, 日本語, 한국어

:set compatible does suggest the possibility for more similar options,
ways to optionally extend the featureset beyond its existing capability,
without upsetting people who rely on the current behaviour, limits and
all.

Another thought occurs to me: UTF-8 sequences vs. Alt keys.

I don't know if it's an accident of implementation, but I have never
been able to get them all working. Lets consider this example

é is U+00e9 in Unicode, encodes to two UTF-8 bytes C3 A9

C is U+0043 in Unicode, encodes to one UTF-8 byte 43

If vim simply uses an 8th-bit-high to indicate Alt+ prefix on simple
ASCII, then what does

C3

mean? Is it the first of two UTF-8 bytes, or is it Alt+C ?

Alternatively, perhaps we'll represent it as a two-byte sequence with an
<Escape> prefix. Now we've conflated someone typing <Escape> C from
someone typing Alt+C. Again, in the 1970s that may be what it was, but
these days it's far from clear from GUIs, that this behaviour is what's
wanted. Again, a simple 'set' option should default this:

:set noaltisescape (defaults to :set altisescape)

The reason I suggested a simple neat orthogonal structure representation
is that it can disambiguate all of the following cases in a clear
logical way; a way that can be easily fed from GUI events, and fed into
map tables or other internals.

Tab { 1, 0, KEY_TAB }
Ctrl-I { 0, MOD_CTRL, 'i' }
Ctrl-Shift-I { 0, MOD_CTRL, 'I' }

Escape C { 1, 0, KEY_ESCAPE }, { 0, 0, 'C' }
Alt+C { 0, MOD_ALT, 'C' }
é { 0, 0, 0xe9 }

Yes. And the scheme I created is no less expressive than the byte
queues. In fact it is more expressive, able to disambiguate neatly
situations that cannot normally be represented - see also my other mail
on the thread a few minutes ago.

http://groups.google.com/group/vim_dev/msg/7bb4cc9b127cddb3

Yes - again, my point of these structures is that they exactly _do_
represent these keypresses.

But every character -can- be represented as a key - namely, the key that
generates it. Let me again repeat my structure:

Any character has a Unicode codepoint (cp). That's represented by

{ 0, 0, cp }

Any keypress has a symbolic key number, taken from some arbitrary
enumeration - I care not where.

{ 1, mod, key }

Now, we can also represent those modified Unicode keys such as Ctrl-I
and Shift-Space which previously were impossible:

{ 0, MOD_CTRL, 'i' } /* Ctrl-I */
{ 0, MOD_SHIFT, ' ' } /* Shift-Space */

OK - well, if you feel confident that the existing prefix-escape
mechanism can completely an unambiguously represent all these possible
keystrokes, then sure, that way might result in less code change
overall.

But do make quite sure it can represent them. Specifically consider the
two tricky special-cases I suggested - repeated here again:

> Then the behaviour of mappings needs to be defined--if there is a

I propose a pair of boolean settings, with the following defaults:

:set nomodifiedunicode
:set altisescape

Under these settings,

:map <Ctrl-I> ... vs :map <Tab> ... shall have the same effect, each
overwriting the effects of the other; last one wins.

Pressing the Tab key or Ctrl-I shall both invoke the last mapping
registered.

:map <Escape>C ... vs :map <Alt+C> ... shall behave similarly, each
overwrites the other. Typing either key sequence will invoke the
last map to be registered.

If a user decides "I want to use those extra Ctrl- keys", they can set
in their .vimrc

:set modifiedunicode

At this point,

:map <Ctrl-I> and :map <Tab> shall fill two -different-
slots of the mapping list, and typing either key will activate the
indicated mapping.

If a user further decides "I want the Alt modifier to not mean Escape
prefixing", then

:set noaltisescape

At which point,

:map <Escape>C and :map <Alt+C> shall also be different.

The only remaining ambiguity to be answered is, what happens in the
following case:

:set nomodifiedunicode
:map <Tab> ONE
:map <Ctrl-I> TWO
:set modifiedunicode

now press Tab or Ctrl-I

I don't have an easy answer for this case; no particular behaviour jumps
out at me as "obviously correct".

I propose that the structure approach makes this simpler. It is now
clearly obvious how e.g. GTK should fill in these structures, as GDK
keypress events already have unicode / modifier bits or symbolic keys.
It's also obvious how to render such a structure as a string, possibly
wrapping <> around it, possibly prefixing C- or A- or whatever as
appropriate.

(( This sort of behaviour is already implemented by libtermkey, even
going so far as to support

char buffer[256];
termkey_snprint_key(tk, buffer, sizeof buffer, &key, TERMKEY_FORMAT_VIM);

for exactly this purpose. :) ))

Yeah, there was never anything lacking in your scheme in terms of
expressive power. It's just that it's not only the input queue that
would need to use it. It's a lot more.

Yes. But what happens when you then edit that macro by putting the
register into a buffer, changing it, and yanking it again? This is not
uncommonly done. How should the registers be stored in .viminfo? How do
you write the input to the feedkeys function as a string in vimscript?
Etc.. These are the kinds of issues I was trying to raise.

When there is one. Or a codepoint in some encoding. Yeah, your struct
does allow for that. The struct isn't inexpressive, it's just that it's
big, and if it needs to be used in a LOT of places (which seem to just
be increasing...input queue, mappings, registers, strings, buffers?!),
suddenly not just one thing is getting a bit bigger--a lot of things are
getting bigger--potentially a lot bigger! If every time you yank text
into a register it gets 12 times larger, that may cause some problems!

Well, a codepoint in whatever encoding you are using.

Yeah. The benefit here is that different keyboard layouts can be
represented, e.g. keyboards which have single keys for accented
characters can represent those keys with modifiers, easily.

I think with some careful design it could, which may well require some
reworking of how it's currently done. There is the issue, of course,
that <80> is a valid character in some encodings, too, and I don't know
if this is accounted for. It needs to be. So maybe the escape mechanism
needs to be a bit more formal (if it can be--but maybe there's no way
around this, or maybe that's a longer-term consideration). Definitely
the issues have to be bashed out. I still think this would be the
easiest and best way forward, though. But it's far from my call!

Regarding that whole meta issue that you raised earlier--I think, yes,
using an 8-bit-high representation for meta is completely out of the
question. Part of the input code should be transforming that to a proper
Vim-internal escape sequence for terminals that use it. Same for other
things. The input code should transform input into Vim's internal
representation which should be carefully designed not to be ambiguous,
etc., but which, nevertheless, is at its most fundamental, a byte stream
so it can be used in the input queue, mappings, registers, strings and
even buffers.

Yes. This all has to be carefully accounted for.

Maybe the way to do it is to have printable characters represented just
as printable characters, 'default' control characters optionally just as
themselves (e.g. <09> for tab), to use when a terminal that can't
distinguish between this and Ctrl-I is used, and (whenever possible),
use <80>+flag_byte+character for control/special keys. One of the
flag_byte flags could be a 'special' flag, which means the 'character'
in the third byte would be some kind of mnemonic for the key (e.g. u for
the up arrow). When the key is not 'special' the character it produces
would be the third byte (and perhaps later for multibyte characters,
e.g. if ctrl is used with an accented character on keyboards that have a
specific key for this). How all this works depends a bit on how the
keyboard interfacing works--maybe it should just use keycodes if
characters are not readily obtainable. And maybe you need two flag bytes
if there are more than 7 modifier keys that need to be distinguished.

What should Vim do in a terminal that can't distinuish between those
keys when this option is in effect? This is the same question as below,
really.

Maybe we need an annotation to the mapping, like <default> or something
that says "this mapping is acceptable to use when we're not sure if this
exact combo was pressed, but we got a code that 'could be it' in some
terminal."

I think there might already be an option for this. Or something in
termcap? I seem to remember reading some stuff about it in the Vim
manual. I guess, though, the 'non-alt prefixing' of Vim probably means
to use 8-bit-high for meta, which we already said, is pretty useless.

I'm not sure two options are needed. Probably a single option suffices
to deal with both situations.

I agree. If we go with the <default> annotation to mappings (preferably
with a better keyword), I guess that is the one that wins and takes
precedence. It may not even need to delete the other ones, so both slots
could remain active, but it would just run the 'default' one.

That's good for mappings, and I guess the \< escape in vimscript
strings, but it doesn't help for registers (which can end up in
buffers), etc..

Maybe it's a sensible 'middle point', though, for the input queue. The
GUI specific code could go into a structure like this, which generic
code can then translate into internal byte-stream representation as
appropriate, and then that byte-stream representation is the standard
throughout the rest of Vim. Basically the structure would only be used
as a mechanism to deal with the code that deals with hardware
keypresses, making it consistent between GUIs, platforms, etc..

> termkey_snprint_key(tk, buffer, sizeof buffer,&key, TERMKEY_FORMAT_VIM);

Ben.

Hi.

Wouldn't be it same as now, only used more often? There is already a
possibility to write "<F4>" or "<S-Space>". The only problem is that, at least
the second, we can't press strongly enough to push it to vim :-)

But, on the other hand, you are right there still will be (and must be)
ambiguities. We can't do anything about that, in general - it's a user who must
decide how does he want to understand his keyboard.

For example: I, as Czech, have some Czech accented letters accessible via
modifier+key on my keyboard. To make the example more specific, think about
Mod5+s as "s with hook" (U0161) and redefined my keymap so Capslock key acts as
Mod5. It's up to me, and only me, if I define some vim mapping as

:map <Mod5+S> ...rhs...
- or -
:map <U0161> ...rhs...

Both do the same on my current keyboard but start to behave differently if I
change my keyboard setting in X window system, of course. If I switched to
another kind of Czech keyboard (called "typewriter one"), <U0161> appears at a
key of "number 3" and Mod5 would be on right Alt or not defined at all.

As I wrote above, we can't do anything about that, as far as I know.

Milan

--
Milan Vancura, Prague, Czech Republic, Europe

2011/2/23 Stephen Lee <[email protected]>:

> ...

Hum, as a user of a home-made variant of dvorak-bépo (because my
keyboard, a kinesis, is *not* practical to use with standard bépo), I
hope it's going to be easy to set up and modify such a setting,
because you are not going to give an exhaustive list of all layouts…

>> [...]

> it with a structured keypress queue. [...]

> libtermkey [...]

> properly-configured xterm. [...]

So, from other responses, it seems (to me) that the problem isn't
necessarily in the input queue, per se. "Special" keys are already
represented in some internal format (perhaps not-well-documented, since
it's internal, but still represented). (example of undocumented nature
is the K_IGNORE special key trick that stopped working when its keycode
was either changed or eliminated [1]).

Taking that as a given, I'm going to ignore that aspect of this issue.
My question is about libtermkey. I was curious enough to download it,
unpack it, `make` it, and run ./demo. And, surprisingly, given that it
seems to be the favored example, pressing <Tab>, then <Ctrl>+<i>, then
<Ctrl>+<Shift>+<i> gave me the output sequence:

<Tab>
<Tab>
<Tab>

So, I'm wondering: what does libtermkey bring to the table?

I tested under both rxvt-unicode (my preferred terminal emulator) and
uxterm (which, AFAIK, is a "properly-configured" xterm). If
"properly-configured" actually means "specially-configured", how would
using libtermkey be anything less than a huge PITA when working on
multiple systems that aren't so configured? Or is this a
chicken-and-egg problem? (Nothing is configured to send anything other
than 70's-style "<C-i>" == "<Tab>", so nothing ever will be.)

--
Best,
Ben

[1] http://groups.google.com/group/vim_use/browse_thread/thread/8c535e5cf2b35f63/319b3113716e52d5

Sorry, but this is not what we are discussing. Vim already has this
functionality. As you discovered, it can work through what it calls
'keymaps' (:help keymap). It also works with a number of different input
methods on different platforms, particularly when using Gvim.

And there is even better than this already: Vim's 'iminsert' option
(:help iminsert). Of course, you can write mappings (as I have done) to
change the 'keymap' option between values, too.

You are miles ahead of us, Stephen! It is indeed possible that this kind
of thing could evolve. In fact, the functionality for this part of it is
already there. What we need, though, is a way to get the structured
kepress information into vim in the first place, so that keymaps (and
more importantly, mappings) can refer to them.

Mappings are more important, because what we are discussing isn't really
about text input, but about shortcut keys. At the moment, Vim sometimes
can't tell the difference between Tab and Control-I (and a lot of other
things). And because Vim sometimes has no way of telling the difference,
at the moment, it doesn't even bother to notice the difference when it
*can* tell. We want to change that.

After we have changed that, then your idea of having multiple keyboard
layouts, some for when these keys can be differentiated, and some for
when they cannot, may become useful. I hope we won't need that idea,
though, but will be able to write one keymap file that works well in
both situations.

But that is for later.

First we need to get all the keys recognised as properly as possible.

Ben.

OK, so you seem to be proposing changing the way the q and @ commands
work, so that instead of exposing internal key representation, they
record and play back <> notation. They would also have to represent < as
<lt> of course. The feedkeys() function would also need to be similarly
changed. It would be hard to know what to do about \<xxx> escapes in
strings (:help expr-string)--whether to make it a no-op or do the old
thing, or something else.

So this would be a backwards-incompatible change. It would not just
affect places where special keys are currently represented in internal
notation, but everywhere macros are used. Now there are currently
probably quite a lot of places where internal keys are represented, and
changing the internal representation as I suggest, may affect them; but
if the current representation is sparse enough, perhaps it can be done
in a backwards compatible way. There may be quite little impact, though,
as wherever possible, plugin authors use <> notation anyway; it is only
when they cannot do this that they resort to internal
representation--and even then, they often use \<xxx> escapes when they
can. There is no way your suggestion could be backwards compatible, I
don't think, and it could potentially affect a lot more macros (e.g. all
macros that insert xml and html, so use < and > a lot). It could break a
lot of plugins as well as home-grown code.

Now, that doesn't rule it out, and I think it is probably a way to solve
the problem. But I think it would need more careful thought than the
alternative option of changing the internal representation while keeping
it byte-stream based. I also think it would be much more work, because
it requires not just macro recording, but every problem involving keys
to be separately solved from scratch, and every one would probably be
backwards incompatible. And I think Vim and Bram have a history of
maintaining backwards compatibility (we still have :set compatible!) so
it may be that a more compatible change is preferred.

This would be easier if we actually had a solid 'new design'. We don't.
We just have a bunch of rough ideas. One of those ideas had a drafted
structure and suggestion that it be used only in the input queue, where
the memory inefficiency would not be a concern. I have pointed out that
there are many other problems to solve: right hand sides of mappings,
feedkeys(), registers for macro recordings (which indirectly affect
buffers, as registers must be able to be put and yanked), representation
in strings, and more (I just thought of the :normal command, and I'm
sure there is still more...). I pointed out that if you use the
structure to solve all these problems, memory efficiency is most
certainly a concern.

So I think I have been quite specific. I think I've been specific enough
for the purpose of this discussion.

Yep, this seems to sum up your approach nicely, which is basically
design and solve every key-related problem from scratch. The most
specific thing you have done is suggested a different way to using the
structure to solve at least the macro recording problem.

My response, to both that specific proposal and that general approach is
that I don't think it is a good way, because it is massively
backwards-incompatible and a lot of work.

I suggested earlier a rough idea in response to the structure proposal
of extending/changing Vim's existing mechanism. That was also
springboarding off Bram's comment that he thought this could be done
using the existing mechanism. The existing mechanism is there for all to
see, so is already a much more well-developed design than any of the
alternatives. My rough idea of how it could be extended is in a previous
post, too. I think it would work, and be more efficient, less work, and
more compatible than the other directions we have explored.

But it's not zero work, and I'm not at the point where I can volunteer
to do work on it. Even putting together a careful detailed proposal is
probably not a contribution I can commit. I'm more than willing to
discuss and brainstorm ideas, though, and comment on any serious
proposal someone wants to make.

In the end, it's up to Bram as maintainer, and whoever's going to put
the (large amount of) time into implementing this, what goes ahead,
though.

But as a community member, I'd prefer it if it affected my code and
plugins as little as possible, and I consider it essential that Vim
doesn't become significantly less efficient because of this. And if it
were me considering an implementation, I'd certainly be looking for the
option that was the least work! YMMV.

Ben.

It parses CSIs, the way that xterm et.al. send modified keys.

Start an old vim. Go into insert mode. Press Ctrl-Left.

xterm sends CSI 1;5D

Without a CSI parser, using only a 1970s-style terminfo-driven prefix
hunting algorithm, there is no terminfo key sequence matching this. So
vim gives up. It then sees Escape [ 1 ; 5 D. The upshot is that vim
leaves insert mode (Escape), does something unimportant with [ 1 ; then
deletes 5 lines of text (5D).

With a CSI parser such as libtermkey, this is recognised as a real CSI D
call, with the parameters 1 and 5. This is reported to the application
as Left (CSI D), with the Ctrl modifier (2nd param being 5).

The primary thing that libtermkey brings, is not blowing up and deleting
5 lines of your text just because xterm happened to send a valid CSI
sequence that isn't in the terminfo.

rxvt uses its own, totally-incompatible encoding scheme for modified
keypresses. This scheme is different to any other terminal that has such
abilities (all the others follow xterm's CSI scheme). It is also
incompatible with ECMA-48 (whereas xterm extends it), inextensible
beyond Ctrl and Shift, and arbitrary (happening to pick three ASCII
characters seemingly at random, to represent the Shift, Ctrl and
Ctrl+Shift states).

Specifically-configured means

modifyOtherKeys: 2

Except that on its own now breaks most other applications, because xterm
is too keen to use the CSI encoding scheme on "normal" keypresses, such
as Ctrl-D. xterm doesn't yet send the correct and useful sequences here,
but it is not difficult to fix its code to do so - I have been in
occasional contact with Thomas Dickey on this issue; I need to poke him
again sometime soon to check he's properly looking into it.

What is needed is a hybrid scheme, whereby the simplest key encoding is
used in a nicer way; where Ctrl-D would send the single 0x04 byte (so
kernel's pty driver can recognise "here's an EOF"), but Ctrl-Shift-D
sends CSI 68;5 u to encode it correctly.

Rightnow, xterm will send the same byte for Tab, Ctrl-I or Ctrl-Shift-I.
Setting modifyOtherKeys to 2 will send real keypresses that it can tell,
but now everything else cannot.

Ultimately, yes. It is chicken and egg, in order for -everything- to
work in a happy shiney way. But even without terminal changes,
applications can benefit from using libtermkey now.

So to answer your question on "how would using libtermkey be anything
less than a huge PITA" - it isn't. It already recognises, via its
terminfo-driven parser, every keypress that terminfo/curses already
recognises. It cannot be worse. It is strictly better than, because of
the full CSI parser I mentioned above. Even if it cannot pass on that
information to its containing program, it can at least parse it. If
libtermkey was used by some application that has no concept of modifier
keys, then modifiers would just be ignored; i.e. that pressing Ctrl-Left
would simply be read as a Left keypress, not resulting in a beep and
deleting 5 lines of text (vim), or beeping and appending ;5D to your
shell prompt (readline, irssi, etc...).

Furthermore it is much more future-proof. Just as in 1970 nobody
considered modifier keys, right now nobody has, say, pressure-sensitive
keyboards, so nobody bothers reporting pressure information. Maybe in
another 20 years time we'll all have keyboards that can recognise a
gentle vs. hard press of a key, or multi-touch with two fingers, or
whatever. It would be quite natural to use the 3rd CSI argument, say, to
encode this.

Up CSI A
Ctrl-Up CSI 1;5 A
Forceful-Ctrl-Up CSI 1;5;3 A

A terminfo-driven string prefix parser will of course blow up here. The
code in libtermkey, right now today, will not blow up. It won't
-understand- that 3rd parameter, it cannot -report- that to its
containing application, but it will correctly parse it to ignore it -
reporting to its application the press of Ctrl-A, by ignoring the
Forceful part.

Using libtermkey today leaves terminal developers of the future the
freedom to extend the CSI parameters in whatever way they see fit,
knowing that if old applications can't make use of this new information,
they are not at least confused and inconvenienced by it.

Hope that helps,

>> [...] I'm wondering: what does libtermkey bring to the table?
>
> It parses CSIs, [...]

> vim gives up. [...]

The whole situation is pretty arbitrary. E.g. ./demo's output under a
modifyOtherKeys:2 UXTerm, compressed to one row, for <Shift>+ the top
row of my typical US 105-key or variant:

<S-~> ! <S-@> # $ % <S-^> & * ( ) <S-_> + <Backspace>

Why the Shift modifier on ~, @, ^, and _, but not !, #, $, etc?
(Guessing hysterical raisins.)

Didn't realize ECMA-48 existed, though. Thanks.

Cool. Works as advertised (i.e. currently not in anything other than
the `demo` program, but still interesting to see the ability is built
in).

How do <Tab> and <Ctrl-i> fit into this scheme? What's the "simplest
key encoding"?

Thanks. This paragraph combined with the first section about parsing
unknown CSI sequences is plenty convincing. (trimmed rest)

--
Best,
Ben

Of course, Vim can already do this to an extent.

:help keycodes
:help xterm-8bit
:help +termresponse

It's only keys like Control-I which are still indistingishable from keys
like Tab (and Enter, and Backspace, and I think shifted Control keys).
And perhaps that can't be fixed in the terminal now, but it can
certainly be fixed in the GUI.

And in doing so, the notation and representation for it can be tidied up
so there aren't a bunch of internal ambiguities and at least all the
remaining ambiguities are external ones that we are stuck with anyway
(and with luck will go away one day).

And AFAIK, Vim doesn't deal with this. Does libtermkey or does it only
do CSI stuff?

Ben.

This can't explain ~, but...

My guess is that these are keys which are low enough in ASCII that they
can be control keys. There should be 32 of them. A-Z makes 26, plus the
three above which are @ ^ and _, then there's [ ] and \, and that's it.
I guess you can test the theory and see if { } and | appear with shift
modifiers--though I guess they may not, since they actually generate
different characters. Still, would be interesting to know.

Maybe ~ by some other convention does something special with control.
IIRC, it is right down the other end of ASCII, at 126 or something.
Maybe it's equivalent to another one of the keys, if ctrl-a is the same
as ctrl-A, maybe ctrl-~ is equivalent to whatever is at
codepoint...126-32=94, which is definitely one of those symbols above...
It's strange, in my terminal, pressing ctrl-~ seems to do nothing,
whereas control with other 'non-control' characters makes a beep.

At any rate, since it's only these few keys which can be control
characters in the first place, it makes sense that these have the
special treatment, and the others don't.

Ben.

It doesn't at the moment, but if someone wants to do the research into
what sequences it sends, and write some code in the library (similar to
driver-csi.c and driver-ti.c), I could add that in sure.

I don't think anybody has refused the idea, and I don't think anybody
wants to keep Vim in the 1970s. We all want this. In fact, people who
don't want this probably wouldn't get involved in the discussion--unless
we looked like we were going to introduce some horribly backwardly
incompatible change that would affect them nastily.

So I would encourage you not to view this discussion as opposition to
the idea. Quite the reverse. This discussion is actually an important
part of making this idea happen. We need to discuss these things so we
can do it the best way possible, without *needlessly* breaking backward
compatibility, without *needlessly* or *significantly* lowering
efficiency, and without *unnecessarily* wasting people's time.

Unless you saw something that I didn't see, I don't think there is any
resistance that necessitates a compromise.

I think it goes without saying that we'll need some kind of
compatibility mode, whether it's by means of a +feature, an 'option', or
just some carefully thought-out behaviours. Some of these issues have
already been raised and solutions brainstormed (including your recent
suggestions, as well as in earlier mails by me and others).

As I said, I don't think there is any resistance to the idea. We're just
discussing how to do it. As someone suggested earlier, it would be best
to discuss and draft some documentation for this stuff before doing the
hard implementation work. Having something solid will help, and a
checklist of issues/concerns along with their solutions.

So, to move it to the next stage, is anyone in a position to volunteer
to write up a more specific design (which probably needs to be written
with reference both to this email discussion and the Vim source code)?

Also, is anyone in a position to volunteer to help with implementation
once we have a design?

Ben.

Yes, it was a much better reply, thanks, Stephen. And welcome to the
list. You picked a good lively discussion to get involved in!

We should be able to do it at a higher level than this, and avoid having
to deal with scan-code -> character mapping. Also, scan codes simply
aren't available when running in terminal mode. Every GUI will be
different, and different terminals will need different handling, but we
should be able to get something structured from the OS/window
manager/terminal which details the key pressed and its modifiers. I
don't think this part of things will be a problem. And since translation
from the GUI-specific form to Vim internal form will happen very
quickly, efficiency won't be an issue here either.

GUIs should be able to get the information, and some terminals (e.g. via
libtermkey as has been discussed in another part of the thread).

The trouble with the structure is that it is big, and contrary to the
original assumption, it't not really useful to restrict its use to the
input queue. We need a solution that will work in other parts of Vim
efficiently--maybe a byte-stream solution, maybe an escape character
followed by a packed structure, maybe something we haven't thought of
yet. I think the byte-stream method will work well, and is closest to
what Vim already does so least work.

It's not out of the question, but it would require significant changes
to a lot of Vim, I think, and is not what really how you would expect
them to work. Also, a massive chunk of disk space isn't all that much an
improvement than a massive chunk of main memory!

Cheers,

Ben.

[...]

So the problem is that many users expect CTRL-M to have the same effect
as Enter, just like people use CTRL-[ instead of Esc. And a few people
would make the CTRL-M act different from Enter, and CTRL-[ different
from Esc.

First problem is to actually detect what key was pressed, in most
terminal emulators this is not possible. In the GUI we can. Changing
terminal emulators to support this and making this work with Vim is a
separate issue, I'll not go into that here.

Then we need a way to make the extra information available to be used in
mappings, without breaking it for users relying on the current way.

Some things that are no acceptable:
- Have a setting to enable "the new way". This will break existing
stuff and make users pull their hair out because they don't know this
setting exists. Forget it.
- Change the input queue from a stream of bytes to some list of structs.
This isn't adding any functionality and breaks all kinds of mapping
and termcode handling, register execution, etc. Forget it.

What we can do is extend the existing modifier byte sequence. This is a
bit tricky, but it should work. So we add a new byte sequence with the
raw key encoded, plus modifiers. Thus for CTRL-[ you get the [ key with
the CTRL modifier.

When this new modifier is not mapped, it is discarded and the following
translated codes are used normally, just like now. Thus without any
mappings that use the new modifier it is guaranteed to work as before.
When executing a register where the new modifiers are missing it works
just like before.

When a new modifier is mapped, it is replaced by what it's mapped to, AND
the following old style byte sequence is consumed.

An alternative is to let the termcap handling translate the new modifer
into the old style key. That works because mappings are applied both to
the raw escape sequences and on the key codes they are converted to.

We also need a way to specify mappings with the new modifier, perhaps
using a special modifier X, thus you could do:
:map <X-C-[> :echo CTRL-[<CR>
:map <X-C-I> :echo CTRL-I<CR>

--
Not too long ago, compress was something you did to garbage...

/// Bram Moolenaar -- [email protected] -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

Another way to extend this might be to switch from a byte queue to a
short integer queue. The lower 8-bits would be as it is now; the upper
eight bits would encode modifier keys (shift, ctrl, alt, meta, whatever).

Regards,
Chip Campbell

I think sticking to the byte queue is best. So much in Vim relies on
text being represented in either a single byte encoding or UTF-8 (e.g.
when you use UCS-2 or UCS-4, Vim will use UTF-8 internally, I believe).
It will break less if we stick to a byte stream with escapes, as now,
IMHO.

Ben.

The current special-key encoding is a bit clumsy. On the other hand,
there are situations where the queue can get very long (recursive
execution of mappings or "1000@@"). Using 16 or 32 bits for every
character is not attractive.

One way to solve this is to use up to 128 bit integers, making sure that
the higher bits are usually zero, and then use variable length encoding.
ASCII characters should then still take only one byte.
http://code.google.com/apis/protocolbuffers/docs/encoding.html

Nevertheless, what gets encoded in registers must still be a byte
sequence. And we need to be able to store double-byte and multi-byte
characters. Thus I don't think this will make things simpler.

--
This is the polymorph virus! Follow these instructions carefully:
1. Send this message to everybody you know.
2. Format your harddisk.
Thank you for your cooperation in spreading the most powerful virus ever!

That's OK. For them it can be mapped. Or even made default.

Changing terminal emulators is a done task; xterm has supported this
since 2008. In fact I asked Thomas Dickey to clarify this for me, and he
informs me it's all present and working.

Really?

Is that what we'd have said right back at the beginning of Vim? On the
subject of the 'compatible setting:

"Adding a setting to enable all the vim-incompatible changes, is too
complex for users to handle."

Right now we have users _all the time_ pulling their hair out _because_
such a setting does not exist. Including myself. I would much rather
have to support hoards of confused users in #vim by saying to them

"Yes, sorry your vim can't recognise that keypress now, but set this
setting and then it will"

instead of

"No, sorry your vim cannot recognise that keypress at all"

If you would prefer to support people by telling them it is not
possible, rather than telling them how to make it possible, please come
to #vim and explain that to them. Every week.

OK, well that's fair. It was intended as an implementation detail in any
case. I hadn't realised it would be possible with the existing byte
queue.

As long as the two triplets of keypresses I suggested originally can all
be represented uniquely, and without reference to timing information in
the Escape vs Alt+ case, then I'm happy with whatever internal
implementation makes it happen.

<snip long description>

That all sounds a bit long and complicated, and I'm not sure I see
whether it's necessarily any easier to implement than the arguably
much-neater and more forward-compatible queue-of-structures idea I had
originally. But again, that's all internal implementation details -
whatever makes it work, I'm happy with.

X for "eXtended"? Could work, though would be useful to check that
doesn't collide with any named modifiers on any existing system. Perhaps
some non-letter punctuation symbol? Though nothing suitable comes to
mind.

That sounds like a buggy implementation. Due to all the vagueness of
what shift means in the context of Unicode (is ! really ! or is it
Shift-1) my specification called for explicitly ignoring the state of
the Shift key in the terminal, at the time the modified Unicode sequence
is created. If that isn't happening, I vote bug.

It's essential in these matters. :)

Once you read the spec part for CSI, you realise how short and neat the
entire scheme really is, how extensible it is provided you actually have
a real CSI parser and not a dumb string-prefix matcher a.la.
termcap/terminfo/curses combinations.

I am also recently informed by Thomas Dickey that

formatOtherKeys: 1

sends a slightly nicer CSI u rather than CSI 27~ encoding

<Tab> 0x08 (ASCII HT)
<Ctrl-I> CSI 105;5u (i == 105 decimal in ASCII)
<Ctrl-Shift-I> CSI 73;5u (I == 73 decimal in ASCII)

Note that, due to what I said above, we have to ignore the shift key in
these CSI u sequences; the distinction between Ctrl-I and Ctrl-Shift-I
is simply that one is lowercase and one is capital.

Another thought occurs to me.

Run vim in xterm. It recognises Ctrl-Left, because vim handles
TERM=xterm specially to recognise this case.

Run vim in screen in xterm. It doesn't recognise Ctrl-Left, instead
interpreting it as Escape, 1; (ignored), 5C. Deletes 5 lines of text and
enters insert mode. Mass fail.

Run libtermkey's demo either in xterm or in screen-in-xterm. Observe
that Ctrl-Left works IN BOTH CASES. This is because it always applies
its CSI parser, so always understands xterm-alike CSI encoding of
Ctrl-Left (et.al.).

This fact -alone- should be enough to convince you. :)

The two triplets in question being

<Tab>
<Ctrl-I>
<Ctrl-Shift-I>

<Escape> C
<Alt+C>
é

Yes I am aware that on current byte-driven terminals there is no way
without using timing information to distinguish all these three cases in
the latter triplet here. This should not stop GUI systems from
distinguishing them however.

Specifically as to the terminal case, I have a solution to this, but it
requires the cooperation of terminals and applications to use libtermkey
or some equivalent code, so right now I don't want to distract the
argument. That is what derailed it last time when I tried this a couple
of years ago - I don't want that to happen again for now. :)

Despite its popularity, xterm is not "most terminal emulators". I think
this is a great move on xterm's part (having a more-or-less consistent
set of escapes for modifiers). But, it's still a chicken-and-egg
problem. Once it's in xterm, it takes a while (if ever) to filter down
to derivatives. And even under xterm the setting that allows this to
work isn't the default.

I almost didn't send this, since I don't think the lack of terminal
support should be a show stopper (If any terminal supports it, Vim
should use that support), but claiming that it's a done deal because
xterm added a feature to support it a few years ago is naïve.

--
Best,
Ben

So, instead of enabling a single setting (usenewkeymaps or whatever),
the :map command needs a new "Yes, do what I'm telling you to do"
modifier? This seems totally backwards to me. The "new way" will cause
problems with old stuff. Why is that breakage unacceptable? This whole
suggestion seems like a major-version change anyway (not a point-version
change, where backwards compatibility is sacrosanct). The stuff it's
breaking was caused by having to do a workaround in the first place.

Consider:

" {intent} should do {rhs}
:map {lhs} {rhs}

Where {intent} describes the intended binding (what key should be
pressed to activate it), and {lhs} is what was actually typed. In the
following chart, {around} indicates that the only reason that particular
{lhs} was chosen was to work around the limitation of key mappings.
{new-works} indicates that using that same {lhs} would still work under
the "new way". (^x indicates entering a literal control character)

{intent} | {lhs} | {around} | {new-works} | Category
---------+---------+----------+-------------+---------
<Enter> | <C-m> | yes | no | Enter vs Ctrl+m
<Enter> | <Enter> | no | yes
<Enter> | ^M | yes | no(?)
<C-m> | <C-m> | no | yes
<C-m> | <Enter> | ? | no
<C-m> | ^M | yes | yes(?)
---------+---------+----------+-------------+---------
<Esc> | <C-[> | ? | no | Esc vs Ctrl+[
<Esc> | <Esc> | no | yes
<Esc> | ^[ | yes | no(?)
<C-[> | <C-[> | no | yes
<C-[> | <Esc> | ? | no
<C-[> | ^[ | yes | yes(?)
---------+---------+----------+-------------+---------
<Tab> | <C-i> | yes | no | Tab vs Ctrl-i
<Tab> | <Tab> | no | yes
<Tab> | ^I | yes | no(?)
<C-i> | <C-i> | no | yes
<C-i> | <Tab> | ? | no
<C-i> | ^I | yes | yes(?)
---------+---------+----------+-------------+---------
é | <M-i> | yes | no | é vs Alt/Meta-i
é | é | no | yes | (issue w/ 8th-bit meta)
<M-i> | <M-i> | no | yes
<M-i> | é | yes | no

Not sure if the chart is clear, but my point is:

I think that the people who expect, e.g., <C-i> to activate a <Tab>
mapping are the people who either:

1. Know what's going on behind the scenes, so are knowledgeable enough
to fix their mappings (fixing plugin mappings is a separate issue)

2. Were bitten by something that didn't work right, but found a
workaround and only use it due to inertia.

I highly doubt there are people who approach the :map command with the
*desire* that <C-i> or ^I would activate a :map whose {lhs} contains
<Tab> or that <Tab> would activate a <C-i> :map. This seems like a
clear case where backwards compatibility is undesirable.

The fact that plugins probably rely on this behavior is unfortunate, but
"Oh, that plugin's <Tab> binding is broken because Vim's key mapping
used to be broken, and so it was using the Ctrl-i workaround" seems like
a preferable position to be in.

--
Best,
Ben

This is regrettable, yes. But not entirely true.

Until xterm added the extended CSI notation using the 2nd parameter for
modifiers in e.g. CSI 1;5C for Ctrl-Left, no terminal in the world did
it.

Now, apart from rxvt which uses its own scheme, prettymuch every
terminal does it, and does it -that way-. These things aren't completely
set in stone, but they do take a long long time to filter around.

I'm very keen to try to shepherd people into one standard and
consistently-designed way of doing things. That rxvt does modified keys
differently is regrettable, and a consequence of splintered reinvention
of wheels. I'm very keen to try not to let that happen any more.

It's to this end I tried initially writing this spec for modified
keypresses:

http://www.leonerd.org.uk/hacks/fixterms/

It comes a little whiney and amateur at the moment, because I'm no good
at the social engineering side of trying to motivate a world-full of
1970's-era engineers into trying to fix a big problem. :) But the
technical details are there, awaiting someone with better writing skills
than I to neaten it up a bit.

Of course it does. Bram wouldn't have suggested it if it wouldn't work
at all....

I don't think we need to keep the same byte-stream representation as we
currently have, and I don't think that's what Bram meant. I think we can
change it. As long as it remains a byte-stream representation.

And I don't think it will be possible to do what we need without at
least extending it. At the moment, keys are a fixed length of 3 bytes
(one of which is the 'escape byte' 0x80, and the other two of which have
restricted values). There simply isn't enough value-space to do what we
need, and it's possibly not very future-proof even if we can squeeze it
in. It's also a bit of a mess, IIRC. It would be better to give it an
overhaul. And it probably needs to use variable-length sequences.

A thought occurred to me a couple of days ago: would it be appropriate
to actually use CSI as the byte-stream representation? Or something like
it. It is a byte-stream representation, right? And it's extensible,
right, so it is less likely to need a big overhaul in future? Does it
allow for 'private use' stuff? Because Vim needs a way to represent a
bunch of other events as keystrokes (mouse clicks, scroll wheel, its
magic script-local meta-character thing, representing the CSI escape
character itself, etc.). Also, is it easy to skip over CSI sequences if
necessary with simple logic?

I think there are some nice/helpful things that could be done
design-wise though. For example, Vim is often used with enc=utf-8 these
days. I think it would be wise to, if possible, ensure all key sequence
representations are invalid UTF-8 in their entirety. Then they could be
put in buffers and yanked back into registers without ambiguity
(hopefully), i.e. there wouldn't be the ambiguity of whether the escape
character was part of a key sequence that should be yanked as such, or
part of the buffer text which should be escaped upon yanking.

See, Vim already deals with the CSI character in its byte-stream code,
and escapes it, so some aspects of this are already somewhat accounted
for. But I wonder when precisely it is escaped, and whether at times it
is escaped when it shouldn't be, or not when it should be. It would be
good to design to avoid this problem.

(I happened to be hacking on some Vim code recently where this came into
play, so I noted it with interest.)

Ben.

Sure. CSI is intended for just this. A CSI sequence is matched by a
regexp

CSI [\x20-\x2f]? [\x30-\x3f]* [\x40-\x7e]

Rougly "possibly-one punctuation character, then a bunch of numbers, ; or
;, then finally a signle letter."

The "private use" area in CSI is the character range \x70-\x7e. One of
these, CSI u (\x75) is already in use by xterm and libtermkey to encode
modified Unicode characters. To my knowledge, the remaining are free.
Plus, vim could even use this encoding internally if it wanted.

Actually, that's exactly the issue that is being addressed. Vim
shouldn't have to address it -- in theory termcap/terminfo were
supposed to do this, but they didn't really do so well. Which is why
we now have difficulty telling the difference between <ESC>o and
<A-o>, as well as things like <ESC> vs <C-[>.

Ultimately the proposal is that we let the raw keyboard input (for
terminals) be handled by libtermkey, and it figures out if what was
pressed was <ESC> or <c-[> or <C-[>o vs <ESC>o vs <A-o>. It then
passes a stream of information to vim, which vim interprets properly.

And yes, by default there would be a mapping (either explicit or
pseudo-builtin) that treats <ESC> and <C-[> as the same, but that can
then be changed by the user, just as they can now with any other
mapping.

The only breakage I can foresee is if someone actually does a :map
<c-i> foo instead of :map <Tab> foo -- then, yes, you'd end up
with different behavior. I think this is within the realm of
acceptable change, particularly if it's clearly stated in the release
notes (and yes, I think this is a Vim 8 thing).

I actually agree -- it should be just the way things work. And if it's
done right then it won't break anything (beyond the above).

The termcode handling will be replaced by this, so that's not an
issue. Mapping is also not an issue beyond the above.

Registers are more of an issue. No doubt. Can we think on this to come
up with a solution? Perhaps the register needs to become more like a
:map, with special keys expressed specially? Otherwise I think we're
stuck since you cannot differentiate between <ESC> and <c-[>, and
there's no way to express things like <c-space>.

Which is precisely what libtermkey emits, and how it can differentiate
between ctrl-[ and alt-ctrl-[, etc.

Really what happens is that the terminal input buffer starts to look a
great deal more like events that the GUI versions get. Instead of
<ESC>o you get "o with alt depressed".

To be very clear -- some of this will depend on a terminal with the
proper support being used. But anything that's built on libvterm
already has that, as well as xterm. But if you don't have those then
moving to this new scheme should not break ANYTHING. Your term would
still send <ESC>o and libtermkey would tell vim it saw <o with alt>.

But if you do have a terminal with the right support, or you're using
the GUI... well, to quote a bit from :help design-multi-platform,
"Support all the keys on the keyboard for mapping."

It would be great to get closer to that.

Tom Sorensen