New in C2y is an operator that does something people have been asking us for, for decades: something that computes the size in elements (NOT bytes) of an array-like thing. This is a great addition and came from the efforts of Alejandro Colomar in N3369, and was voted into C2y during the recently-finished Minneapolis, MN, USA 2024 standardization meeting. But, there’s been some questions about whether we chose the right name or not, and rather than spend an endless amount of Committee time bikeshedding and arguing about this, I wanted to put this question to you, the user, with a survey! (Link to the survey at the bottom of the article.)

The Operator

Before we get to the survey (link at the bottom), the point of this article is to explain the available choices so you, the user, can make a more informed decision. The core of this survey is to provide a built-in, language name to the behavior of the following macro named SIZE_KEYWORD:

#define SIZE_KEYWORD(...) (sizeof(__VA_ARGS__) / sizeof(*(__VA_ARGS__)))

int main () {
	int arfarf[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9};
	return SIZE_KEYWORD(arfarf); // same as: `return 10;`
}

This is called nitems() in BSD-style C, ARRAY_SIZE() by others in C with macros, _countof() in MSVC-style C, std::size() (a library feature) and std::extent_v<...> in C++, len() in Python, ztdc_size() in my personal C library, extent in Fortran and other language terminology, and carries many other names both in different languages but also in C itself.

The survey here is not for the naming of a library-based macro (though certain ways of accessing this functionality could be through a macro): there is consensus in the C Standard Committee to make this a normal in-language operator so we can build type safety directly into the language operator rather than come up with increasingly hideous uses of _Generic to achieve the same goal. This keeps compile-times low and also has the language accept responsibility for things that it, honestly, should’ve been responsible for since 1985.

This is the basic level of knowledge you need to access the survey and answer. Further below is an explanation of each important choice in the survey related to the technical features. We encourage you to read this whole blog article before accessing the survey to understand the rationale. The link is at the bottom of this article.

The Choices

The survey has a few preliminary questions about experience level and current/past usage of C; this does not necessarily change how impactful your choice selection will be! It just might reveal certain trends or ideas amongst certain subsets of individuals. It is also not meant to be extremely specific or even all that deeply accurate. Even if you’re not comfortable with C, but you are forced to use it at your Day Job because Nobody Else Will Do This Damn Work, well. You may not like it, but that’s still “Professional / Industrial” C development!

The core part of the survey, however, revolve around 2 choices:

• the usage pattern required to get to said operator/keyword;
• and, the spelling of the operator/keyword itself.

There’s several spellings, and three usage patterns. We’ll elucidate the usage patterns first, and then discuss the spellings. Given this paper and feature were already accepted to C2y, but that C2y has only JUST started and is still in active development, the goal of this survey is to determine if the community has any sort of preference for the spelling of this operator. Ideally, it would have been nice if people saw the papers in the WG14 document log and made their opinions known ahead-of-time, but this time I am doing my best to reach out to every VIA this article and the survey that is linked at the bottom of the article.

Usage Pattern

Using SIZE_KEYWORD like in the first code sample, this section will explain the three usage patterns and their pros/cons. The program is always meant to return 42.

const double barkbark[] = { 0.0, 0.5, 7.0, 14.7, 23.3, 42.0 };

static_assert(SIZE_KEYWORD(barkbark) == 6, "must have a size of 6");

int main () {
	return (int)barkbark[SIZE_KEYWORD(barkbark) - 1];
}

Underscore and capital letter _Keyword; Macro in a New Header

This technique is a common, age-old way of providing a feature in C. It avoids clobbering the global user namespace with a new keyword that could be affected by user-defined or standards-defined macros (from e.g. POSIX or that already exist in your headers). A keyword still exists, but it’s spelled with an underscore and a capital letter to prevent any failures. The user-friendly, lowercase name is only added through a new macro in a new header, so as to prevent breaking old code. Some notable features that USED to be like this:

• _Static_assert/static_assert with <assert.h>
• _Alignof/alignof with <stdalignof.h>
• _Thread_local/thread_local with <threads.h>
• _Bool/bool with <stdbool.h>

As an example, it would look like this:

#include <stdkeyword.h>

const double barkbark[] = { 0.0, 0.5, 7.0, 14.7, 23.3, 42.0 };

_Static_assert(keyword_macro(barkbark) == 6, "must have a size of 6");

int main () {
	return (int)barkbark[_Keyword(barkbark) - 1];
}

Underscore and capital letter _Keyword; No Macro in Header

This is a newer way of providing functionality where no effort is made to provide a nice spelling. It’s not used very often, except in cases where people expect that the spelling won’t be used often or the lowercase name might conflict with an important concept that others deem too important to take for a given spelling. This does not happen often in C, and as such there’s really only one prominent example that exists in the standard outside of extensions:

• _Generic; no macro ever provided in a header

As an example, it would look like this:

// no header
const double barkbark[] = { 0.0, 0.5, 7.0, 14.7, 23.3, 42.0 };

static_assert(_Keyword(barkbark) == 6, "must have a size of 6");

int main () {
	return (int)barkbark[_Keyword(barkbark) - 1];
}

Lowercase keyword; No Macro in Header

This is the more bolder way of providing functionality in the C programming language. Oftentimes, this does not happen in C without a sister language like C++ bulldozing code away from using specific lowercase identifiers. It can also happen if a popular extension dominates the industry and makes it attractive to keep a certain spelling. Technically, everyone acknowledges that the lowercase spelling is what we want in most cases, but we settle for the other two solutions because adding keywords of popular words tends to break somebody’s code. That leads to a lot of grumbling and pissed off developers who view code being “broken” in this way as an annoying busywork task added onto their workloads. For C23, specifically, a bunch of things were changed from the _Keyword + macro approach to using the lowercase name since C++ has already effectively turned them into reserved names:

• true, false, and bool
• thread_local
• static_assert
• alignof
• typeof (already an existing extension in many places)

As an example, it would look like this:

// no header
const double barkbark[] = { 0.0, 0.5, 7.0, 14.7, 23.3, 42.0 };

static_assert(keyword(barkbark) == 6, "must have a size of 6");

int main () {
	return (int)barkbark[keyword(barkbark) - 1];
}

Keyword Spellings

By far the biggest war over this is not with the usage pattern of the feature, but the actual spelling of the keyword. This prompted a survey from engineer Chris Bazley at ARM, who published his results in N3350 Feedback for C2y - Survey results for naming of new nelementsof() operator. The survey here is not going to query the same set of names, but only the names that seemed to have the most discussion and support in the various e-mails, Committee Meeting discussion, and other drive-by social media / Hallway talking people have done.

Most notably, these options are presented as containing both the lowercase keyword name and the uppercase capital letter _Keyword name. Specific combinations of spelling and usage pattern can be given later during an optional question in the survey, along with any remarks you’d like to leave at the end in a text box that can handle a fair bit of text. There are only 6 names, modeled after the most likely spellings similar to the sizeof operator. If you have another name you think is REALLY important, please add it at the end of the comments section. Some typical names not included with the reasoning:

• size/SIZE is too close to sizeof and this is not a library function; it would also bulldoze over pretty much every codebase in existence and jeopardize other languages built on top of / around C.
• nitems/NITEMS is a BSD-style way of spelling this and we do not want to clobber that existing definition.
• ARRAY_SIZE/stdc_size and similar renditions are not provided because this is an operator exposed through a keyword and not a macro, but even then array_size/_Array_size were deemed too awkward to spell.
• dimsof/dimensionsof was, similarly, not all that popular and dimensions as a word did not convey the meaning very appropriately to begin with.
• Other brave but unfortunately unmentioned spellings that did not make the cut.

The options in the survey are as below:

lenof / _Lenof

A very short spelling that utilizes the word “length”, but shortened in the typical C fashion. Very short and easy to type, and it also fits in with most individual’s idea of how this works. It is generally favored amongst C practitioners, and is immediately familiar to Pythonistas. A small point of contention: doing _Lenof(L"barkbark") produces the answer “9”, not “8” (the null terminator is counted, just as in sizeof("barkbark")). This has led some to believe this would result in “confusion” when doing string processing. It’s unclear whether this worry is well-founded in any data and not just a nomenclature issue.

As “len” and lenof are popular in C code, this one would likely need a underscore-capital letter keyword and a macro to manage its introduction, but it is short.

const double barkbark[] = { 0.0, 0.5, 7.0, 14.7, 23.3, 42.0 };

static_assert(_Lenof(barkbark) == 6, "must have an length of 6");

int main () {
	return (int)barkbark[lenof(barkbark) - 1];
}

lengthof / _Lengthof

This spelling won in Chris Bazley’s ARM survey of the 40 highly-qualified C/C++ engineers and is popular in many places. Being spelled out fully seems to be of benefit and heartens many users who are sort of sick of a wide variety of C’s crunchy, forcefully shortened spellings like creat (or len, for that matter, though len is much more understood and accepted). It is the form that was voted into C2y as _Lengthof, though it’s noted that the author of the paper that put _Lengthof into C is strongly against its existence and thinks this choice will encourage off-by-one errors (similarly to lenof discussed above). Still, it seems like both the least hated and most popular among the C Committee and the adherents who had responded to Alejandro Colomar’s GCC patch for this operator. Whether it will continue to be popular with the wider community has yet to be seen.

As “length” and lengthof are popular in C code, this one would likely need a underscore-capital letter keyword and a macro to introduce it carefully into existing C code.

const double barkbark[] = { 0.0, 0.5, 7.0, 14.7, 23.3, 42.0 };

static_assert(_Lengthof(barkbark) == 6, "must have an length of 6");

int main () {
	return (int)barkbark[lengthof(barkbark) - 1];
}

countof / _Countof

This spelling is a favorite of many people who want a word shorter than length but still fully spelled out that matches its counterpart size/sizeof. It has strong existing usage in codebases around the world, including a definition of this macro in Microsoft’s C library. It’s favored by a few on the C Committee, and I also received an e-mail about COUNT being provided by the C library as a macro. It was, unfortunately, not polled in the ARM survey. It also conflicts with C++’s idea of count as an algorithm rather than an operation (C++ just uses size for counting the number of elements). It is dictionary-definition accurate to what this feature is attempting to do, and does not come with off-by-one concerns associated with strings and “length”, typically.

As “count” and countof are popular in C code, this too would need some management in its usage pattern to make it available everywhere without getting breakage in some existing code.

const double barkbark[] = { 0.0, 0.5, 7.0, 14.7, 23.3, 42.0 };

static_assert(_Countof(barkbark) == 6, "must have an length of 6");

int main () {
	return (int)barkbark[countof(barkbark) - 1];
}

nelemsof / _Nelemsof

This spelling is an alternative spelling to nitems() from BSD (to avoid taking nitems from BSD). nelemsof is also seem as the short, cromulent spelling of another suggestion in this list, nelementsof. It is a short spelling but lacks spaces between n and elems, but emphasizes this is the number of elements being counted and not anything else. The n is seen as a universal letter for the count of things, and most people who encounter it understand it readily enough. It lacks problems about off-by-one counts by not being associated with strings in any manner, though n being a common substitution for “length” might bring this up in a few people’s minds.

As “nelems” and nelems are popular in C code, this too would need some management in its usage pattern to make it available everywhere without getting breakage in some existing code.

const double barkbark[] = { 0.0, 0.5, 7.0, 14.7, 23.3, 42.0 };

static_assert(_Nelemsof(barkbark) == 6, "must have an length of 6");

int main () {
	return (int)barkbark[nelemsof(barkbark) - 1];
}

nelementsof / _Nelementsof

This is the long spelling of the nelemsof option just prior. It is the preferred name of the author of N3369, Alejandro Colomar, before WG14 worked to get consensus to change the name to _Lengthof for C2y. It’s a longer name that very clearly states what it is doing, and all of the rationale for nelems applies.

This is one of the only options that has a name so long and unusual that it shows up absolutely nowhere that matters. It can be standardized without fear as nelements with no macro version whatsoever, straight up becoming a keyword in the Core C language without any macro/header song-and-dance.

const double barkbark[] = { 0.0, 0.5, 7.0, 14.7, 23.3, 42.0 };

static_assert(nelementsof(barkbark) == 6, "must have an length of 6");

int main () {
	return (int)barkbark[nelementsof(barkbark) - 1];
}

extentof / _Extentof

During the discussion of the paper in the Minneapolis 2024 meeting, there was a surprising amount of in-person vouching for the name extentof. They also envisioned it coming with a form that allowed to pass in which dimension of a multidimensional array you wanted to get the extent of, similar to C++’s std::extent_v and std::rank_v, as seen here and here. Choosing this name comes with the implicit understanding that additional work would be done to furnish a rankof/_Rankof (or similar spelling) operator for C as well in some fashion to allow for better programmability over multidimensional arrays. This option tends to appeal to Fortran and Mathematically-minded individuals in general conversation, and has a certain appeal among older folks for some reason I have not been able to appropriately pin down in my observations and discussions; whether or not this will hold broadly in the C community is anyone’s guess.

As “extent” is a popular word and extentof similarly, this one would likely need a macro version with an underscore capital-letter keyword, but the usage pattern can be introduced gradually and gracefully.

const double barkbark[] = { 0.0, 0.5, 7.0, 14.7, 23.3, 42.0 };

static_assert(_Extentof(barkbark) == 6, "must have an extent of 6");

int main () {
	return (int)barkbark[extentof(barkbark) - 1];
}

The Survey

Here’s the survey: https://www.allcounted.com/s?did=qld5u66hixbtj&lang=en_US.

There is an optional question at the end of the survey, before the open-ended comments, that allows for you to also rank and choose very specific combinations of spelling and feature usage mechanism. This allows for greater precision beyond just answering the two core questions, if you want to explain it.

Employ your democratic right to have a voice and inform the future of C, today!

Good Luck! 💚

• Banner and Title Photo by Luka, from Pexels

N3366 - Restartable Functions for Efficient Character Conversions has made it into the C2Y Standard (A.K.A., “the next C standard after C23”). And one of my longest struggles — the sole reason I actually came down to the C Standards Committee in the first place —has come to a close.

Yes.

When I originally set out on this journey, it was over 6 years ago in the C++ Unicode Study Group, SG16. I had written a text renderer in C#, and then in C++. As I attempted to make that text renderer cross-platform in the years leading up to finally joining Study Group 16, and kept running into the disgustingly awful APIs for doing text conversions in C and C++. Why was getting e.g. Windows Command Line Arguments into UTF-8 so difficult in standard C and C++? Why was using the C standard functions on a default-rolled Ubuntu LTS at the time handing me data that was stripping off accent marks? It was terrible. It was annoying. It didn’t make sense.

It needed to stop.

Originally, I went to C++. But the more I talked and worked in the C++ Committee, the more I learned that they weren’t exactly as powerful or as separate from C as they kept claiming. This was especially when it came to the C standard library, where important questions about wchar_t, the execution encoding, and the wide execution encoding were constantly punted to the C standard library rather than changed or mandated in C++ to be better. Every time I wanted to pitch the idea of just mandating a UTF-8 execution encoding by default, or a UTF-8 literal encoding by default, I just kept getting the same qualms: “C owns the execution encoding” and “C owns the wide encoding” and “C hasn’t really separated wchar_t from its historical mistakes”. And on and on and on. So, well.

I went down there.

Of course, there were even more problems. Originally, I had proposed interfaces that looked fairly identical to the existing set of functions already inside of <wchar.h> and <uchar.h>. This was, unfortunately, a big problem: the existing design, as enumerated in presentation after presentation and blog post after blog post, was truly abysmal. These 1980s/1990s functions are wholly incapable of handling the encodings that were present even at 1980, and due to certain requirements on types such as wchar_t we ended up creating problematic functions with unbreakable Application Binary Interfaces (ABIs).

During a conversation on very-very-old Twitter now, I was expressing my frustration about these functions and how they’re fundamentally broken. But that if I wanted to see success, there was probably no other way to get the job done. After all, what is the most conservative and new-stuff-hostile language if not C, the language that’s barely responded to everything from world-shattering security concerns to unearthed poor design decisions for some 40 years at that point? And yet, Henri Sivonen pointed out that going that route was still just as bad: why would I standardize something I know is busted beyond all hope?

Contending with that was difficult. Why should I be made to toil due to C’s goofed up 1989 deficiencies? But, at the same time, how could I be responsible for continuing that failure into the future in-perpetuity? Neither of these questions was more daunting than the fact that what was supposed to be a “quick detour” into C would instantly crumble away if I accepted this burden. Doing things the right way meant I was signing up for not just a quick, clean, 1-year-max brisk journey, but a deep dungeon dive that could take an unknown and untold amount of time. I had to take a completely different approach from iconv and WideCharToMultiByte and uconvConvert and mbrtowc; I would need to turn a bunch of things upside down and inside out and come up with something entirely new that could handle everything I was talking about. I had to take the repulsive force of the oldest C APIs, and grasp the attractive forces of all of the existing transcoding APIs,

Henri was right.

It took a lot of me to make this happen. But, I made it happen. Obviously, it will take some time for me to make the patches to implement this for glibc, then musl-libc. I don’t quite remember if the Bionic team handling Android’s standard library takes submissions, and who knows if Apple’s C APIs are something I can contribute usefully to. Microsoft’s C standard library, unlike its C++ one, is also still proprietary and hidden. Microsoft still does a weird thing where, on some occasions, it completely ignores its own Code Page setting and just decides to use UTF-8 only, but only for very specific functions and not all of them.

I GENUINELY hope Microsoft doesn’t make the mistake in these new functions to not provide proper conversions to UTF-8, UTF-16, and UTF-32 through their locale-based execution encoding. These APIs are supposed to give them all the room to do proper translation of locale-based execution encoding data to the UTFs, so that customers can rely on the standard to properly port older and current application data out into Unicode. They can use the dedicated UTF-8-to-UTF-16 and vice versa functions if needed. The specification also makes it so they don’t have to accumulate data in the mbstate_t except for radical stateful encodings, meaning there’s no ABI concerns for their existing stuff so long as they’re careful!

But Microsoft isn’t exactly required to listen to me, personally, and the implementation-defined nature of execution encoding gives them broad latitude to do whatever the hell they want. This includes ignoring their own OEM/Active CodePage settings and just forcing the execution encoding for specific functions to be “UTF-8 only”, while keeping it not-UTF-8 for other functions where it does obey the OEM/Active CodePage.

All in All, Though?

The job is done. The next target is for P1629 to be updated and to start attending SG16 and C++ again (Hi, Tom!). There’s an open question if I should just abandon WG14 now that the work is done, and it is kind of tempting, but for now… I’m just going to try to get some sleep in, happy in the thought that it finally happened.

We did it, chat.

A double-thanks to TomTom and Peter Bindels, as well as the Netherlands National Body, NEN. They allowed me to attend C meetings as a Netherlands expert for 5 years now, ensuring this result could happen. A huge thanks to all the Sponsors and Patrons too. We haven’t written much in either of those places so it might feel barren and empty but I promise you every pence going into those is quite literally keeping me and the people helping going.

And, most importantly, an extremely super duper megathanks h-vetinari, who spent quite literally more than a year directly commenting on every update to the C papers directly in my repository and keeping me motivated and in the game. It cannot be understated how much those messages and that review aided me in moving forward.

God Bless You. 💚

• Banner and Title Photo by Coco Championship, from Pexels
• Imaginary Technique: Purple Image by ZerosPanda (NSFW Artist, Careful Clicking Through!)

The first two meetings of C after C23 was finalized are over, and we have started working on C2y. We decided that this cycle we’re not going to do that “Bugfix” followed by “Release” stuff, because that proved to be a REALLY bad idea that killed a ton of momentum and active contributors during the C11 to C17 timeframe. So, this time, we’re hitting both bugfixes AND features so we can make sure we don’t lose valuable contributions and fixes by stalling for 5 to 6 years again. So, with that… on to fixes!

Generic Selection, a Primer

_Generic — the keyword that’s used for a feature that is Generic Selection — is a deeply hated C feature that everyone likes to dunk on for both being too much and also not being good enough at the same time. It was introduced during C11, and the way it works is simple: you pass in an expression, and it figures out the type of that expression and allows you to match on that type. With each match, you can insert an expression that will be executed thereby giving you the ability to effectively have “type-based behavior”. It looks like this:

int f () {
	return 45;
}

int main () {
	const int a = 1;
	return _Generic(a,
		int: a + 2,
		default: f() + 4
	);
}

As demonstrated by the snippet above, _Generic(...) is considered an expression itself. So it can be used anywhere an expression can be used, which is useful for macros (which was its primary reason for being). The feature was cooked up in C11 and was based off of a GCC built-in (__builtin_choose_expr) and an EDG special feature (__generic) available at the time, after a few papers came in that said type-generic macros were absolutely unimplementable. While C has a colloquial rule that the C standard library can “require magic not possible by normal users”, it was exceedingly frustrating to implement type-generic macros — specifically, <tgmath.h> — without any language support at all. Thus, _Generic was created and a language hole was patched out.

There are, however, 2 distinct problems with _Generic as it exists at the moment.

Problem 0: “l-value conversion”

One of the things the expression put into a _Generic expression undergoes is something called “l-value conversion” for determining the type. “l-value conversion” is a fancy “phrase of power” (POP) in the standard that means a bunch of things, but the two things we’re primarily concerned about are:

• arrays turn into pointers;
• and, qualifiers are stripped off.

This makes some degree of sense. After all, if we took the example above:

int f () {
	return 45;
}

int main () {
	const int a = 1;
	return _Generic(a,
		int: a + 2,
		default: f() + 4
	);
}

and said that this example returns 49 (i.e., that it takes the default: branch here because the int: branch doesn’t match), a lot of people would be mad. This helps _Generic resolve to types without needing to write something very, very convoluted and painful like so:

int f () {
	return 45;
}

int main () {
	const int a = 1;
	return _Generic(a,
		int: a + 2,
		const int: a + 2,
		volatile int: a + 2,
		const volatile int: a + 2,
		default: f() + 4
	);
}

In this way, the POP “l-value conversion” is very useful. But, it becomes harder: if you want to actually check if something is const or if it has a specific type, you have to make a pointer out of it and make the expression a pointer. Consider this TYPE_MATCHES_EXPR bit, Version Draft 0:

#define TYPE_MATCHES_EXPR(DESIRED_TYPE, ...) \
	_Generic((__VA_ARGS__),\
		DESIRED_TYPE: 1,\
		default: 0 \
	)

If you attempt to use it, it will actually just straight up fail due to l-value conversion:

static const int a;
static_assert(TYPE_MATCHES_EXPR(const int, a), "AAAAUGH!"); // fails with "AAAAUGH!"

We can use a trick of hiding the qualifiers we want behind a pointer to prevent “top-level” qualifiers from being stripped off the expression:

#define TYPE_MATCHES_EXPR(DESIRED_TYPE, ...) \
	_Generic(&(__VA_ARGS__),\
		DESIRED_TYPE*: 1,\
		default: 0\
	)

And this will work in the first line below, but FAIL for the second line!

static const int a;
static_assert(TYPE_MATCHES_EXPR(const int, a), "AAAAUGH!"); // works, nice!
static_assert(TYPE_MATCHES_EXPR(int, 54), "AAAAUGH!"); // fails with "AAAAUGH!"

In order to combat this problem, you can use typeof (standardized in C23) to add a little spice by creating a null pointer expression:

#define TYPE_MATCHES_EXPR(DESIRED_TYPE, ...) \
	_Generic((typeof((__VA_ARGS__))*)0,\
		DESIRED_TYPE*: 1,\
		default: 0\
	)

Now it’ll work:

static const int a;
static_assert(TYPE_MATCHES_EXPR(const int, a), "AAAAUGH!"); // works, nice!
static_assert(TYPE_MATCHES_EXPR(int, 54), "AAAAUGH!"); // works, yay!

But, in all reality, this sort of “make a null pointer expression!!” nonsense is esoteric, weird, and kind of ridiculous to learn. We never had typeof when _Generic was standardized so the next problem just happened as a natural consequence of “standardize exactly what you need to solve the problem”.

Problem 1: Expressions Only?!

The whole reason we need to form a pointer to the DESIRED_TYPE we want is to (a) avoid the consequences of l-value conversion and (b) have something that is guaranteed (more or less) to not cause any problems when we evaluate it. Asides from terrible issues with Variably-Modified Types/Variable-Length Arrays and all of the Deeply Problematic issues that come from being able to use side-effectful functions/expressions as part of types in C (even if _Generic guarantees it won’t evaluate the selection expression), this means forming a null pointer to something is the LEAST problematic way we can handle any given incoming expression with typeof.

More generally, however, this was expected to just solve the problem of “make type-generic macros in C to implement <tgmath.h>”. There was no other benefit, even if a whole arena of cool uses grew out of _Generic and its capabilities (including very very basic type inspection / queries at compile-time). The input to type-generic macros was always an expression, and so _Generic only needed to take an expression to get started. There was also no standardized typeof, so there was no way to take the INPUT parameter or __VA_ARG__ parameter of a macro and get a type out of it in standard C anyways. So, it only seemed natural that _Generic took only an expression. Naturally, as brains got thinking about things,

someone figured out that maybe we can do a lot better than that!

Moving the Needle

Implementers had, at the time, been complaining about not having a way to match on types directly without doing the silly pointer tricks above because they wanted to implement tests. And some of them complained that the standard wasn’t giving them the functionality to solve the problem, and that it was annoying to reinvent such tricks from first principles. This, of course, is at the same time that implementers were also saying we shouldn’t just bring papers directly to the standard, accusing paper authors of “inventing new stuff and not standardizing existing practice”. This, of course, did not seem to apply to their own issues and problems, for which they were happy to blame ISO C for not figuring out a beautiful set of features that could readily solve the problems they were facing.

But, one implementer then got a brilliant idea. What if they flexed their implementer muscles? What if they improved _Generic and reported on that experience without waiting for C standard to do it first? What if implementers fulfilled their end of the so-called “bargain” where they actually implemented extensions? And then, as C’s previous charters kept trying to promise (and then fail to deliver on over and over again over decades), what if those implementers then turned around to the C standard to standardize their successful existing practice so that we could all be Charter-Legal about all of this? After all, it would be way, WAY better than being perpetually frozen with fear that if they implemented a (crappy) extension they’d be stuck with it forever, right? It seems like a novel idea in this day and age where everything related to C seems conservative and stiff and boring. But?

Aaron Ballman decided to flex those implementer muscles, bucking the cognitive dissonance of complaining that ISO C wasn’t doing anything, not writing a paper, and not follow up on his own implementation. He kicked off the discussion. He pushed through with the feature. And you wouldn’t believe it, but:

it worked out great.

N3260 - Generic Selection Expression with Type Operand

It’s as simple as the paper title: N3260 puts a type where the expression usually goes. Aaron got it into Clang in a few months, since it was such a simple paper and had relatively small wording changes. Using a type name rather than an expression in there, _Generic received the additional power to get direct matching with no l-value conversion. This meant that qualifier stripping — and more – did not happen. So we can now write TYPE_MATCHES_EXPR like so:

#define TYPE_MATCHES_EXPR(DESIRED_TYPE, ...) \
	_Generic(typeof((__VA_ARGS__)),\
		DESIRED_TYPE: 1,\
		default: 0\
	)

static const int a;
static_assert(TYPE_MATCHES_EXPR(const int, a), "AAAAUGH!"); // works, nice!
static_assert(TYPE_MATCHES_EXPR(int, 54), "AAAAUGH!"); // works, nice!

This code looks normal. Reads normal. Has no pointer shenanigans, no null pointer constant casting; none of that crap is included. You match on a type, you check for exactly that type, and life is good.

Clang shipped this quietly after some discussion and enabled it just about everywhere. GCC soon did the same in its trunk, because it was just a good idea. Using the flag -pedantic will have it be annoying about the fact that it’s a “C2y extension” if you aren’t using the latest standard flag, but this is C. You should be using the latest standard flag, the standard has barely changed in any appreciable way in years; the risk is minimal. And now, the feature is in C2y officially, because Aaron Ballman was willing to kick the traditional implementer Catch-22 in the face and be brave.

Thank you, Aaron!

The other compilers are probably not going to catch up for a bit, but now _Generic is much easier to handle on the two major implementations. It’s more or less a net win! Though, it… DOES provide for a bit of confusion when used in certain scenarios, however. For example, using the same code from the beginning of the article, this:

int f () {
	return 45;
}

int main () {
	const int a = 1;
	return _Generic(typeof(a),
		int: a + 2,
		default: f() + 4
	);
}

does not match on int anymore, IF you use the type-based match. In fact, it will match on default: now and consequently will call f() and add 4 to it to return 49. That’s gonna fuck some people’s brains up, and it will also expose some people to the interesting quirks and flaws about whether certain expressions — casts, member accesses, accesses into qualified arrays, etc. — result in specific types. We’ve already uncovered one fun issue in the C standard about whether this:

struct x { const int i; };

x f();

int main () {
	return _Generic(typeof(f().i),
		int: 1,
		const int: 2,
		default: 0
	);
}

will make the program return 1 or 2 (the correct answer is 2, but GCC and Clang disagree because of course they do). More work will need to be done to make this less silly, and I have some papers I’m writing to make this situation better by tweaking _Generic. _Generic, in general, still needs a few overhauls so it works better with the compatibility rules and also doesn’t introduce very silly undefined behavior with respect to Variable-Length Arrays and Fixed-Size Array generic types. But that’s a topic

for another time. 💚

Constant integer-typed (including enumeration-typed) object declarations in C that are immediately initialized with an integer constant expression should just be constant expressions. That’s it. That’s the whole article; it’s going to be one big propaganda piece for an upcoming change I would like to make to the C standard for C2y/C3a!

Doing The “Obvious”, Obviously

As per usual, everyone loves complaining about the status quo and then not doing anything about it. Complaining is a fine form of feedback, but the problem with a constant stream of crticism/feedback is that nominally it has to be directed — eventually — into some kind of material change for the better. Otherwise, it’s just a good way to waste time and burn yourself out! As one would correctly imagine, this “duh, this is obvious” feature is not in the C standard. But, it seemed like making this change would take too much time, effort, and would be too onerous to wrangle. However, this is no longer the case anymore!

Thanks to changes made in C23 by Eris Celeste and Jens Gustedt (woo, thanks you two!), we can now write a very simple and easy specification for this that makes it terrifyingly simple to accomplish. We also know this will not be an (extra) implementation burden to conforming C23 compilers for the next revision of the standard thanks to constexpr being allowed in C23 for object declarations (but not functions!). As we now have such constexpr machinery for objects, there is no need to go the C++ route of trying to accomplish this in the before-constexpr times. This makes both the wording and the semantics easy to write about and reason about.

How It Works

The simple way to achieve this is to take every non-extern, const-qualified (with no other storage class specifiers except static in some cases) integer-typed (including enum-typed) declaration and upgrade it implicitly to be a constexpr declaration. It only works if you’re initializing it with an integer constant expression (a specific kind of Phrase of Power in C standardese), as well as a few other constraints. There are a few reasons for it to be limited to non-extern declarations, and a few reasons for it to be limited to integer and integer-like types rather than the full gamut of floating/struct/union/etc. types. Let’s take a peak into some of the constraints and reasonings, and why it ended up this way.

Non-extern only!

An extern object declaration could refer to read-only memory that is only read-only from the perspective of the C program. For example, it could refer to a location in memory written to by the OS, or handled by lower level routines that pull their values from a register or other hardware. (Typically, these are also marked volatile, but the point still stands.) We cannot have things that are visible outside of the translation unit and (potentially) affected by other translation units / powers outside of C marked as true constants; it would present a sincere conflict as interest. But, because of extern, we have a clear storage class specifier that allows us to know when things follow this rule or when things do not. This makes it trivially simple to know when something is entirely internal to the translation unit and the C program and does not “escape” the C abstract machine!

This makes it easy to identify which integer typed declarations would meet our goals, here. Though, it does bring up the important question of “why not the other stuff, too?”. After all, if we can do this for integers, why not structures with compound literals? Why not with string literals? Why not with full array initializers and array object declarations inside of a function?! All of these things can be VERY useful to make standards-mandated available to the optimizer.

Integer-Typed Declarations? Why Not “Literally Everything™”?

Doing this for integer types is more of a practicality than a full-on necessity. The reason it is practical is because 99% of all compilers already compute integer constant expressions for the purposes of the preprocessor and the purposes of the most basic internal compiler improvements. Any serious commercial compiler (and most toy compilers) can compute 1 + 1 at compile-time, and not offload that expression off to a run-time calculation.

However, we know that most C compilers do not go as far as GCC or Clang which will do its damnedest to compute not only every integer constant expression, but compound literal and structure initialization expression and string/table access at compile-time. If we extend this paper to types beyond integers, then we quickly exit the general blessing we obtain from “We Are Standardizing Widely-Deployed Existing Practice”. At that point, we would not be standardizing widespread existing practice, but instead the behavior of a select few powerful compilers whose built-in constant folders and optimizers are powerhouses among the industry and the flagships of their name.

C++ does process almost everything it can at compile-time when possible, under the “manifestly constant evaluated” rules and all of its derivatives. This has resulted in serious work on the forward progress of constant expression parsers, including a whole new constant expression interpreter in Clang¹. However, C is not really that much of a brave language; historically, standard and implementation-provided C has been at least a decade (or a few decades) behind what could be considered basic functionality, requiring an independent hackup of what are bogstandard basic features from users and vendors alike. Given my role as “primary agitator for the destruction of C” (or improvement of C; depends on who’s being asked at the time), it seems fitting to take yet another decades-old idea and try to get it through the ol’ Standards Committee Gauntlet.

With that being the case, the changes to C23’s constant expression rules were already seen as potentially harmful for smaller implementations. (Personally, I think we went exactly as far as we needed to in order to make the situation less objectively awful.) So, trying to make ALL initializers be parsed for potential constant expressions would likely be a bridge too far and ultimately tank the paper and halt any progress. Plus, it turns out we tried to do the opposite of what I’m proposing here! And,

it actually got dunked on by C implementers?!

We Failed To Do It The Opposite Way

A while back, I wrote about the paper N2713 and how it downgraded implementation-defined integer constant expressions to be treated like normal numbers “for the purposes of treatment by the language and its various frontends”. This was a conservative fix because, as the very short paper stated, there was implementation divergence and smaller compilers were not keeping up with the larger ones. Floating point-to-integer conversions being treated as constants, more complex expressions, even something like __builtin_popcount(…) function calls with constants being treated as a constant expression by GCC and Clang were embarrassing the smaller commercial offerings and their constant expression parsers.

It turns out that implementation divergence mattered a lot. A competing paper got published during the “fix all the bugs before C23” timeframe, and it pointed all of this out in paper N3138 “Rebuttal to N2713”. The abstract of N3138 makes it pretty clear: “[N2713] diverges from existing practice and breaks code.” While we swear up and down that existing implementations are less important in our Charter (lol), the Committee DOES promise that existing code in C (and sometimes, C-derivative) languages will be protected and prioritized as highly as is possible. This ultimately destroyed N2713, and resulted in it being considered implementation-defined again whether or not non-standards-blessed constant expressions could be considered constants.

Effectively, the world rejected the idea that being downgraded and needing to ignore warnings about potential VLAs (that would get upgraded to constant arrays at optimization time) was appropriate. Therefore, if C programmers rejected going in the direction that these had to be treated for compiler frontend purposes as not-constants, we should instead go in the opposite direction, and start treating these things as constant expressions. So, rather than downgrading the experience (insofar as making certain expressions be not constants and not letting implementations upgrade them in their front-ends, but only their optimizers), let’s try upgrading it!

Formalizing the Upgrade

In order to do this, I have written a paper currently colloquially named NXXX1 until I order a proper paper number. The motivation is similar to what’s in this blog post, and it contains a table that can explain the changes better than I possibly ever could in text. So, let’s take a look:

int file_d0 = 1;
_Thread_local int file_d1 = 1;
extern int file_d2;
static int file_d3 = 1;
_Thread_local static int file_d4 = 1;
const int file_d5 = 1;
constexpr int file_d6 = 1;
static const int file_d7 = 1;

int file_d2 = 1;

int main (int argc, char* argv[]) {
	int block_d0 = 1;
	extern int block_d1;
	static int block_d2 = 1;
	_Thread_local static int block_d3 = 1;
	const int block_d4 = 1;
	const int block_d5 = file_d6;
	const int block_d6 = block_d4;
	static const int block_d7 = 1;
	static const int block_d8 = file_d5;
	static const int block_d9 = file_d6;
	constexpr int block_d10 = 1;
	static constexpr int block_d11 = 1;
	int block_d12 = argc;
	const int block_d13 = argc;
	const int block_d14 = block_d0;
	const volatile int block_d15 = 1;

	return 0;
}

int block_d1 = 1;

For the actual “words in the standard” changes, we’re effectively just making a small change to “§6.7 Declarations, §6.7.1 General” in the latest C standard. It’s an entirely new paragraph that just spins up a bulleted list, saying:

^(NEW)13✨ If one of a declaration’s init declarator matches the second form (a declarator followed by an equal sign = and an initializer) meets the following criteria:

— it is the first visible declaration of the identifier;

— it contains no other storage-class specifiers except static, auto, or register;

— it does not declare the identifier with external linkage;

— its type is an integer type or an enumeration type that is const-qualified but not otherwise qualified, and is non-atomic;

— and, its initializer is an integer constant expression (6.6);

then it behaves as if a constexpr storage-class specifier is implicitly added for that declarator specifically. The declared identifier is then a named constant and is valid in all contexts where a named constant of the corresponding type is valid to form a constant expression of that specific kind (6.6).

Thanks to the improvements to §6.6 from Celeste and Gustedt, and their work on constexpr, the change here is very small, simple, and minimal. This covers all the widely-available existing practice we care about, without providing undue burden for many serious C implementations of C23 and beyond. It also would make a wide variety of integer constant expressions from the “Rebuttal” paper N3138 into valid constant expressions, according to the current rules of the latest C standard. This would be an improvement as it would mean the constant expressions written by users could be relied on across platforms that use a -std=c2y flag or claim to conform to the latest (working draft) C standard.

All in All, Though?

I’m just hoping I can get something as simple as this into C. It’s been long overdue given the number of ways folks complain about how C++ has this but C doesn’t, and it would deeply unify existing practice across implementations. It also helps to remove an annoying style of diagnostic warnings from -Wpedantic/-Wall-style warning lists, too!

The next meeting for C is around October, 2024. I’ll be trying to bring the paper there, to get it formalized, along with the dozens of other papers and features I am working on. Even if my hair will go fully grey by the time this is available on all platforms, I will keep working at it. We deserve the C that people keep talking about, on all implementations.

If not in my lifetime, in yours. 💚

Ever since I finished publishing the “defer” paper and successfully defended it on its first go-around (it now has tentative approval to go to a Technical Specification, I just need to obtain the necessary written boilerplate to do so), an old criticism repeats itself frequently. Both internally to the C and C++ Standards Committee, as well as to many outside, the statement is exactly as the title implies: to implement a general-purpose undo mechanism for C, why not just make Objects with Well-Scoped, Deterministic Lifetimes and build it out of that like C++? This idiom, known as Resource Acquisition Is Initialization (RAII), is C++’s biggest bread and butter and its main claim to fame over just about every other language that grew up near it and after it (including all of the garbage collected languages such as Managed C++, D, Go, etc.). I have received no less than 5 external-to-WG14 (the formal abbreviation for the C Standards Committee) requests/asks about this, and innumerable posts internal to the C Standard mailing lists.

So, let’s just get this off the table right now so I can keep referring to this post every time somebody asks:

You ✨Cannot✨ Have “Simple RAII” in C

That’s the entire premise of this article. There’s a few reasons this is not possible – some mentioned in the defer paper version N3199, and others that I just sort of took for granted that people would understand but do not – and so, to clear up confusion, they will be written down here. There are two MAJOR reasons one cannot take the object-oriented semantics and syntax of RAII from C++ as-is, without jeopardizing sincere qualities about C:

• RAII is syntactically difficult to achieve in C due to the semantics imbued on those syntax constructs by C++;
• and, RAII is semantically impossible in C due to C’s utterly underwhelming type/object model.

To start with, let’s go over the syntax of C++, and how it achieves RAII. We will also discuss a version of RAII that uses not-C++ syntax, which would work…. at least until the second bulleted reason above dropkicks that in the face. So, let’s begin:

RAII: C++ Syntax

As a quick primer for those who are not familiar, C++ achieves its general purpose do-and-undo mechanism through the use of constructors and destructors. Constructors are function calls that are always invoked on the creation of an object, and destructors are always invoked when an object leaves scope. One can handle doing the construction and destruction manually, but we don’t have to talk about such complicated cases yet. The syntax looks as follows:

#include <cstdlib>

struct ObjectType {
	int a;
	double b;
	void* c;

	/* CONSTRUCTOR: */
	ObjectType() : a(1), b(2.2), c(malloc(30)) {

	}

	/* DESTRUCTOR: */
	~ObjectType() {
		free(c);
	}
};

In the above code snippet, we have a structure named ObjectType. It has a single constructor, that takes no arguments, and initializes all 3 of its members to some default values. It also has a destructor, which is meant to “undo” anything in the class that the programmer likes. In this case, I an using it to purposefully free the data that I originally mallocd into the member c during construction. Thus, when I use the class in this manner:

#include <cstdio>

int main () {
	ObjectType thing = {};
	printf("%d %f %p", thing.a, thing.b, thing.c);
	return 0;
}

despite not seeing any other code in that snippet, that code will:

1. create automatic storage duration memory to put thing in (A.K.A. stack space for a stack variable);
2. call the constructor on that automatic storage duration memory location (A.K.A. the thing that sets those values and does malloc)
3. perform the printf call
4. prepares the return statement with the value of 0
5. call the destructor on that automatic storage duration memory location (A.K.A. the thing that calls free to release the memory)
6. release the automatic storage duration memory (A.K.A. cleans up the stack)
7. return from the function with the value 0 being transported in whatever manner the implementation has defined

This is a fairly simple set of steps, but it’s a powerful concept in C++ because no matter what happens (modulo some of the more completely bananas situations), once an object is “properly constructed” (all the data members are initialized from the TypeName (...) : … { list and reach the opening brace) in some region of memory, the compiler will always deterministically call the destructor at a fixed location. There is no wibbly-wobbly semantics like .NET IL finalizers or Lua __gc methods: the object is created, the objected is destroyed, always. (Again, we are ignoring more manual cases at the moment such as using new/delete, its array friends, or placement new & other sorts of shenanigans.) As Scott Meyers described it, this is a “powerful, general-purpose undo mechanism” and its one of the most influential concepts in deterministic, non-garbage-collected systems programming. Every other language worth being so much as spit on either employs deep garbage collection (Go, D, Java, Lua, C#, etc.) or automatic reference counting (Objective-C, Objective-C++, Swift, etc.), uses RAII (Rust with Drop, C++, etc.), or does absolutely nothing while saying to Go Fuck Yourself™ and kicking the developer in the shins for good measure (C, etc.).

The first problem with this, however, is a technical hangup. When C++ created their constructors, they created them with a concept called function overloading in mind. This very quickly gets into the weeds of Application Binary Interfaces and other thorny issues, which are thankfully already thoroughly written about in this expansive blog post, but for the sake of brevity revisiting these concepts is helpful to understand the issue.

Problem 0: Function Overloading

Function overloading is a technique where software engineers, in source code and syntactically, name what are at their core two different functions the same name. That single name is used as a way to referring to two different, distinct function calls by employing extra information, such as the number of arguments, the types of the arguments, and other clues when that single name gets used:

// FUNCTION 0
int func (int a);
// FUNCTION 1
double func (double b);

int main () {
    int x = func(2); // calls FUNCTION 0, f(int)
    double y = func(3.3); // calls FUNCTION 1, f(double)
    return (int)(x + y);   
}

However, when the source code has to stop being source code and instead needs to be serialized as an actual, runnable, on-the-0s-and-1s-machine binary, linkers and loaders do not have things like compile-time “type” information and what not at-the-ready. It is too expensive to carry that information around, all the time, in perpetuity so that when someone runs a program it can differentiate between “call f that does stuff with an integer” versus “call f that does stuff with a 64-bit IEEE 754 floating point number”. So, it undergoes a transformation that transforms f(int) or f(double) into something that looks like this at the assembly level:

main:
        push    rbx
        mov     edi, 2
        call    _Z4funci # call FUNCTION 0
        movsd   xmm0, QWORD PTR .LC0[rip]
        mov     ebx, eax
        call    _Z4funcd # call FUNCTION 1
        movapd  xmm1, xmm0
        pxor    xmm0, xmm0
        cvtsi2sd        xmm0, ebx
        pop     rbx
        addsd   xmm0, xmm1
        cvttsd2si       eax, xmm0
        ret
.LC0:
        .long   1717986918
        .long   1074423398

The code looks messy because we’re working with doubles and so it generates all sorts of stuff for passing arguments and also casting it down to a 32-bit int for the return expression, but the 2 important lines are call _Z4funci and call _Z4funcd. Believe it or not, these weird identifiers in the assembly correspond to the func(int) and func(double) identifiers in the code. This technique is called “name mangling”, and it powers a huge amount of C++’s featureset. Name mangling is how, so long a argument types or number of arguments change, things like the Application Binary Interface (ABI) of function calls can be preserved. The compiler is taking the name of the function func and the arguments int/double and mangling it into the final identifier present in the binary, so that it can call the right function without having a full type system present at the machine instruction level. This has the obvious benefit that the same conceptual name can be used multiple different ways in code with different data types, mapping strongly to the “this is the algorithm, and it can work with multiple data types” idea. Thus, the compiler worries about the actual dispatch details and resolves at compile-time, which means there no runtime cost to do matching on argument count or argument types. Having it resolved at compile-time and mapped out through mangling allows it to just directly call the right code during execution. The reason this becomes important is because this is how constructors must be implemented.

Problem 1: Member Functions

Consider the same ObjectType from before, but with multiple constructors:

#include <cstdlib>

struct ObjectType {
	int a;
	double b;
	void* c;

	/* CONSTRUCTOR 0: */
	ObjectType() : a(1), b(2.2), c(malloc(30)) {

	}

	/* CONSTRUCTOR 1: */
	ObjectType(double value) : a((int)(value / 2.0)), b(value), c(malloc(30)) {

	}

	/* DESTRUCTOR: */
	~ObjectType() {
		free(c);
	}
};

#include <cstdio>

int main () {
	ObjectType x = {};
	ObjectType y = {50.0};
	printf("x: %d %f %p\n", x.a, x.b, x.c);
	printf("y: %d %f %p\n", y.a, y.b, y.c);
	return 0;
}

We can see the following assembly:

.LC1:
	.string "x: %d %f %p\n"
.LC2:
	.string "y: %d %f %p\n"
main:
	push    r12
	push    rbp
	push    rbx
	sub     rsp, 64
	mov     rdi, rsp
	lea     rbp, [rsp+32]
	mov     rbx, rsp
	call    _ZN10ObjectTypeC1Ev
	movsd   xmm0, QWORD PTR .LC0[rip]
	mov     rdi, rbp
	call    _ZN10ObjectTypeC1Ed
	mov     rdx, QWORD PTR [rsp+16]
	movsd   xmm0, QWORD PTR [rsp+8]
	mov     edi, OFFSET FLAT:.LC1
	mov     eax, 1
	mov     esi, DWORD PTR [rsp]
	call    printf
	mov     rdx, QWORD PTR [rsp+48]
	movsd   xmm0, QWORD PTR [rsp+40]
	mov     edi, OFFSET FLAT:.LC2
	mov     eax, 1
	mov     esi, DWORD PTR [rsp+32]
	call    printf
	mov     rdi, rbp
	call    _ZN10ObjectTypeD1Ev
	mov     rdi, rsp
	call    _ZN10ObjectTypeD1Ev
	add     rsp, 64
	xor     eax, eax
	pop     rbx
	pop     rbp
	pop     r12
	ret
	mov     r12, rax
	jmp     .L3
	mov     r12, rax
	jmp     .L2
main.cold:
.L2:
	mov     rdi, rbp
	call    _ZN10ObjectTypeD1Ev
.L3:
	mov     rdi, rbx
	call    _ZN10ObjectTypeD1Ev
	mov     rdi, r12
	call    _Unwind_Resume
.LC0:
	.long   0
	.long   1078525952 

Again, we notice in particular the use of these special, mangled identifiers for the call instructions: call _ZN10ObjectTypeC1Ev, call _ZN10ObjectTypeC1Ed, and call _ZN10ObjectTypeD1Ev. It has the name of the type (…10ObjectType…) in it this time, but more or less just mangles it out. This is where the heart of our problems lie. If C wants to steal C++’s syntax for RAII, and C wants to be able to share (header file) source code that enjoys simple RAII objects, every single C implementation needs to implement a Name Mangler compatible with C++ for the platforms they target. And how hard could that possibly be?

Hm.

Here are some name manglings for the one argument ObjectType constructor:

• _ZN10ObjectTypeC1Ed (GCC/Clang on Linux; x86-64, ARM, ARM64, and i686)
• ??0ObjectType@@QEAA@N@Z (MSVC; x86-64, ARM64)
• ??0ObjectType@@QAE@N@Z (MSVC; i686)

That’s three different name manglings for only a handful of platforms! And while some name manglers are partially documented or at least provided as a library so that it can be built upon, the name manglers for others are not only utterly undocumented but completely inscrutable. So much so that on some platforms (like MSVC on any architecture), certain name manglings are not guaranteed to be 1:1 and can infact “demangle” into multiple different plausible entities. If an implementation gets the name mangling wrong, well, that’s just a damn shame for the end user who has to deal with it! Of course, nobody’s claiming that name mangling is an unsolvable problem; it is readily solved in codebases such as Clang and GCC. But, it is worth noting that, as C’s specification stands now, there is no requirement to mangle any functions.

This is both a blessing, and a curse. The former because functions that users write are pretty much 1:1 when they are written under a C compiler. If a functioned is named glorbinflorbin in C, the name that shows up in the binary is glorbinflorbin with maybe some extra underscores added in places somewhere on certain implementations. But, the latter comes in to play for precisely this reason: if there is no name mangling performed that considers things such as related enclosing member object, argument types, and similar, then it is impossible to have even mildly useful features that can do things like avoid name clashes a function prototype is generated with the wrong types. It is, in fact, the primary reason that C ends up in interesting problems when using typedefs inside of its function declarations. Even if the typedefs change, the function names do not because there is no concept of “member functions” or “function overloading” or anything like that. It’s why the intmax_t problem is such an annoying one.

What Does This Have To Do With RAII?

Well, the devil is in these sorts of details. In order to introduce nominal support for something like constructors, name mangling (or something that allows the user to control how names come out on the other side) need to be made manifest in C. If name mangling is chosen as the implementation choice and a syntax identical to C++ is chosen, the implementation becomes responsible for furnishing a name mangler. And, because people are (usually) not trying to be evil, there should be ABI compatibility between the C implementation’s name mangler and C++’s name mangler so that code written with constructors in one language interoperate just fine with the other, without requiiring extern "C" to be placed on every constructor. (Right now, extern "C" is not legal to place on any member function in any C++ implementation.)

The reason this is desirable is obvious: header code could be shared between the languages, which makes sense in a world where “just steal C++’s constructors and destructors” is the overall design decision for C. But this is very much a nonstarter implementation reasons. Most implementers get annoyed when we require them to implement things that might take significant effort. While Clang and GCC likely won’t give an over damn so long as its not C++-modules levels of difficult (and MSVC ignores us until it ships in a real standard), there’s hundreds of C compilers and implementers of WILDLY varying quality. Unlike the 4-5 C++ compilers that exist today, C compilers and their implementers are still cranking things out, sometimes as significant pillars of their (local) software economy. Now, while I personally loathe to use things like lines of code as a functional metric for code, it can help us estimate complexity in a very crude, contextless way. Checking in on Clang’s Itanium Mangler, it clocks in somewhere on the order of about 7,000 lines of code. Which really doesn’t sound so bad,

until chibicc’s entire codebase measures somewhere around 7,300 lines of code.

“Double the amount of code in my entire codebase excluding tests for this feature” very much does not pass the smell test of implementability for C. This is also not including, you know, all the rest of the code required for actually implementing the “parse constructors and destructors” bit. Though, thankfully, that part is a lot less work than the name mangler. and I can guarantee that since there’s quite literally hundreds of C implementations, many of them will… “have fun”. If two or three different ways to mangle ObjectType::ObjectType(double) is bad, wait until a couple dozen implementers who have concerns outside of “C++ Compatibility” – some even with an active bone to pick with C++ – are handed a gaggle of features that relies on a core mechanic that is entirely unspecified. I am certainly not the smartest person out there,

but I know a goddamn interoperability bloodbath when I see one.

But… What If Name Mangling Was not a Problem?

This is the other argument I have received a handful of times on both the C mailing list, and in my inbox. It’s not a bad argument; after all, the entire above argument hinges on the idea of stealing the syntax from C++ entirely and copying their semantics bit-for-bit. By simply refusing to do it the way C++ does it, does it make the above argument go away? Thusly appears the following suggestion, which boils down to something like the following snippet. However, before we continue, note that this syntax comes partially from an e-mail sent to me. PLEASE, second-to-last person who sent me an e-mail about this and notices the syntax looks similar to what was in the e-mail: I am not trying to make fun of you or the syntax you have shown me, I am just trying to explain as best as I can. With that said:

#include <stdlib.h>

struct nya {
	void* data_that_must_be_freed;
};

_Constructor void nya_init(struct nya *nya, int n) {
	nya->data_that_must_be_freed = malloc(n);
}

_Destructor void nya_clear(struct nya *nya) {
	free(nya->data_that_must_be_freed);
}

int main () {
	struct nya n = {30};
	return 0;
}

The following uses the _Constructor and _Destructor tags on function declarations/definitions to associate either the returned type struct nya and the destructed type struct nya * (a pointer to an already-existing struct nya to destroy). The sequence of events, here, is pretty simple too:

1. n’s memory is allocated (off of the stack), its memory is taken from the appropriate location on the stack and passed to;
2. nya_init, which then calls malloc to initialize its data member;
3. the return 0 is processed, storing the 0 value to do the actual return later, while;
4. nya_clear is called on the memory for n, and the data member is appropriately freed;
5. finally, main returns 0.

It has the same deterministic destruction properties as RAII here. But, notably, it is attached to a free-floating function.

This does the smart thing and gets around the name mangling issue! The person e-mailing me here has sidestepped the whole issue about sharing syntax with C++ and its function overloading issue, which is brilliant! If you can associate a regular, normal function call with these actions, it is literally no longer necessary to provide a name mangling scheme. It does not need to exist, so nobody will implement one: it’s just calling a normal function. (Kudos to Rust for figuring part of this out themselves as well, though they still need name mangling thanks to Traits and Generics.) It avoids all of the very weird fixes other people tried to propose on the C standards internal mailing list by saying things like “only allow one constructor” or “make C++ have extern "C" on constructors work and then have C and C++ mangle them differently” or “just implement name manglers for all C compilers that implement C2y/C3a, it’s fine”. Implementability can certainly be achieved with this.

Other forms of this come from a derivation of the two existing Operators proposals (Marcus Johnson’s n3201 and Jacob Navia’s n3051), most particularly n3201. The recommendation for n3201 by the author was to just use a different “association” that did not actually affect the syntax of the function itself, so the above code that produces the same affect but under n3201’s guidance (but slightly modified from the way it was presented in n3201 because that syntax has Problems™) might look like:

#include <stdlib.h>

struct nya {
	void* data_that_must_be_freed;
};

void nya_init(struct nya *nya, int n) {
	nya->data_that_must_be_freed = malloc(n);
}

void nya_clear(struct nya *nya) {
	free(nya->data_that_must_be_freed);
}

_Operator = nya_init;
_Operator ~ nya_clear;

int main () {
	struct nya n = {30};
	return 0;
}

Completely ignoring syntax choices here and the consequences therein, these _Operator statements would associate a function call with an action. = in this case seems to apply to construction, and ~ seems to apply to destruction. As usual, because the association is made using a statement and type information at compile-time, the compiler can know to simply call nya_init and nya_clear without needing to set up a complex, implementation-internal name mangling scheme to figure out which constructor/member/whatever function it needs to call to initialize the object correctly. It also doesn’t rob C++ of its syntax but try to impose different semantics. Nor does it just tell C implementations the functional equivalent of “git gud” with respect to implementing the name mangler(s) required to play nice with other systems. There is, unfortunately, one really annoying problem with having this form of constructors and destructors, and it’s the same problem that C++ had when it first started out trying to tackle the same problem back in the 80s and 90s:

none of these proposals come with an Object Model, and C does not have a real Object Model aside from its Effective Types model!

RAII: C++ Semantics

While the syntax problem can be designed around with any number of interesting permutations or fixes, whether it’s _Operator or _Constructor or whatever, the actual brass-and-tack semantics that C++ endows on memory obtained from these objects is very strict and direct. When someone allocates some memory and casts it to a type and begins to access it, both [c.malloc] and [intro.object]/11-13 cover them by giving them implicitly created objects, so long as those types satisfy the requirements of being trivial and implicitly-creatable types. On top of that, for constructors and destructors, there is an ENORMOUSLY robust system that comes with it beyond these implicitly created objects. This post was going to be extremely long, but thanks to an excellent writeup by Basit Ayuntande, everything anyone needs to know about the C++ object model is already all written up. To fully understand all the details, shortcuts, tricks, and more, please read Basit’s article; becoming a better C++ developer (if that’s desirable) is an inevitably from digesting it.

This, of course, leaves us to talk about just C and RAII and how those semantics play out.

C: Effective Types

In C, we do not have a robust object model. The closest are effective type rules, and they work VIA lvalue accesses rather than applying immediately on cast. The full wording is found in §6.5.1 “General” of N3220, which states:

The effective type of an object for an access to its stored value is the declared type of the object, if any. If a value is stored into an object having no declared type through an lvalue having a type that is not a non-atomic character type, then the type of the lvalue becomes the effective type of the object for that access and for subsequent accesses that do not modify the stored value. If a value is copied into an object having no declared type using memcpy or memmove, or is copied as an array of character type, then the effective type of the modified object for that access and for subsequent accesses that do not modify the value is the effective type of the object from which the value is copied, if it has one. For all other accesses to an object having no declared type, the effective type of the object is simply the type of the lvalue used for the access.

This is a bunch of text to say something really simple: if a region of memory (like a pointer obtained from malloc) is present, and it is cast to a specific type for the purposes of reading or writing, that region is marked with a given type and the type plus region informs what is the effective type of the memory. The first write or access is what solidifies it as such. The effective type follows a memory region through memmove or memcpy done with appropriate objects of the appropriate size. Fairly straightforward, standard stuff. The next paragraph after this then creates a list of scenarios wherein about any accesses or writes performed through casts or pointers aliasing that region afterwards:

• a type compatible with the effective type of the object,
• a qualified version of a type compatible with the effective type of the object,
• the signed or unsigned type compatible with the underlying type of the effective type of the object,
• the signed or unsigned type compatible with a qualified version of the underlying type of the effective type of the object,
• an aggregate or union type that includes one of the aforementioned types among its members (including, recursively, a member of a subaggregate or contained union), or
• a character type.

This is, effectively, C’s aliasing rules. Once a type is set into that region of memory, once casting happens from one type to another (e.g. casting it first to uint32_t* to write to it, and then try to read it as a float* next), that action must be on that list to be standard-sanctioned. If it isn’t, then undefined behavior is invoked and programs are free to behave in very strange ways, at the whim of implementations or hardware or whatever. While I am not holding the person who sent me the simple one-off e-mail accountable to this, in the wider C ecosystem and in discussion even on the C mailing list, there seemed to be a distinct lack of appreciation for how thought-through the C++ system is and why it is this way in the first place. This also becomes glaringly clear after reading n3201 and going through 95% of the discussions around “RAII in C” that just tries to boil it down to simple syntactical solutions with basic code motion. The bigger picture is NOT being considered. There is not even a tiny amount of respect for where C or C++ comes from. It is not just about effective types and shadowy rules about how do they handle dynamic memory: even simpler things just completely fall apart in these counterproposals. Take, for example, a very simple question.

“How do you handle copies?”

Taking the _Operator example from above again, let’s add a single line of spice to this:

#include <stdlib.h>

struct nya {
	void* data_that_must_be_freed;
};

void nya_init(struct nya *nya, int n) {
	nya->data_that_must_be_freed = malloc(n);
}

void nya_clear(struct nya *nya) {
	free(nya->data_that_must_be_freed);
}

_Operator = nya_init;
_Operator ~ nya_clear;

int main () {
	struct nya n = {30};
	struct nya n2 = n; // OH SHIT--
	return 0;
}

In a proposal like n3201, what happens here? The actual answer is “the proposal literally does not answer this question”. Assuming (briefly, if I can be allowed such for a moment) the “basic” or “default” for how it works right now, the answer is probably “just memcpy like before”, which is wrong. n3201 is not the first “just do a quick RAII in C” proposal sent to me over e-mail to make this mistake. Simply performing a memberwise copy of struct nya from n to n2 leads to an obvious double-free when n2 goes out of scope, frees the memory pointed to by data_that_must_be_freed, and then n will attempt attempt to free that data as well. This is an infinitely classic blunder, and in critical enough code becomes a security blunder. The suggestions that stem from pointing this out range from unserious to just disappointing, including things like “just ban copying the structure”. Nobody needs a degree in Programming Language Design to communicate that “just ban simple automatic storage duration structure copying” is a terrible usability and horrific ergonomics decision to make, but that’s where we are. And it’s so confusingly baffling that it is impossible to be mad that the suggestion is brought up.

Or, take in n3201’s case (which updates the previous paper, n3182). When responding to the (ever-present) criticism that operators – including for initialization/assignment – that someone could do something weird inside of the operator, n3201 adds a constraint which reads:

Functions must contain the matching operator in their function bodies. i.e. _Operator declarations that associate the compares-equal operator with a function, must contain the compares-equal operator in the body of the function named in the _Operator declaration. (iostream-esque shenanigans with overloading the bitwise shift operators to read/write characters and strings isn’t allowed).

The fact that the proposal has something for initialization (but not cleanup), does not mention anything about the fact that the code snippet in the proposal itself apparently (?) leaks memory, that this constraint is very much deeply unsettling to impose on any type (there’s plenty of vec4 or other mathematics code where I’m using intrinsics that look nothing like the operators they’re being implemented for) does not seem to bother the author in the slightest. Instead, there’s just a palpable hatred of C++ there, apparently so strong that it overrides any practical engagement with the problem space. The proposal – and much of the counter-backlash I had to sift through on the mailing lists and elsewhere as people proposed stripped down RAII solutions for C under the guise of being “simple” – is too busy taking potshots at C++ to address clear and present dangers to its own functionality.

C as an Anti-C++

And this is where things just keep getting worse, because so much of C’s culture seems to swirl around the idea of either being “budget, simple, understandable C++” or “Anti/Nega-C++”. Instead of engaging on C’s stated merits or goals, like:

• what-you-write-is-what’s-inside (a function foo produces a binary symbol named foo);
• uncompromised, direct access to the hardware (through close collaboration with implementation-defined asm, intrinsics, and unparalleled control of the compiler (severe work in progress, honestly));
• simple enough that it can always be used to glue two or more languages together (for any single given platform/compiler combination);
• and, being a smaller language focused on its use cases (K&R literally sold C on being good at strings – we can see how that’s been going in the last 30 years).

We instead get “why doesn’t this PRIMITIVE, UNHOLY C just become C++” proposals, and similar just-as-ill-considered “here is my simpler (AND BETTER THAN THAT CRAPPY C++ STUFF) feature” proposals. Sometimes, like the person who e-mailed me with the struct nya example, there’s a genuine curiosity for exploring a different design space that serves as an actually better basis. But at even our highest echelons, the constant spectre of C++ that continually drives an underlying and utterly unhelpful antagonism that prevents actual technical evaluation. It results in things like _Operator throwing itself in the way of RAII, to try and half-heartedly solve the RAII problem without actually engaging with the sincere, instructive merit of the C++ object model. It also prevents actually evaluating the things that make RAII weak, including problems with the strong association with objects that actually manifest in its own standard library.

The negative tradeoffs for defer are numerous, especially since it absolutely loses many of the abilities that come from being a regular object with a well-defined lifetime. This means it is not as powerful as constructors and destructors, including that it is prone to Repeat-Yourself-Syndrome since the defer entity itself is not reusable. It cannot be attached to individual members of a structure, nor can it be passed through function calls or stored on the heap. It cannot be transferred with move constructors or duplicated with copy constructors in a natural way, or in any way as a matter of fact! It can only exist at function scope, not at file scope, and only exists procedurally.

The beneficial tradeoffs are it avoids the Static Initialization Order Fiasco that comes with having objects with constructors at file scope or marked static at function scope. It also does not combine lambdas with object-based destructors to torch 15+ years of life asking the C++ Standards Committee to standardize std::scope_guard only to ultimately be denied success at retirement age (sorry, Peter Sommerlad) because of the C++ Standard Library’s ironclad exceptions-and-destructors rule. And, to be clear, it was the right decision for them to do that! Poking a hole in the “all destructors from the standard library are noexcept” mandate adds needless library complexity gymnastics for a feature that the language should be taking care of! The proper realization after that would be that a language feature is required to sidestep the concerns that come with the Object Model. Of course, I do not expect the C++ Standard Committee’s Evolution Working Group to take that situation seriously as a body; likely, they will leave Library Evolution Working Group out to dry on the matter.

Coming to these sorts of conclusions only arises through behaving as an engineer that is looking to improve at their craft and strengthen their tools, rather than getting into a hammer-measuring pissing contest with the engineers down the hall.

But. Alas!

It still leaves a sour taste, though. It sort of lingers at the back of anyone’s mouth when they sit down to think about it, because it is kind of distasteful.

Genuinely, I understand that C can be behind. Very behind, in fact: taking 30 years to standardize typeof, not performing macro-gymnastics to get to typeof_unqual in the same 30 years, and not making any meaningful moves to work on things like e.g. “Statement Expressions” (something even the Tiny C Compiler implements) easily illustrates just how gut wrenchingly difficult it is to move the needle just a centimeter in this increasingly Godless industry. But when people propose a feature that has had 40+ years of work and refinement and care put into it, but at no point do they sit down and think about “what happens if I copy this object using the usual syntax” or “do we need some syntax for moving objects from one place to another” or “maybe I should not provoke a double free in the world’s most harmless looking code”, the thoughts start coming in. Is this being taken seriously? Is it just forgetfulness? Is it just so automatic nobody thinks about it? Is the pedagogy what is behind here, and is there a teaching crisis for this language?

So Many Questions

And yet, I will see not one damn answer, that’s for sure. Genuinely, I yearn for it because getting things half-baked things like they are in n3201 or similar is kind of rough to deal with. On one hand there’s the overwhelming urge to just grab the proposal and rip it up and get a white board and just go “here, HERE. WHERE IS YOUR OBJECT MODEL. WHAT HAPPENS TO THE EFFECTIVE TYPE RULES. DID YOU THINK ABOUT COPYING AND MOVING THINGS. WHAT HAPPENS IF SOMEBODY USES THESE IN AN COMPOUND ASSIGNMENT EXPRESSION. WHAT HAPPENS IF THEY ARE ASSIGNED FROM A TEMPORARY. HOW DO YOU PASS THAT IN TO THE USER. WHAT ARE THE THINGS THEY CAN CONTROL. HOW DO WE HANDLE THIS FROM HEAP MEMORY OR A STACK ARRAY UNSIGNED CHARACTERS.”

But that kind of tone, that sort of engagement is antagonistic, probably in the extreme.

It’s also not how I would like to engage with anyone. Like, the person who sent me an e-mail with the cute struct nya and the very simple and nice _Constructor syntax might not even have gotten that deep in the C standard and likely barely knows the effective type rules; I sure as hell barely understand them and I’m in charge of goddamn editing them when a few of the big upcoming papers finally make their way through the C Committee.

If I respond to an e-mail like that – with all the capital letters and everything – it would be completely out of line and also would be very unfair, because it is not their fault. I haven’t done that to anyone so far, but the fact that the thought exists in my head is Not Fun™. It’s not anyone’s fault, it’s just an internal struggle with thinking the whole industry is a lot farther along on these problems and continuously feeling like I am very much too stupid to be here. Like, I’m a goddamn moron, a genuine idiot, I cannot be ahead of the game, am I being pranked? Am I being tested, to see if I really belong here? Is someone going to swing in out of the blue and go “AHA, YOU MISSED THE OBVIOUS!”? Something is absolutely not adding up.

The utterly pervasive and constant feeling that a lot of people – way too many people – are really trying to invent these things from first principles and pretend like they were the first people to ever conceive of these ideas… it feels pretty miserable, all things considered. Going through life evaluating effectively no prior art in other languages, domains, C codebases as they exist today, just… anything. It’s a constant nagging pull when working on things like standard C that for the life of me I cannot seem to shake no matter how carefully I break it down. Hell, part of writing this post is so I can stick a link to it in my defer paper and in the defer Technical Specification when it happens so I don’t have to sit down and walk through why I chose a procedural-style, object-less idiom for C rather than trying to load the RAII shotgun and point it at our beloved 746-and-counting page C standard.

Changing a programming language’s whole object model is hard. Adding “things that must be run in order to bring an object into existence, and thing that must be run in order to kill an object, modulo Effective Type rules, with No Other Exceptions” is a big deal. Where in the proposals do they discuss new/delete, and why they are used as wrappers around malloc to ensure construction and destruction are coupled with memory creation to prevent very common bugs? Where is the consideration for placement new or being able to call destructors manually on an object or a region of memory? RAII enables simple idioms but it is not a simple endeavor! Weakening portions of RAII makes it so much less useful and so much less powerful, which is really weird! Is not the thing people keep telling me about C is that its the language of ultimate power and ultimate control? Why does that repeatedly not show up in these discussions?!

It feels so bizarre to have to actually sit down and explain some of these things sometimes because a lot of these things have become second nature to me, but it is just a part of the curse.

“It was Just Some E-mails, Man, Calm Down!”

To be very clear, the person who sent the e-mail – whose syntax I stole using struct nya * for this post for the _Constructor/_Destructor idea – is not someone I actually expect to send me a 5 page e-mail thesis on enhancements to the C object model. That person CLEARLY was just trying to give me a quick simple idea they thought up of that made it easy on them / solved the problem at hand, and I certainly don’t fault them for thinking of it! Their initiative actually demonstrates that rather than just doing the copy-paste roboticism of people who would blindly steal syntax from C++ and then strip off the bits they don’t like and go “See? Simple!” they’re actually thinking about and engaging with the technical merits of the problem. I certainly wish n3201 and other solutions had a fraction of that spark and curiosity and eagerness to explore the space and actually push the needle for C forward, rather than just being driven by trying to define C as “anti-C++”.

My intention is to keep moving forward with proposals like defer, among many others over the next few years, to start actually improving C for C’s sake. Sometimes this will mean cloning an idea right out of C++ and putting it in C; other times, weighing the pros and cons and addressing the inherent deficiencies in approaches to produce something better will be much more desirable. Knee jerk reactions like those in n3201 rarely serve to help either language and are producing demonstrably worse outcomes; which also concern me because I had an idea for handling operators in C for a long time now and seeing the current proposals do a poor job of handling the landscape is not going to bolster anyone’s confidence in how to do it…!

But, the person who inquired VIA e-mail deserves an enthusiastic “NICE”, a thumbs up, and maybe a cookie and a warm glass of milk for actually thinking about the problem domain. … In fact.

Cookies and milk sounds real good right now… 💚

I received a few complaints that #embed was difficult to implement and hard to optimize. And, the people making these claims are not exactly wrong. While std::embed was designed to be very simple and easy, the new #embed directive does the usual C thing: it’s “simple” on its face, but because of how C and C++ work and how the languages gel it has a ton of devils in the details. In this post, I’m going to describe the way I implemented #embed in both GCC and Clang and the style of work I used to support the few companies/vendors I did for an early version of #embed. I’ll use the publicly available version of #embed that I offered to Clang as a tool to display one of the usable techniques to get the guaranteed speedup for the subset of cases that matter (e.g., char/signed char/unsigned char array initialization).

Let’s get started.

Support Level 0: Basic #embed Expansion

Before we talk about the fast version of #embed, we need to discuss what it is specified to be. Consider the following two data files:

single_byte.txt:

a

art.txt:

           __  _
       .-.'  `; `-._  __  _
      (_,         .-:'  `; `-._
    ,'o"(        (_,           )
   (__,-'      ,'o"(            )>
      (       (__,-'            )
       `-'._.--._(             )
          |||  |||`-'._.--._.-'
                     |||  |||

We posit these are UTF-8 encoded text files, meaning the byte value of a is 97 (hexadecimal 0x61) with a size of 1 for the single_byte.txt, and the art.txt file has multiple values with a size of 275 (including the trailing \n newline). We then deploy these files using #embed, a new directive standardized in C23 and in-progress for standardization for C++26:

const unsigned char arr[] = {
#embed <art.txt>
};

int main () {
	return
#embed <single_byte.txt>
	;
}

The way #embed works is, conceptually, very simple: the preprocessor (stages 1 through 4 of the 7 stage compilation process of C and C++) expands the directive, according to any embed parameters and the file, and produces an “comma-delimited list of integer constant expressions” (or “integral constant expressions cast to unsigned char” for C++, but they mean the same thing here¹). Each value goes from 0 to 255². So, for the files above and the given program, that would look like this:

const unsigned char arr[] = {
0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x5f,
0x5f, 0x20, 0x20, 0x5f, 0x0a, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
0x2e, 0x2d, 0x2e, 0x27, 0x20, 0x20, 0x60, 0x3b, 0x20, 0x60, 0x2d, 0x2e,
0x5f, 0x20, 0x20, 0x5f, 0x5f, 0x20, 0x20, 0x5f, 0x0a, 0x20, 0x20, 0x20,
0x20, 0x20, 0x20, 0x28, 0x5f, 0x2c, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
0x20, 0x20, 0x20, 0x2e, 0x2d, 0x3a, 0x27, 0x20, 0x20, 0x60, 0x3b, 0x20,
0x60, 0x2d, 0x2e, 0x5f, 0x0a, 0x20, 0x20, 0x20, 0x20, 0x2c, 0x27, 0x6f,
0x22, 0x28, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x28, 0x5f,
0x2c, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
0x29, 0x0a, 0x20, 0x20, 0x20, 0x28, 0x5f, 0x5f, 0x2c, 0x2d, 0x27, 0x20,
0x20, 0x20, 0x20, 0x20, 0x20, 0x2c, 0x27, 0x6f, 0x22, 0x28, 0x20, 0x20,
0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x29, 0x3e,
0x0a, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x28, 0x20, 0x20, 0x20, 0x20,
0x20, 0x20, 0x20, 0x28, 0x5f, 0x5f, 0x2c, 0x2d, 0x27, 0x20, 0x20, 0x20,
0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x29, 0x0a, 0x20,
0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x60, 0x2d, 0x27, 0x2e, 0x5f, 0x2e,
0x2d, 0x2d, 0x2e, 0x5f, 0x28, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x29, 0x0a, 0x20, 0x20, 0x20, 0x20,
0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x7c, 0x7c, 0x7c, 0x20, 0x20, 0x7c,
0x7c, 0x7c, 0x60, 0x2d, 0x27, 0x2e, 0x5f, 0x2e, 0x2d, 0x2d, 0x2e, 0x5f,
0x2e, 0x2d, 0x27, 0x0a, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
0x20, 0x7c, 0x7c, 0x7c, 0x20, 0x20, 0x7c, 0x7c, 0x7c, 0x0a
};

int main () {
	return
0x61
	;
}

Simple enough. The problem with this — which is the problem with depending on program outputs from e.g. xxd -i or random python scripts you wrote because xxd is packaged only VIA vim for some inexplicable reason — is that it is slow. Horrifically slow, in fact. Taking a computer with the following specification:

OS Name: Microsoft Windows 10 Pro
Version: 10.0.19045 Build 19045
System Type: x64-based PC
Processor: AMD Ryzen 9 5950X 16-Core Processor, 3401 MHz, 16 Core(s), 32 Logical Processor(s)
Installed Physical Memory (RAM): 32.0 GB
Total Physical Memory: 31.9 GB
Total Virtual Memory: 36.7 GB

and dropping in a simple 40 MB file potato.bin filled with random data processed through xxd -i takes over 70 seconds to process. And, the worst part is, no matter how much we try to optimize a C++ frontend to parse things faster, the numbers do not get any better! So, we know expanding to a list of integer constants is very bad for build speed: why, then, is #embed specified in this manner? The reality on-the-ground is that C compilers are very weak creatures. Compared to the central 4/5 C++ compilers that exist in the world, there are easily over 100 different C compilers of varying flavors, powers, and implementation effort. At the end of the day, we had to write a specification that allowed the world’s worst compiler to continue being the world’s worst compiler (presumably, because their implementers are making a tradeoff for some other aspect of C they like more).

Therefore, at support level 0, just “expanding to a list of integers” (or a single integer if there is only one byte in the file) is the core behavior. This behavior is not entirely useless, however, and it will notably be used for some of the more interesting cases we will start outlining as we keep on implementing more and more specialized behavior to increase speed.

The first step is, obviously, adding flags to ensure that the compiler frontend knows where to find data. Do NOT use #include paths for this specified through -I: this is a surefire way to make life for users terrifically annoying and difficult and pull in inclusion of headers or data nobody ever wanted. Use a separate flag that provides directories for this. The implementation I made for Clang used -embed-dir WHATEVER and -embed-dir=WHATEVER. Given my data is in a directory called ./media, the invocation would look like: clang -embed-dir=./media/ -o main.exe main.c. All of the search directories are accumulated in order; additionally, the current directory of the file we are working with (e.g., main.c) is used for lookup when #embed "whatever.h" (with quotes) is used.

Now that we can find the files, the way this works in Clang is simple. We create a pseudo-file inside of the compiler, give it a fancy name, and then quite literally just dump the integer literal tokens into it. Stepping back, this:

const unsigned char arr[] = {
#embed <art.txt>
};

int main () {
	return
#embed <single_byte.txt>
	;
}

Is more faithfully represented by a multi-file split:

////////////////////////////////////////////////
// Enter `main.xxd.cpp`
////////////////////////////////////////////////
const unsigned char arr[] = {
////////////////////////////////////////////////
// Enter `art.txt`-generated
// file internally named `<built-in:embed:1>`
////////////////////////////////////////////////
0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x5f,
0x5f, 0x20, 0x20, 0x5f, 0x0a, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
0x2e, 0x2d, 0x2e, 0x27, 0x20, 0x20, 0x60, 0x3b, 0x20, 0x60, 0x2d, 0x2e,
0x5f, 0x20, 0x20, 0x5f, 0x5f, 0x20, 0x20, 0x5f, 0x0a, 0x20, 0x20, 0x20,
0x20, 0x20, 0x20, 0x28, 0x5f, 0x2c, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
0x20, 0x20, 0x20, 0x2e, 0x2d, 0x3a, 0x27, 0x20, 0x20, 0x60, 0x3b, 0x20,
0x60, 0x2d, 0x2e, 0x5f, 0x0a, 0x20, 0x20, 0x20, 0x20, 0x2c, 0x27, 0x6f,
0x22, 0x28, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x28, 0x5f,
0x2c, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
0x29, 0x0a, 0x20, 0x20, 0x20, 0x28, 0x5f, 0x5f, 0x2c, 0x2d, 0x27, 0x20,
0x20, 0x20, 0x20, 0x20, 0x20, 0x2c, 0x27, 0x6f, 0x22, 0x28, 0x20, 0x20,
0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x29, 0x3e,
0x0a, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x28, 0x20, 0x20, 0x20, 0x20,
0x20, 0x20, 0x20, 0x28, 0x5f, 0x5f, 0x2c, 0x2d, 0x27, 0x20, 0x20, 0x20,
0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x29, 0x0a, 0x20,
0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x60, 0x2d, 0x27, 0x2e, 0x5f, 0x2e,
0x2d, 0x2d, 0x2e, 0x5f, 0x28, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x29, 0x0a, 0x20, 0x20, 0x20, 0x20,
0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x7c, 0x7c, 0x7c, 0x20, 0x20, 0x7c,
0x7c, 0x7c, 0x60, 0x2d, 0x27, 0x2e, 0x5f, 0x2e, 0x2d, 0x2d, 0x2e, 0x5f,
0x2e, 0x2d, 0x27, 0x0a, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
0x20, 0x7c, 0x7c, 0x7c, 0x20, 0x20, 0x7c, 0x7c, 0x7c, 0x0a
////////////////////////////////////////////////
// Return to `main.xxd.cpp`
////////////////////////////////////////////////
};

int main () {
	return
////////////////////////////////////////////////
// Enter `single_byte.txt`-generated
// file internally named `<built-in:embed:2>`
////////////////////////////////////////////////
0x61
////////////////////////////////////////////////
// Return to `main.xxd.cpp`
////////////////////////////////////////////////
	;
}

Internally, there is just a memory buffer called <built-in:embed:1> (where 1 just represents it’s the first file being inserted, 2 would be for the second, and so on and so forth). It is presented as a “file”, and we just “enter” that memory buffer as a “file” and parse it like normal. Very simple stuff, it behaves exactly like #include. As a compiler developer, you also need to make sure you update your support for /showIncludes file dependency generation (MSVC) or -MMD Makefile dependency generation (GCC, Clang, or literally most other compilers). This allows #embed to work pretty much out-of-the-box with your makefile generators and other types of dependency-parsing tools that exist out in the wild, without requiring any updates on the build system side.

A Clang-Specific Explosion

Another Clang-specific part of this that’s very awkward is that you need to generate an actual in-memory source file with this data in it, rather than just directly creating a token stream and pushing that into the compiler’s frontend to handle. The reason here is much less language-design oriented and more compiler-architecture oriented. An earlier version of this code simply generated a sequence of tokens and jammed it back into the parser to deal with. This suddenly caused an inadvertent, potentially infinite number of out-of-bounds reads the part of Clang responsible for dumping token representations back to represent a fully preprocessed file.

The problem was that, somewhat hilariously, rather than Clang hardcoding the write out of things such as comma tokens by using a "," in the compiler’s code to be dumped into the output stream for the preprocessed file, it would simply assume there was a comma in the (original or generated) source code that represented the comma token. That caused the Clang “write preprocessed file” action to go look up a source location for a magic comma token that was being generated but had no backing source data in its SourceManager, and whose source location was just pointing at where the #embed had been. The result was effectively performing random reads of unknown data and piping that straight into the output stream.

It was a fun bug to track down:

Compiler-specific shenanigans aside…

If you could get away with generating tokens directly rather than source code, that would save you a bit of time performing what most compilers call “tokenization” of source code. But, because I did not feel like dealing with Clang’s source location-based assumptions, I simply generated a source file and had clang process that instead. This results in a fraction of lost time (not too significant, really, but still some work always takes longer than simply not doing the work at all). A more optimized version of this would sidestep these problems deftly and avoid having to re-tokenize raw generated source code back into a sequence of {integer literal} {comma} {integer literal} … tokens.

Nevertheless, solving this issue meant that we could dump out a fully preprocessed file when given the -E option. This meant that specific C and C++ tools that just preprocessed source files and did not retain include flag or embed directory information could reliably parse/process these all-bits-included files that just had the integer list expansion baked right in. This served as the baseline support for #embed. There was just one more thing to do to round out Level 0 support…

Support Level 0, Part II: Preprocessor Parameters

Preprocessor parameters are a newer way to pass additional information to preprocessor directives in C23. They are a whitespace-delimited sequence of foo, bar(potentially-empty-balanced-token-sequence) vendor::baz, or vendor::quux(potentially-empty-balanced-token-sequence) arguments that can be given to a preprocessor directive. They only utilized for the #embed directive at the moment, but as compiler implementers find their bravery to actually start implementing extensions again instead of just constantly poking the Standards Committee to act first, it may start showing up in other places as a means to perform fun tasks.

Fun ideas aside, there’s 4 different preprocessor parameters that are mandated by the C standard for #embed: limit, prefix, suffix, and if_empty.

• limit( constant-expression ) takes an integer constant expression and lets a file be up to (but no bigger than) the provided limit. This is useful for #embed <infinity_file> limit(value), like #embed </dev/urandom> limit(64).
• prefix(balanced-token-sequence)/suffix(balanced-token-sequence) both take a sequence of tokens and apply it to the beginning or end of any generated integer token sequence, respectively. If there is no data in the file (or if it is set to limit(0), which will trigger the file to be considered empty), then this parameter has no effect.
• if_empty(balanced-token-sequence) takes the sequence of tokens and expands the directive to those tokens, if there is no data in the file (or if it is set to limit(0)).

Implementing these are not hard: all one has to do is drop the token/text sequence out where expected. So when one encounters the #embed directive and parses the token sequence for prefix or suffix, all they need to do is search for the file. If it’s empty, then they ignore either of the tokens; otherwise, it gets placed before or after the embed directive’s contents. Conversely, if if_empty is present, and the file is empty, then the token sequence appears where the integer sequence would have.

limit(…) is just doing min(limit-expression, size-of-file); if the file size is larger than the limit, than the limit should be chosen. Otherwise, the file size should be chosen. limit(…) specifically refers to the number of integer literals that will be created in the sequence list, and not necessarily the number of bytes. They hold as a 1:1 correlation on most implementations (e.g., CHAR_BIT == size-of-filesystem-byte), but care needs to be taken on the World’s Weirdest Implementations™ (e.g., CHAR_BIT == 9 and fs-byte == 8, or similar foolish shenanigans). The actual wording in the specification for C and C++ has protections against this, but very literally talking about the bit size of the file (or the provided limit-expr ✖ bit width), the bit width of each integer literal, and how the second must cleanly divide into the first. A diagnostic is required if it does not cleanly divide. The full available range can then be defined in interval notation as

\([0, min(limit, file size))\).

There is one other parameter that is part of the Clang implementation that was asked for frequently when I was standardizing #embed. Unfortunately, I am not superhuman and did not have enough time to roll it out into the standard. Part of standardization is, of course, Standardizing Existing Practice, and so as part of the next level of support, adding a few vendor-specific parameters will help bolster adding them to the next C standard.

Support Level 1: clang::offset

This will obviously have to be called gnu::offset for GCC, and then everybody will copy from there. But, the goal is effectively to create and offset( constant-expression ) preprocessor parameter. This does exactly what you’d expect: it would drop up to constant-expression elements from the beginning the read data. This also has the chance to turn the data empty as well, if the offset is greater than the data (after the limit is applied). So, for example:

#embed <single_byte.txt> limit(0) /* empty */
#embed <single_byte.txt> offset(1) /* empty */
#embed </dev/urandom> offset(1) limit(1) /* empty */
#embed </dev/urandom> offset(458946493) limit(1) /* empty */

Notably, the last one is not a constraint violation: it simply just does the min(offset-expression, size-of-file). The full available range can then be defined with interval notation as

\([min(offset, limit, file size), min(limit, file size))\).

There are also many more advanced parameters that can be provided, such as a parameter for width( constant-expression ). This would define the number of bits that would be used for each element to generate the integer literal, which could be useful for initializing larger integral types or custom types when the data is type-punned. But, with that done, I could now move on to speeding the whole thing up! Retaining the support for various constructs above is nuanced, as we will see as we start talking about the next level: built-in speed support.

Support Level 2: Speedy Builtins

So, we implemented a basic preprocessor directive and dumped the contents to a file. It:

• is slow for large files even though we’re generating the data directly in the preprocessor;
• has tooling support (e.g. icecc/distcc) through “data is directly inside the generated preprocessed file”;
• allows us to use it in places where only a single expressions (integer literal) is expected, such as return from int main();
• and, works to initialize an array of unsigned char type (or any other type that accepts a list of comma-delimited integer literals).

We need to retain all of these properties, while speeding up the invocation significantly. For this, we implement a compiler-specific built-in. We will call this built-in __builtin_pp_embed. It will take 3 arguments:

• the expected type of each element (for now, always unsigned char);
• the filename as a string literal;
• and, the data encoded as a base64 string literal.

There are more advanced³ versions of this built-in that I have implemented in other versions of this code, but I am not talking about such implementations here. Of course, I am glossing over the most interesting facet of this list: that last bullet point about “base64 string literal”. Some may read that and go ❓❓, and it would not be a bad reaction honestly! It does sound very silly, but it is actually an important facet of the new built-in.

Surviving the -E Tools

One of the requirements for this functionality is that it survives existing tooling. This includes icecc or distcc that employs -E upon the code to generate a single file before throwing it up to a server to build that single preprocessed source file. If you want a “fast” built-in that respects this, then that necessarily means that every time a file is processed with -E — every time data is pulled into a single source file — all of the data must be present. This means that you cannot just put an (absolute) file path into the built-in; icecc and distcc do not replicate the source file tree in any way, shape, or form. Most other tools also do not include full source tree information work this way, nor do any those “send us a single preprocessed file” bug reporting tools for C and C++ toolchains expect your whole working include (and now, embed) directory structure.

Thusly, when you “finish” preprocessing, you need to contain all of the data in a friendly-to-tools manner. Friendly in this case includes being friendly to tools that break source code down into logical source code lines and then use regex to find #include or #embed directives. So, when processing this main.cpp file:

const unsigned char arr[] = {
#embed <art.txt>
};

int main () {
	return
#embed <single_byte.txt>
	;
}

things end up looking like this when you generate the built-in based code after preprocessing (with large comment block annotations, similar to above code examples):

////////////////////////////////////////////////
// Enter `main.cpp`
////////////////////////////////////////////////
const unsigned char arr[] = {
////////////////////////////////////////////////
// Enter `art.txt`-generated
// file internally named `<built-in:embed:1>`
////////////////////////////////////////////////
__builtin_pp_embed(unsigned char, "/home/derp/pp_embed/examples/media/art.txt",
"ICAgICAgICAgICBfXyAgXwogICAgICAgLi0uJyAgYDsgYC0uXyAgX18g"
"IF8KICAgICAgKF8sICAgICAgICAgLi06JyAgYDsgYC0uXwogICAgLCdv"
"IiggICAgICAgIChfLCAgICAgICAgICAgKQogICAoX18sLScgICAgICAsJ"
"28iKCAgICAgICAgICAgICk+CiAgICAgICggICAgICAgKF9fLC0nICAgIC"
"AgICAgICAgKQogICAgICAgYC0nLl8uLS0uXyggICAgICAgICAgICAgKQo"
"gICAgICAgICAgfHx8ICB8fHxgLScuXy4tLS5fLi0nCiAgICAgICAgICAg"
"ICAgICAgICAgIHx8fCAgfHx8Cg==");
////////////////////////////////////////////////
// Return to `main.cpp`
////////////////////////////////////////////////
};

int main () {
	return
////////////////////////////////////////////////
// Enter `single_byte.txt`-generated
// file internally named `<built-in:embed:1>`
////////////////////////////////////////////////
__builtin_pp_embed(unsigned char, "/home/derp/pp_embed/examples/media/single_byte.txt", "YQ==");
////////////////////////////////////////////////
// Return to `main.cpp`
////////////////////////////////////////////////
	;
}

Notice how this source file only contains constructs that are:

• blindly ASCII parse-ready;
• do not require access to the original source files anymore;
• and, understandable as normal C or C++ source code.

This means that icecc/distcc-style tools would not trip up a re-run of the compiler on the single unified source file. Base64 encoding the data in the second string literal argument is important, because data from a file could look like either valid C++ source when it is meant to be data or could contain bytes in the data that would absolutely destroy traditional/typical C and C++ tooling (like actual embedded nulls).

A Poorly Conceived Idea

A few C++ implementers had a poorly thought-through idea for how to handle this during -E processing. Particularly, their idea was to inject a special, compiler-specific _Pragma rather than something like __builtin_pp_embed; it would indicate the number of bytes of the #embed‘d file before dumping the data raw into the source file. As you can imagine, doing the _Pragma would mean the fully-preprocessed version of this file:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int main(void) {
	char src[] = {
#embed __FILE__
	}, *argv[] = { "./out", NULL };
	FILE *fd = fopen("src.c", "w+");
	fwrite(src, sizeof(src), 1, fd);
	fclose(fd);
	system("${CC} src.c -o out");
	return execv(argv[0], argv);
}

Would trip most tools up. Tools would not understand a generated compiler-specific _Pragma/#pragma that would contain C++ source code, such as:

/* stdio.h expansion here */
/* stdlib.h expansion here */
/* unistd.h expansion here */

int main(void) {
	char src[] = {
///////////////
// start pragma
#pragma embed 286 #include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int main(void) {
	char src[] = {
#embed __FILE__
	}, *argv[] = { "./out", NULL };
	FILE *fd = fopen("src.c", "w+");
	fwrite(src, sizeof(src), 1, fd);
	fclose(fd);
	system("${CC} src.c -o out");
	return execv(argv[0], argv);
}
///////////////
// end pragma
	}, *argv[] = { "./out", NULL };
	FILE *fd = fopen("src.c", "w+");
	fwrite(src, sizeof(src), 1, fd);
	fclose(fd);
	system("${CC} src.c -o out");
	return execv(argv[0], argv);
}

This is, of course, a travesty of new lines and other directives nested in on itself. This absolutely destroys and breaks tooling built on top of -E preprocessed source files. Therefore, the data must be turned into a form that is palpably understandable by something that can handle “regex for function calls” or “regex after logical line processing for preprocessor directives”. Anything that interferes with that idea breaks too much tooling to be (widely) viable, though it may be suitable for internal-only processing. However, if someone has a compiler with a fused preprocessor, C or C++ language frontend, and backend, they could skip this hullabaloo about _Pragmas or built-ins or what-have-you and just blast the memory into the optimal place in the compiler on the first go-around.

All in all, not a worthwhile long-term implementation strategy and one I almost lost a bunch of time trying to make happen; here’s to you not having to make the same mistake as I did.

Nevertheless,

Adding support for this is actually more complicated than imagined. For example, because this is a preprocessor directive, melting things down into a built-in can produce many surprising consequences for where it appears. It’s not just return statements or function invocations; it can appear in arguments, in template parameters, in places where nothing is expected, and so much more:

#embed <media/empty>
;

void f (unsigned char x) { (void)x;}
void g () {}
void h (unsigned char x, int y) {(void)x; (void)y;}
int i () {
	return
#embed <single_byte.txt>
		;
}

_Static_assert(
#embed <single_byte.txt> suffix(,)
""
);
_Static_assert(
#embed <single_byte.txt>
, ""
);
_Static_assert(sizeof(
#embed <single_byte.txt>
) ==
sizeof(unsigned char)
, ""
);
_Static_assert(sizeof
#embed <single_byte.txt>
, ""
);
_Static_assert(sizeof(
#embed <jk.txt>
) ==
sizeof(unsigned char)
, ""
);

#ifdef __cplusplus
template <int First, int Second>
void j() {
	static_assert(First == 'j', "");
	static_assert(Second == 'k', "");
}
#endif

void do_stuff() {
	f(
#embed <single_byte.txt>
	);
	g(
#embed <media/empty>
	);
	h(
#embed <jk.txt>
	);
	int r = i();
	(void)r;
#ifdef __cplusplus
	j<
#embed <jk.txt>
	>(
#embed <media/empty>
	);
#endif
}

This is what cost the majority of the implementation time when working on the built-in. Because the built-in is generated by the preprocessor but parsed by the frontend, handling this was the largest portion of what made it difficult. There’s a few things we did to make all of this work out rather simply, at least in Clang. Different compilers have different architectures, but many of these ideas can be applied universally.

0. Implement __builtin_pp_embed as a Keyword

Trying to parse __builtin_pp_embed as a “built-in function call” versus just grafting support directly into the parser with __builtin_pp_embed explicitly as a keyword is sufficiently more nightmarish. It opts into a lot of mechanisms and code around function calls that assume a single return (not the case for empty embeds or embeds that produce multiple integer literals). It absolutely requires manual tweaking if you want to do things like read a type name as an argument without having your typical function-body parser explode. There are also rules in both C and C++ that automatically decay arrays to pointers when put in normal function calls, making it difficult to retrieve information when you get to the “Semantic Analysis” part of working with code.

Instead, parsing __builtin_pp_embed as a keyword and then simply expecting the parentheses, type name, file name string literal, and string literal arguments results in far less code and far less post-hoc adjustments. It’s also marginally faster than reverse-engineering the proper data during Semantic Analysis and Constant Expression parsing. Internally, this produces a distinct PPEmbedExpr that contains the base64-decoded data as a StringLiteral (Clang) or a distinct VECTOR_CST-style tree node with a string node stored as part of the VECTOR_CST’s data and pattern (GCC). Most compilers have special __builtin_* markers that are treated as private keywords, even TCC. This implementation technique allows you to get right into the special internal format necessary for later processing and speed recognition.

1. Stringent Speed Requirements

In your compiler architecture, you want to eliminate any node or leaf object that represents the built-in as soon as is technologically possible. Specifically for Clang, this is possible by recognizing a sequence of conditions:

• if the built-in is being used to initialize an array of character type (e.g. char or unsigned char or even signed char);
• and if there is only ONE initializer in the list of initializers for an object that is the built-in;

then, the built-in node or tree element just gets replaced with a magic string literal that was generated from the decoded base64 data. The realization here comes from noting that string literals are, quite literally, the fastest array initializers in almost every C and C++ compiler today. String literals and their initializers are also often rarely copied, making them supremely ideal for the goal of initializing these arrays. This also prevents us from having to give a single damn about further downstream portions of either Clang or GCC’s compiler architecture: just substitute in a single string literal and let the usual “array initialization from a string literal” take hold.

This seemed like a hack, because it meant I did not have to really touch much if any-at-all semantic analysis of Clang’s compiler (earlier versions of my patch got completely lost on the SemaExpr sauce and the constant expression parser trying to gain bigger and bigger speedups) nor did I have to so much as look at the Code Gen. But, it actually paid off enormously, in both implementation speed, implementation correctness, and end result speed. I encourage almost every single compiler to follow the above 2 guidelines; if they cannot form a single initializer with all of the provided initializers so that it can simply be folded down into a typical array initialization of one of the character arrays, then go to Step #2.

2. Aggressively Expand Everywhere Else

If the two conditions above aren’t met and the initializer data cannot be massaged into the moral, spiritual, and factual equivalent of a string literal initialization, expand the directive. This is where things become really difficult, because not everything is allowed to expand in-place. For example, the return __builtin_pp_embed(…); statement cannot handle having 2 integers present. It can work with 1 integer, or with 0 (for a void function that does return ;). This requires manual diagnose when handling a return expression, but has to be done as early as possible in either the parser, or the semantic analyzer. This is where implementation difficulty fully ramps up, and is where I spent the least amount of time for the patch. A lot of things work correctly in the basic case, but extended and honestly completely asinine usages of #embed can and do break the compiler.

Because the compiler was not built to accept a comma-delimited list of integers anywhere and everywhere, the idea that a single expression — __builtin_pp_embed(…) — could turn into one is a fine way to make every part of the compiler scream. So, instead, I focused my energy on getting things correct for the typical usages and a few odd places, and leaving the rest to, effectively, undefined behavior and fate.

Thankfully, there’s a few key places in Clang where All Function Call Arguments are finalized/massaged, and central locations where All Template Arguments are processed, so the two big cases where this may happen are easy to handle. Initialization also has a Single Coalescing Location, which takes care of all structure and array initialization and makes it easy to get the speedup. It’s all the tiny little stragglers that need to be cleaned up, and that’s where my energy levels hit rock bottom. Having already done a lot of this boilerplate over the last 5 years, implementing it all again for #embed and std::embed over and over and over and over is… draining!

3. Recognize Simple Cases in the Preprocessor

Another way to avoid problems with #embed ruining things in unexpected ways in your compiler is — when doing the transition from #embed to __builtin_pp_embed in the preprocessor — simply not generating the built-in when it would not be useful. So, for example, if you detect that the file is empty (limit is too small or offset is too high or both, or the file is legitimately empty), there is absolutely no reason to produce the built-in at all. Just expand it out to nothing and leave. If there’s an if_empty(…), just pour the text directly into a buffer, make it a new file, and deem that the expansion. Let it parse normally, like any other preprocessor expansion. It avoids a wide class of issues related to “how do I delete this tree node / expression leaf out from itself?!”. The inverse of this, of course, is…

4. Recognize When Something Is Not Built-in-able

If you took the advice in the preceding section, then the only time you’re going to make a built-in is if there is data. So, we know for a fact that any suffix(…) or prefix(…) has to be valid and will be put out by the directive. If there is a suffix(…) or prefix(…) parameter, you can do a quick check to see if it is worth your time to turn it into a built-in. For prefixes, check if there is a comma delimited list of integers that end with a comma. For suffixes, check if it starts with a comma and then becomes a comma delimited list of integers after that. If one or both of these hold true, then you can just immediately slurp that data up into whatever binary data was produced, making sure that each integer constant is within the range \([0, 2^{CHAR\_BIT})\). Then you emit a single __builtin_pp_embed without adding any additional tokens before or after it.

If either is not, then just bail and expand the list of integers as programmed in Support Level 0. This was actually pointed out during the C Meeting by Joseph Myers — a prominent contributor to glibc, GCC’s C code, and several other highly used projects — as something implementations can do to keep code optimized as early as possible and not trip up the conditions above. This was also a primary reason why suffix(…) and prefix(…) were kept as embed parameters, despite being able to program this in multiple different ways. For example, all of these will make a null-terminated string:

const char non_optimized0[] = {
#embed "shaders/super_glitter.glsl"
	, 0
};

const char non_optimized1[] = {
#embed "shaders/super_glitter.glsl"
#if __has_embed("shaders/super_glitter.glsl") == __STDC_EMBED_FOUND__
	,
#endif
	0
};

const char optimized0[] = {
#embed "shaders/super_glitter.glsl" \
	suffix(, 0)
};

It is worth noting that non_optimized0 also just completely breaks if shaders/super_glitter.glsl is empty. But, assuming there is data in the file, the first two will not optimize cleanly; the current directive will just vomit it out into a mess. Contrarily, the last one will much more easily be optimized by most implementations. Doing this for both prefix(…) and suffix(…) will become increasingly important as people use it to do things such as provide integer sequences that map to things such as "#version 420 or other common top-level string boilerplate for all sorts of files inside of prefix() or suffix() clauses.

But…

That’s it, in terms of implementation prowess. Ostensibly, Support Level 0 is enough to be a conforming implementation. There are tons of examples in the Clang pull request and other places for this. There are also many more extensions that can be implemented for this functionality. I can only hope that implementers that read this are emboldened to add more directives, get spicy with how they implement things, and try expanding on their techniques into the future. A stagnant implementer culture that always wants to reach for assured, standards-mandated things is no fun in a world as vast and as lovely as Computing. And, of course, having dreams and realizing them means that all of us, together, get to see…

Just how much faster things can be compared to the tools we’ve been using for 40+ years:

A better future is possible. A future that’s at least 37x as fast as the one we’re living in. We just have to grasp it.

With our own two hands. 💚⁴

Footnotes

Following up from the last post, there is a lot more we need to cover. This was intended to be the post where we talk exclusively about benchmarks and numbers. But, I have unfortunately been perfectly taunted and status-locked, like a monster whose “aggro” was pulled by a tank. The reason, of course, is due to a few folks taking issue with my outright dismissal of the C and C++ APIs (and not showing them in the last post’s teaser benchmarks).

Therefore, this post will be squarely focused on cataloguing the C and C++ APIs in detail, and how to design ourselves away from those mistakes in C.

Part of this post will add to the table from Part 1, talking about why the API is trash (rather than just taking it for granted that industry professionals, hobbyists, and academic experts have already discovered how trash it is). I also unfortunately had to add it to the benchmarks we are doing (which means using it in anger). As a refresher, here’s where the table we created left off, with all of what we discovered (including errata from comments people sent in):

In this article, what we’re going to be doing is sizing up particularly the standard C and C++ interfaces, benchmarking all of the APIs in the table, and discussing in particular the various quirks and tradeoffs that come with doing things in this manner. We will also be showing off the C-based API that we have spent all this time leading up to, its own tradeoffs, and if it can tick all of the boxes like ztd.text does. The name of the C API is going to be cuneicode, a portmanteau of Cuneiform (one of the first writing systems) and Unicode (of Unicode Consortium fame).

First, we are going to thoroughly review why the C API is a failure API, and all the ways it precipitates the failures of the encoding conversions it was meant to cover (including the existing-at-the-time Big5-HKSCS case that it does not support).

Then, we will discuss the C++-specific APIs that exist outside of the C standard. This will include going beneath std::wstring_convert’s deprecated API, to find that powers the string conversions that it used to provide. In particular, we will discuss std::codecvt<ExternCharType, InternCharType, StateObject> and the various derived classes std::codecvt(_utf8/_utf16/_utf8_utf16). We will also talk about how the C API’s most pertinent failure leaks into the C++ API, and how that pitfall is the primary reason why Windows, specific IBM platforms, lots of BSD platforms, and more cannot properly support UTF-16 or UTF-32 in its core C or C++ standard library offerings.

Finally, we will discuss ztd.cuneicode / cuneicode, a C library for doing encoding conversions that does not make exceedingly poor decisions in its interfaces.

Standard C

Standard C’s primary deficiency is its constant clinging to and dependency upon the “multibyte” encoding and the “wide character” encoding. In the upcoming C23 draft, these have been clarified to be the Literal Encoding (for "foo" strings, at compile-time (“translation time”)), the Wide Literal Encoding (for L"foo" strings, at compile-time), Execution Encoding (for any const char*/[const char*, size_t] that goes into run time (“execution time”) function calls), and Wide Execution Encoding (for any const wchar_t*/[const wchar_t*, size_t] that goes into run time function calls). In particular, C relies on the Execution Encoding in order to go-between UTF-8, UTF-16, UTF-32 or Wide Execution encodings. This is clear from the functions present in both the <wchar.h> headers and the <uchar.h> headers:

// From Execution Encoding to Unicode
size_t mbrtoc32(char32_t* restrict pc32, const char* restrict s,
	size_t n, mbstate_t* restrict ps );
size_t mbrtoc16(char16_t* restrict pc16, const char* restrict s,
	size_t n, mbstate_t* restrict ps );
size_t mbrtoc8(char8_t* restrict pc8, const char* restrict s,
	size_t n, mbstate_t* restrict ps ); // ⬅ C23 addition

// To Execution Encoding from Unicode
size_t c32rtomb(char* restrict s, char32_t c32,
	mbstate_t* restrict ps);
size_t c16rtomb(char* restrict s, char16_t c16,
	mbstate_t* restrict ps);
size_t c8rtomb(char* restrict s, char8_t c8,
	mbstate_t* restrict ps); // ⬅ C23 addition

// From Execution Encoding to Wide Execution Encoding
size_t mbrtowc(wchar_t* restrict pwc, const char* restrict s,
	size_t n, mbstate_t* restrict ps);
// Bulk form of above
size_t mbsrtowcs(wchar_t* restrict dst, const char** restrict src,
	size_t len, mbstate_t* restrict ps);
// From Wide Execution Encoding to Execution Encoding
size_t wcrtomb(char* restrict s, wchar_t wc,
	mbstate_t* restrict ps);
// Bulk form of above
size_t wcsrtombs(char* restrict dst, const wchar_t** restrict src,
	size_t len, mbstate_t* restrict ps);

The naming pattern is “(prefix)(s?)(r)to(suffix)(s?)”, where s means “string” (bulk processing), r” means “restartable” (takes a state parameter so it a string can be re-processed by itself), and the core to which is just to signify that it goes to the prefix-identified encoding to the suffix-identified encoding. mb means “multibyte”, wc means “wide character”, and c8/16/32 are “UTF-8/16/32”, respectively.

Those are the only functions available, and with it comes an enormous dirge of problems that go beyond the basic API design nitpicking of libraries like simdutf or encoding_rs/encoding_c. First and foremost, it does not include all the possible pairings of encodings that it already acknowledges it knows about. Secondly, it does not include full bulk transformations (except in the case of going between execution encoding and wide execution encoding). All in all, it’s an exceedingly disappointing offering, as shown by the tables below.

For “Single Conversions”, what’s provided by the C Standard is as follows:

For “Bulk Conversions”, what’s provided by the C Standard is as follows:

As with the other table, the “✅” is for that conversion sequence being supported, and the “❌” is for no support. As you can see from all the “❌” in the above table, we have effectively missed out on a ton of functionality needed to go to and from Unicode encodings. C only provides bulk conversion functions for the “mbs”/”wcs” series of functions, meaning you can kiss any SIMD or other bulk-processing optimizations goodbye for just about every other kind of conversion in C’s API, including any UTF-8/16/32 conversions. Also note that C23 and C++20/23 had additional burdens to fix:

• u"" and U"" string literals did not have to be UTF-16 or UTF-32 encoded;
• c16 and c32 did not have to actually mean UTF-16 or UTF-32 for the execution encoding;
• the C and C++ Committee believed that mb could be used as the “UTF-8” through the locale, which is why they left c8rtomb and mbrtoc8 out of the picture entirely until it was fixed up in C23 by Tom Honermann.

These are no longer problems thanks to C++’s Study Group 16, with papers from R.M. Fernandes, Tom Honermann, and myself. If you’ve read my previous blog posts, I went into detail about how the C and C++ implementations could simply define the standard macros __STDC_UTF_32__ and __STDC_UTF_16__ to 0. That is, an implementation could give you a big fat middle finger with respect to what encoding was being used by the mbrtoc16/32 functions, and also not tell you what is in the u"foo" and U"foo" string literals.

This was an allowance that we worked hard to nuke out of existence. It was imperative that we did not allow yet another escalation of char16_t/char32_t and friends ending up in the same horrible situation as wchar_t where it’s entirely platform (and possibly run time) dependent what encoding is used for those functions. As mentioned in the previously-linked blog post talking about C23 improvements, we were lucky that nobody was doing the “wrong” thing with it and always provided UTF-32 and UTF-16. This made it easy to hammer the behavior in all current and future revisions of C and C++. This, of course, does not answer why wchar_t is actually something to fear, and why we didn’t want char16_t and char32_t to become either of those.

So let’s talk about why wchar_t is literally The Devil.

C and wchar_t

This is a clause that currently haunts C, and is — hopefully — on its way out the door for the C++ Standard in C++23 or C++26. But, the wording in the C Standard is pretty straight forward (emphasis mine):

wide character

value representable by an object of type wchar_t, capable of representing any character in the current locale

— §3.7.3 Definitions, “wide character”, N3054

This one definition means that if you have an input into mbrtowc that needs to output more than one (1) wchar_t for the desired output, there is no appreciable way to describe that in the standard C API. This is because there is no reserved return code for mbrtowc to describe needing to serialize a second wchar_t into the wchar_t* restrict pwc. Furthermore, despite being a pointer type, pwc expects only a single wchar_t to be output into it. Changing the definition in future standards to allow for 2 or more wchar_t’s to be written to pwc is a recipe for overwriting the stack for code that updates its standard library but does not re-compile the application to use a larger output buffer. Taking this kind of action would ultimately end up breaking applications in horrific ways (A.K.A, an ABI Break), so it is fundamentally unfixable in Standard C.

This is why encodings like Big5-HKSCS cannot be used in Standard C. Despite libraries advertising support for them like glibc and its associated locale machinery, they return non-standard and unexpected values to cope with inputs that need to write out two UTF-32 code points for a single indivisible unit of input. Most applications absolutely cannot cope with these return values, and so they start just outputting straight up garbage values as they don’t know how to bank up the characters and perform reads anymore, let alone do writes. It’s doubly-fun when others get to see it in real-time, too:

oh wow, even better: glibc goes absolutely fucking apeshit (returns 0 for each mbrtowc() after the initial one that eats 2 bytes; herein wc modified to write the resulting character)

— наб, July 9th, 2022

This same reasoning applies to distributions that attempt to use UTF-16 as their wchar_t encoding. Similar to how Big5-HKSCS requires two (2) UTF-32 code points for some inputs, large swaths of the UTF-16 encoded characters use a double code unit sequence. Despite platforms like e.g. Microsoft Windows having the demonstrated ability to produce wchar_t strings that are UTF-16 based, the standard library must explicitly be an encoding that is called “UCS-2”. If a string is input that requires 2 UTF-16 code units — a leading surrogate code unit and its trailing counterpart surrogate code unit — there is no expressible way to do this in C. All that happens is that, even if they recognize an input sequence that generates 2 UTF-16 wchar_t code units, it will get chopped in half and the library reports “yep, all good”. Microsoft is far from the only company in this boat: IBM cannot fix many of their machines, same as Oracle and a slew of others that are ABI-locked into UTF-16 because they tried to adopt Unicode 1.0 faster than everyone else and got screwed over by the Unicode Consortium’s crappy UCS-2 version 1.0 design.

This, of course, does not even address that wchar_t on specific platforms does not have to be either UTF-16 or UTF-32, and thanks to some weasel-wording in the C Standard it can be impossible to detect if even your string literals are appropriately UTF-32, let alone UTF-16. Specifically, the predefined macro __STDC_ISO_10646__ can actually be turned off by a compiler because, as far as the C Committee is concerned, it is a reflection of a run time property (whether or not mbrtowc can handle UTF-32mor UTF-16, for example), which is decided by locale (yes, wchar_t can depend on locale, like it does on several IBM and *BSD-based machines). Thusly, as __STDC_ISO_10646__ is a reflection of a run time property, it becomes technically impossible to define before-hand, at compile time, in the compiler.

So, the easiest answer — even if your compiler knows it encodes L"foo" strings as 32-bit wchar_t with UTF-32 code points — is to just advertise its value 0. It’s 100% technically correct to do so, and that’s exactly what compilers like Clang do. GCC would be in a similar boat as well, but they cut a backdoor implementation deal with numerous platforms. A header called stdc-predef.h is looked up at the start of compilation and contains a definition determining whether __STDC_ISO_10646__ may be considered UTF-32 for the platform and other such configuration parameters. If so, GCC sets it to 1. Clang doesn’t want to deal with stdc-predef.h, or lock in any of the GCC-specific ways of doing things in this avenue too much, so they just shrug their shoulders and set it to 0.

I could not fix this problem exactly, myself. It was incredibly frustrating, but ultimately I did get something for a few implementations. In coordination with Corentin Jabot’s now-accepted P1885, I provided named macro string literals or numeric identifiers to identify the encoding of a string literal¹. This allows a person to identify (at least for GCC, Clang, and (now) MSVC) the encoding they use with some degree of reliability and accuracy. The mechanism through which I implemented this and suggested it is entirely compiler-dependent, so it’s not as if other frontends for C or C++ will do this. I hope they’ll follow through and not continue to leave their users out to dry. For C++26 and beyond, Corentin Jabot’s paper will be enough to solve things on the C++ side. C is still left in the dark, but that’s just how it is all the time anyways these days so it’s not like C developers will be any less sad than when they started.

C and “multibyte” const char* Encodings

As mentioned briefly before, the C and C++ Committee believed that the Execution Encoding could just simply be made to be UTF-8. This was back when people still had faith in locales (an attitude still depressingly available in today’s ecosystem, but in a much more damaging and sinister form to be talked about later). In particular, there are no Unicode conversions except those that go through the opaque, implementation-defined Execution Encoding. For example, if you wanted to go from the Wide Execution Encoding (const wchar_t*) to UTF-8, you cannot simply convert directly from a const wchar_t* wide_str string — whatever encoding it may be — to UTF-8. You have to:

• set up an intermediate const char temp[MB_MAX_LEN]; temporary holder;
• call wcrtomb(temp, *wide_str, …);
• feed the data from temp into mbrtoc8(temp, …);
• loop over the wide_str string until you are out of input;
• loop over any leftover intermediate input and write it out; and,
• drain any leftover state-held data by checking mbsinit(state) (if using the r-based restartable functions).

The first and most glaring problem is: what happens if the Execution Encoding is not Unicode? It’s an unfortunately frightfully common case, and as much as the Linux playboys love to shill for their platform and the “everything is UTF-8 by default” perspective, they’re not being honest with you or really anyone else on the globe. For example, on a freshly installed WSL Ubuntu LTS, with sudo apt-get update and sudo apt-get dist-upgrade freshly run, when I write a C or C++ program to query what the locale is with getlocale and compile that program with Clang 15 with as advanced of a glibc/libstdc++ as I can get, this is what the printout reads:

=== Encoding Names ===
Literal Encoding: UTF-8
Wide Literal Encoding: UTF-32
Execution Encoding: ANSI_X3.4-1968
Wide Execution Encoding: UTF-32

If you look up what “ANSI_X3.4-1968” means, you’ll find that it’s the most obnoxious and fancy way to spell a particularly old encoding. That is to say, my default locale when I ask and use it in C or C++ — on my brand new Ubuntu 20.04 Focal LTS server, achieved from just pressing “ok” to all the setup options, installing build essentials, and then going out of my way to get the most advanced Clang I can and combine it with the most up-to-date glibc and libstdc++ I can —

is ASCII.

Not UTF-8. Not Latin-1!

Just ASCII.

Post-locale, “const char* is always UTF-8”, “UTF-8 is the only encoding you’ll need” world, eh? 🙄

Windows fares no better, pulling out the generic default locale associated with my typical location since the computer’s install. This means that if I decide today is a good day to transcode between UTF-16 and UTF-8 the “standard” way, everything that is not ASCII will simply be mangled, errored on, or destroyed. I have to adjust my tests when I’m using code paths that go through standard C or C++ paths, because apparently “Hárold” is too hardcore for Ubuntu 22.04 LTS and glibc to handle. I have long since had to teach not only myself, but others how to escape the non-UTF-8 hell on all kinds of machines. For a Windows example, someone sent me a screenshot of a bit of code whose comments looked very much like it was mojibake’d over Telegram:

Visual Studio was, of course, doing typical Microsoft-text-editor-based Visual Studio things here. It was clear what went down, so I gave them some advice:

And, ‘lo and behold:

Of course, some questions arise. One might be “How did you know it was Windows 1251?”. The answer is that I spent a little bit of time in The Mines™ using alternative locales on my Windows machine — Japanese, Chinese, German, and Russian — and got to experience first-hand how things got pretty messed up by an overwhelming high number of programs. And that’s just the tip of the iceberg: Windows 1251 is the most persistent encoding for Cyrillic data into/out of Central & Eastern Europe, as well as Far North Asia. There’s literally an entire Wiki that contains common text sequences and their Mojibake forms when incorrectly interpreted as other forms of encodings, and most Cyrillic users are so used to being fucked over by computing in general that they memorized the various UTF and locale-based mojibake results, able to read pure mangled text and translate that in-real-time to what it actually means in their head. (I can’t do that: I have to look stuff up.) It’s just absurdly common to be dealing with this:

Me: “Why is the program not working?”
[looks at error logs]
Me: “Aha.”

— Ólafur Waage, May 22nd, 2023

Even the file name for the above embedded image had to be changed from ólafur-tweet.png to waage-tweet.png, because Ruby — when your Windows and Ruby is not “properly configured” (????) — will encounter that name from Jekyll, then proceed to absolutely both crap and piss the bed about it by trying to use the C standard-based sysopen/rb_sysopen on it. By default, that will use the locale-based file APIs on Windows, rather than utilizing the over 2-decade old W-based Windows APIs to open files. It’s extraordinary that despite some 20+ years of Unicode, almost every programming language, operating system, core library, or similar just straight up does not give a single damn about default Unicode support in any meaningful way! (Thank God at least Python tries to do something and gets pretty far with its myriad of approaches.)

There are other ways to transition your application to UTF-8 on Windows, even if you might receive Windows 1251 data or something else. Some folks achieve it by drilling Application Manifests into their executables. But that only works for applications; ztd.text and ztd.cuneicode are libraries. How the hell am I supposed to Unicode-poison an application that consumes my library? The real answer is that there is still no actual solution, and so I spend my time telling others about this crappy world when C and C++ programs inevitably destroy people’s data. But, there is one Nuclear Option you can deploy as a Windows user, just to get UTF-8 by-default as the default codepage for C and C++ applications:

Yep, the option to turn on UTF-8 by default is buried underneath the new Settings screen, under the “additional clocks” Legacy Settings window on the first tab, into the “Region” Legacy Settings window on the second tab (“Administrative”), and then you need to click the “Change system locale” button, check a box, and reset your computer.

But sure, after you do all of that, you get to live in a post-locale world². 🙃

And It Gets Worse

Because of course it gets worse. The functions I listed previously all have an r in the middle of their names; this is an indicator that these functions take an mbstate_t* parameter. This means that the state used for the conversion sequence is not taken from its alternative location. The alternative location is, of course, implementation-defined when you are not using the r-styled functions.

This alternative mbstate_t object might be a static storage duration object maintained by the implementation. It may be thread_local, it may not, and whether or not it is thread safe there is still the distinct horribleness that it is an opaquely shared object. So even if the implementation makes it thread-safe and synchronizes access to it (kiss your performance good-bye!), if, at any point, someone uses the non-r versions of the above standard C functions, any subsequent non-r functions downstream of them have their state changed out from underneath them. Somehow, our systems programming language adopted scripting-language style behavior, where everything is connected to everything else is a jumble of hidden and/or global state, grabbing variables and functionality from wherever and just loading it up willy-nilly. This is, of course, dependable and rational behavior that can and will last for a long time and absolutely not cause severe problems down the road. It definitely won’t lead to ranting screeds³ from developers who have to deal with it.

Of course, even using the r functions still leaves the need to go through the multibyte character set. Even if you pass in your own mbstate_t object, you still have to consult with the (global) locale. If at any point someone calls setlocale("fck.U"); you become liable to deal with that change in further downstream function calls. Helpfully, the C standard manifests this as unspecified behavior, even if we are storing our own state in an mbstate_t! If one function call starts in one locale with one associated encoding, but ends up in another locale with a different associated encoding during the next function call, well. Eat shit, I guess! This is because mbstate_t, despite being the “state” parameter, is still beholden to the locale when the function call was made and the mbstate_t object is not meant to store any data about the locale for the function call! Most likely you end up with either hard crashes or strange, undecipherable output even for what was supposed to be a legal input sequences, because the locale has changed in a way that is invisible to both the function call and the shared state between function calls with mbstate_t.

So, even if you try your hardest, use the restartable functions, and track your data meticulously with mbstate_t, libraries in the stack that may set locale will blow up everyone downstream of them, and applications which set locale may end up putting their upstream dependencies in an untested state of flux that they are entirely unprepared for. Of course, nobody sees fit to address this: there’s no reasonable locale_t object that can be passed into any of these functions, no way of removing the shadowy specter of mutable global state from the core underlying functionality of our C libraries. You either use it and deal with getting punched in the face at seemingly random points in time, or you don’t and rewrite large swaths of your standard library distribution.

All in all, just one footgun after another when it comes to using Standard C in any remotely scalable fashion. It is not surprise that the advice for these functions about their use is “DO. NOT.”, which really inspires confidence that this is the base that every serious computation engine in the world builds on for their low-level systems programming. This, of course, leaves only the next contender to consider: standard C++.

Standard C++

When I originally discussed Standard C++, I approached it from its flagship API — std::wstring_convert<…> — and all the problems therein. But, there was a layer beneath that I had filed away as “trash”, but that could still be used to get around many of std::wstring_convert<…>’s glaring issue. For example, wstring_convert::to_bytes always returns a new std::string-alike by-value, meaning that there’s no room to pre-allocate or pre-reserve data (giving it the worst of the allocation penalty and any pessimistic growth sizing as the string is converted). It also always assumes that the “output” type is char-based, while the input type is Elem-based. Coupled with the by-value, allocated return types, it makes it impossible to save on space or time, or make it interoperable with a wide variety of containers (e.g., TArray<…> from Unreal Engine or boost::static_vector), requiring an additional copy to put it into something as simple as a std::vector.

But, it would be unfair to judge the higher-level — if trashy — convenience API when there is a lower-level one present in virtual-based codecvt classes. These are member functions, and so the public-facing API and the protected, derive-ready API are both shown below:

template <typename InternalCharType,
	typename ExternalCharType,
	typename StateType>
class codecvt {
public:
	std::codecvt_base::result out( StateType& state,
		const InternalCharType* from,
		const InternalCharType* from_end,
		const InternalCharType*& from_next,
		ExternalCharType* to,
		ExternalCharType* to_end,
		ExternalCharType*& to_next ) const;

	std::codecvt_base::result in( StateType& state,
		const ExternalCharType* from,
		const ExternalCharType* from_end,
		const ExternalCharType*& from_next,
		InternalCharType* to,
		InternalCharType* to_end,
		InternalCharType*& to_next ) const;

	// …

protected:
	virtual std::codecvt_base::result do_out( StateType& state,
		const InternalCharType* from,
		const InternalCharType* from_end,
		const InternalCharType*& from_next,
		ExternalCharType* to,
		ExternalCharType* to_end,
		ExternalCharType*& to_next ) const;

	virtual std::codecvt_base::result do_in( StateType& state,
		const ExternalCharType* from,
		const ExternalCharType* from_end,
		const ExternalCharType*& from_next,
		InternalCharType* to,
		InternalCharType* to_end,
		InternalCharType*& to_next ) const;

	// …
}

Now, this template is not supposed to be anything and everything, which is why it additionally has virtual functions on it. And, despite the poorness of the std::wstring_convert<…> APIs, we can immediately see the enormous benefits of the API here, even if it is a little verbose:

• it cares about having both a beginning and an end;
• it contains a third pointer-by-reference (rather than using a double-pointer) to allow someone to know where it stopped in its conversion sequences; and,
• it takes a StateType, allowing it to work over a wide variety of potential encodings.

This is an impressive and refreshing departure from the usual dreck, as far as the API is concerned. As an early adopter of codecvt and wstring_convert, however, I absolutely suffered its suboptimal API and the poor implementations, from Microsoft missing wchar_t specializations that caused wstring_convert to break until they fixed it, or MinGW’s patched library deciding today was a good day to always swap the bytes of the input string to produce Big Endian data no matter what parameters were used, it was always a slog and a struggle to get the API to do what it was supposed to do.

But the thought was there. You can see how this API could be the one that delivered C++ out of the “this is garbage nonsense” mines. Maybe, it could even be the API for C++ that would bring us all to the promised land over C. They even had classes prepared to do just that, committing to UTF-8, UTF-16, and UTF-32 while C was still struggling to get past “char* is always (sometimes) UTF-8, just use the multibyte encoding for that”:

template <typename Elem,
	unsigned long Maxcode = 0x10ffff,
	std::codecvt_mode Mode = (std::codecvt_mode)0
> class codecvt_utf8 : public std::codecvt<Elem, char, std::mbstate_t>;

template <typename Elem,
	unsigned long Maxcode = 0x10ffff,
	std::codecvt_mode Mode = (std::codecvt_mode)0
> class codecvt_utf16 : public std::codecvt<Elem, char, std::mbstate_t>;

template <typename Elem,
	unsigned long Maxcode = 0x10ffff,
	std::codecvt_mode Mode = (std::codecvt_mode)0
> class codecvt_utf8_utf16 : public std::codecvt<Elem, char, std::mbstate_t>;

UTF-32 is supported by passing in char32_t as the Elem element type. The codecvt API was byte-oriented, meaning it was made for serialization. That meant it would do little-endian or big-endian serialization by default, and you had to pass in std::codecvt_mode::little_endian to get it to behave. Similarly, it sometimes would generate or consume byte order markers if you passed in std::codecvt_mode::consume_header or std::codecvt_mode::generate_header (but it only generates a header for UTF-16 or UTF-8, NOT for UTF-32 since UTF-32 was considered the “internal” character type for these and therefore not on the “serialization” side, which is what the “external” character type was designated for). It was a real shame that the implementations were fairly lackluster when it first came out because this sounds like (almost) everything you could want. By virtue of being a virtual-based interface, you could also add your own encodings to this, which therefore made it both compile-time and run-time extensible. Finally, it also contained error codes that went beyond just “yes the conversion worked” and “no it didn’t lol xd”, with the std::codecvt_base::result enumeration:

enum result {
	ok,
	partial,
	error,
	noconv
};

whose values mean:

• result::ok — conversion was completed with no error;
• result::partial — not all source characters were converted;
• result::error — encountered an invalid character; and,
• result::noconv — no conversion required, input and output types are the same.

This is almost identical to ztd.text’s ztd::text::encoding_error type, with the caveat that ztd.text also accounts for the “all source characters could be converted, but the write out was partial” while gluing the result::noconv into its version of result::ok instead. This small difference, however, does contribute in one problem. And that one problem does, eventually, fully cripple the API.

The “1:N” and “N:1” Rule

Remember how this interface is tied to the idea of “internal” and “external” characters, and the normal “wide string” versus the internal “byte string”? This is where something sinister leaks into the API, by way of a condition imposed by the C++ standard. Despite managing to free itself from wchar_t issues by way of having an API that could allow for multiple input and multiple outputs, it reintroduces them by applying a new restriction focused exclusively on basic_filebuf-related containers.

A codecvt facet that is used by basic_­filebuf ([file.streams]) shall have the property that if

do_out(state, from, from_end, from_next, to, to_end, to_next)

would return ok, where from != from_­end, then

do_out(state, from, from + 1, from_next, to, to_end, to_next)

shall also return ok, and that if

do_in(state, from, from_end, from_next, to, to_end, to_next)

would return ok, where to != to_­end, then

do_in(state, from, from_end, from_next, to, to + 1, to_next)

shall also return ok.²⁵²

— Draft C++ Standard, §30.4.2.5.3 [locale.codecvt.virtuals] ¶4

And the footnote reads:

²⁵²⁾ Informally, this means that basic_­filebuf assumes that the mappings from internal to external characters is 1 to N: that a codecvt facet that is used by basic_­filebuf can translate characters one internal character at a time.

— Draft C++ Standard, §30.4.2.5.3 [locale.codecvt.virtuals] ¶4 Footnote 252

In reality, what this means is that, when dealing with basic_filebuf as the thing sitting on top of the do_in/do_out conversions, you must be able to not only convert 1 element at a time, but also avoid returning partial_conv and just say “hey, chief, everything looks ok to me!”. This means that if someone, say, hands you an incomplete stream from inside the file, you’re supposed to be able to read only 1 byte of a 4-byte UTF-8 character, say “hey, this is a good, complete character — return std::codecvt_mode::ok!!”, and then let the file proceed even if it never provides you any other data.

It’s hard to find a proper reason for why this is allowed. If you are always allowed to feed in exactly 1 internal character, and it is always expected to form a complete “N” external/output characters, then insofar as basic_filebuf is concerned it’s allowed to completely mangle your data. This means that any encoding where it does not produce 1:N data (for example, produces 2:N or really anything with M:N where M ≠ 1) is completely screwed. Were you writing to a file? Well, good luck with that. Network share? Ain’t that a shame! Manage to open a file descriptor for a pipe and it’s wrapped in a basic_filebuf? Sucks to be you! Everything good about the C++ APIs gets flushed right down the toilet, all because they wanted to support the — ostensibly weird-as-hell — requirement that you can read or write things exactly 1 character at a time. Wouldn’t you be surprised that some implementation, somewhere, used exactly one character internally as a buffer? And, if we were required to ask for more than that, it would be an ABI break to fix it? (If this is a surprise to you, you should go back and read this specific section in this post about ABI breaks and how they ruin all of us.)

Of course, they are not really supporting anything, because in order to avoid having to cache the return value from in or out on any derived std::codecvt-derived class, it just means you can be fed a completely bogus stream and it’s just considered… okay. That’s it. Nothing you or anyone else can do about such a situation: you get nothing but suffering on your plate for this one.

An increasingly nonsensical part of how this specification works is that there’s no real way for the std::codecvt class to know that it’s being called up-stream by a std::basic_filebuf, so either every derived std::codecvt object has to be okay with artificial truncation, or the developers of the std::basic_filebuf have to simply throw away error codes they are not interested in and ignore any incomplete / broken sequences. It seems like most standard libraries choose the latter, which results in, effectively, all encoding procedures for all files to be broken in the same way wchar_t is broken in C, but for literally every encoding type if they cannot figure out how to get their derived class or their using class and then figure out if they’re inside a basic_filebuf or not.

Even more bizarrely, because of the failures of the specification, std::codecvt_utf16/std::codecvt_utf8 are, in general, meant to handle UCS-2 and not UTF-16⁴. (UCS-2 is the Unicode Specification v1.0 “wide character” set that handles 65535 code points maximum, which Unicode has already surpassed quite some time ago.) Nevertheless, most (all?) implementations seem to defy the standard, further making the class’s stated purpose in code a lot more worthless. There are also additional quirks that trigger undefined behavior when using this type for text-based or binary-based file writing. For example, under the deprecated <codecvt> description for codecvt_utf16, the standard says in a bullet point

The multibyte sequences may be written only as a binary file. Attempting to write to a text file produces undefined behavior.

Which, I mean. … What? Seriously? My encoding things do not work well with my text file writing, the one place it’s supposed to be super useful in? Come on! At this point, there is simply an enduring horror that leads to a bleak — if not fully disturbed — fascination about the whole situation.

Fumbling the Bag

If it was not for all of these truly whack-a-doodle requirements, we would likely have no problems. But it’s too late: any API that uses virtual functions are calcified for eternity. Their interfaces and guarantees can never be changed, because changing and their dependents means breaking very strong binary guarantees made about usage and expectations. I was truly excited to see std::codecvt’s interface surpassed its menial std::wstring_convert counterpart in ways that actually made it a genuinely forward-thinking API. But. It ultimately ends up going in the trash like every other Standard API out there. So close,

yet so far!

The rest of the API is the usual lack of thought put into an API to optimize for speed cases. No way to pass nullptr as a marker to the to/from_end pointers to say “I genuinely don’t care, write like the wind”, though on certain standard library implementations you could probably just get away with it⁵. There’s also no way to just pass in nullptr for the entire to_* sets of pointers to say “I just want you to give me the count back”; and indeed, there’s no way to compute such a count with the triple-input-pointer, triple-output-pointer API. This is why the libiconv-style of pointer-to-pointer, pointer-to-size API ends up superior: it’s able to capture all use cases without presenting problematic internal API choices or external user use choices (even if libiconv itself does not live up to its API’s potential).

This is, ostensibly, a part of why the std::wstring_convert performance and class of APIs suck as well. They ultimately cannot perform a pre-count and then perform a reservation, after doing a basic from_next - from check to see if the input is large enough to justify doing a .reserve(…)/.resize() call before looping and push_back/insert-ing into the target string using the .in and .out APIs on std::codecvt. You just have to make an over-estimate on the size and pre-reserve, or don’t do that and just serialize into a temporary buffer before dumping into the output. This is the implementation choice that e.g. MSVC makes, doing some ~16 characters at a time before vomiting them into the target string in a loop until std::codecvt::in/out exhausts all the input. You can imagine that encoding up to 16 characters at-most in a loop for a string several megabytes long is going to be an enormous no-no for many performance use cases, so that tends to get in the way a little bit.

There is, of course, one other bit about the whole C++ API that once again comes to rear its ugly head in our faces.

Old Habits Die Hard

There is also another significant problem with the usage of std::codecvt for its job; it relies on a literal locale object / locale facet to get its job done. Spinning up a std::codecvt can be expensive due to its interaction with std::locale and the necessity of being attached to a locale. It is likely intended that these classes can be used standalone, without attaching it to the locale at all (as their destructors, unlike other locale-based facets) were made public and callable rather than hidden/private. This means they can be declared on the stack and used directly, at least.

This was noticeably bad, back when I was still using std::codecvt and std::wstring_convert myself in sol2. Creating a fresh object to do a conversion resulted in horrible performance characteristics for that convert-to-UTF-8 routine relying on standard library facilities. These days, I have been doing a hand-written, utterly naïve UTF-8 conversions, which has stupidly better performance characteristics simply because it’s not dragging along whatever baggage comes with locales, facets, wstring_convert, codecvt, and all of its ilk. Which is just so deeply and bewilderingly frustrating that I can get a thumbs up from users by just doing the most head-empty, braindead thing imaginable and its just so much better than the default actions that come with the standard library.

Constantly, we are annoyed in the Committee or entirely dismissive of game development programmers (and I am, too!) of many of their concerns. But it is ENTIRELY plausible to see how they can start writing off entire standard libraries when over and over again you can just do the world’s dumbest implementation of something and it kicks the standard library’s ass for inputs small and large. This does not extrapolate to other areas, but it only takes a handful of bad experiences — especially back 10 or 20 years ago when library implementations were so much worse — to convince someone not to waste their time investigating and benchmarking when it is so much easier on the time-financials tradeoff to just assume it is godawful trash and write something quick ‘n’ dirty that was going to perform better anyways.

What a time to be alive trying to ask people to use Standard C and C++, when they can throw a junior developer at a problem at get better performance and compilation times to do a very normal and standard thing like convert to UTF-8.

I certainly don’t endorse the attitude of taking 20 year old perceptions and applying them to vastly improved modern infrastructure that has demonstrably changed, but it doesn’t take a rocket scientist to see how we ended up on this particular horizon of understanding.

But, That Done and Dusts That

C and C++ are now Officially Critiqued™ and hopefully I don’t have to have anyone crawl out of the woodwork to talk about X or Y thing again and how I’m not being fair enough by just calling it outright garbage. Now, all of us should thoroughly understand why it’s garbage and how unusable it is.

Nevertheless, if these APIs are garbage, how do we build our own good one? Clearly, if I have all of this evidence and all of these opinions, assuredly I’ve been able to make a better API? So, let’s try to dig in on that. I already figured out the C++ API in ztd.text and written about it extensively, so let’s cook up ztd.cuneicode (or just cuneicode), from the ground up, with a good interface.

A Wonderful API

For a C function, we need to have 4 capabilities, as outlined by the table above.

• Single conversions, to transcode one indivisible unit of information at a time.
• Bulk conversions, to transcode a large buffer as fast as possible (a speed optimization over single conversion with the same properties).
• Validation, to check whether an input is valid and can be converted to the output encoding (or where there an error would occur in the input if some part is invalid).
• Counting, to know how much output is needed (often with an indication of where an error would occur in the input, if the full input can’t be counted successfully).

We also know from the ztd.text blog post and design documentation, as well as the analysis from the previous blog post and the above table, that we need to provide specific information for the given capabilities:

• how much input was consumed (even in the case of an error);
• how much output was written (even in the case of an error);
• that “input read” should only include how much was successfully read (e.g., stops before the error happens and should not be a partial read);
• that “output written” should only include how much was successfully converted (e.g., points to just after the last successful serialization, and not any partial writes); and,
• that we may need additional state associated with a given encoding to handle it properly (any specific “shift sequences” or held-onto state; we’ll talk about this more thoroughly when demonstrating the new API).

It turns out that there is already one C API that does most of what we want design-wise, even if its potential was not realized by the people who worked on it and standardized its interface in POSIX!

Borrowing Perfection

This library has the perfect interface design and principles with — as is standard with most C APIs — the worst actual execution on said design. To review, let’s take a look at the libiconv conversion interface:

size_t iconv(
	iconv_t cd, // any necessary custom information and state
	char ** inbuf, // an input buffer and how far we have progressed
	size_t * inbytesleft, // the size of the input buffer and how much is left
	char ** outbuf, // an output buffer and how far we have progressed
	size_t * outbytesleft); // the size of the output buffer and how much is left

As stated in Part 1, while the libiconv library itself will fail to utilize the interface for the purposes we will list just below, we ourselves can adapt it to do these kinds of operations:

• normal output writing (iconv(cd, inbuf, inbytesleft, outbuf, outbytesleft));
• unbounded output writing (iconv(cd, inbuf, inbytesleft, outbuf, nullptr));
• output size counting (iconv(cd, inbuf, inbytesleft, nullptr, outbytesleft)); and,
• input validation (iconv(cd, inbuf, inbytesleft, nullptr, nullptr)).

Unfortunately, what is missing most from this API is the “single” conversion handling. But, you can always create a bulk conversion by wrapping a one-off conversion, or create a one-off conversion by wrapping a bulk conversion (with severe performance implications either way). We’ll add that to the list of things to include when we hijack and re-do this API.

So, at least for a C-style API, we need 2 separate class of functions for one-off and bulk-conversion. In Standard C, they did this by having mbrtowc (without an s to signify the one-at-a-time conversion nature) and by having mbsrtowcs (with an s to signify a whole-string conversion). Finally, the last missing piece here is an “assume valid” conversion. We can achieve this by providing a flag or a boolean on the “state”; in the case of iconv_t cd, it would be done at the time when the iconv_t cd object is generated. For the Standard C APIs, it could be baked into the mbstate_t type (though they would likely never, because adding that might change how big the struct is, and thus destroy ABI).

With all of this in mind, we can start a conversion effort for all of the fixed conversions. When I say “fixed”, I mean conversions from a specific encoding to another, known before we compile. These will be meant to replace the C conversions of the same style such as mbrtowc or c8rtomb, and fill in the places they never did (including e.g. single vs. bulk conversions). Some of these known encodings will still be linked to runtime based encodings that are based on the locale. But, rather than using them and needing to praying to the heaven’s the internal Multibyte C Encoding is UTF-8 (like with the aforementioned wcrtomb -> mbrtoc8/16/32 style of conversions), we’ll just provide a direction conversion routine and cut out the wchar_t encoding/multibyte encoding middle man.

Static Conversion Functions for C

The function names here are going to be kind of gross, but they will be “idiomatic” standard C. We will be using the same established prefixes from the C Standard group of functions, with some slight modifications to the mb and wc ones to allow for sufficient differentiation from the existing standard ones. Plus, we will be adding a “namespace” (in C, that means just adding a prefix) of cnc_ (for “cuneicode”), as well as adding the letter n to indicate that these are explicitly the “sized” functions (much like strncpy and friends) and that we are not dealing with null terminators at all in this API. Thusly, we end up with functions that look like this:

// Single conversion
cnc_XnrtoYn(size_t* p_destination_buffer_len,
	CharY** p_maybe_destination_buffer,
	size_t* p_source_buffer_len,
	CharX** p_source_buffer,
	cnc_mcstate_t* p_state);

// Bulk conversion
cnc_XsnrtoYsn(size_t* p_destination_buffer_len,
	CharY** p_maybe_destination_buffer,
	size_t* p_source_buffer_len,
	CharX** p_source_buffer,
	cnc_mcstate_t* p_state);

As shown, the s indicates that we are processing as many elements as possible (historically, s would stand for string here). The tags that replace X and Y in the function names, and their associated CharX and CharY types, are:

The optional encoding suffix is for the left-hand-side (from, X) encoding first, before the right-hand side (to, Y) encoding. If the encoding is the default associated encoding, then it can be left off. If it may be ambiguous which tag is referring to which optional encoding suffix, both encoding suffixes are provided. The reason we do not use mb or wc (like pre-existing C functions) is because those prefixes are tainted forever by API and ABI constraints in the C standard to refer to “bullshit multibyte encoding limited by a maximum output of MB_MAX_LEN”, and “can only ever output 1 value and is insufficient even if it is picked to be UTF-32”, respectively. The new name “mc” stands for “multi character”, and “mwc” stands for — you guessed it — “multi wide character”, to make it explicitly clear there’s multiple values that will be going into and coming out of these functions.

This means that if we want to convert from UTF-8 to UTF-16, bulk, the function to call is cnc_c8snrtoc16sn(…). Similarly, converting from the Wide Execution Encoding to UTF-32 (non-bulk) would be cnc_mwcnrtoc32n(…). There is, however, a caveat: at times, you may not be able to differentiate solely based on the encodings present, rather than the character type. In those cases, particularly for legacy encodings, the naming scheme is extended by adding an additional suffix directly specifying the encoding of one or both of the ends of the conversion. For example, a function that deliberate encodings from Punycode (RFC) to UTF-32 (non-bulk) would be spelled cnc_mcnrtoc32n_punycode(…) and use char for CharX and char32_t for CharY. A function to convert specifically from SHIFT-JIS to EUC-JP (in bulk) would be spelled cnc_mcsnrtomcsn_shift_jis_euc_jp(…) and use char for both CharX and CharY. Furthermore, since people like to use char for UTF-8 despite associated troubles with char’s signedness, a function converting from UTF-8 to UTF-16 in this style would be cnc_mcsnrtoc16sn_utf8(…). The function that converts the execution encoding to char-based UTF-8 is cnc_mcsnrtomcsn_exec_utf8(…).

The names are definitely a mouthful, but it covers all of the names we could need for any given encoding pair for the functions that are named at compile-time and do not go through a system similar to libiconv. Given this naming scheme, we can stamp out all the core functions between the 5 core encodings present on C and C++-based, locale-heavy systems (UTF-8, UTF-16, UTF-32, Execution Encoding, and Wide Execution Encoding), and have room for additional functions using specific names.

Finally, there is the matter of “conversions where the input is assumed good and valid”. In ztd.text, you get this from using the ztd::text::assume_valid_handler error handler object and its associated type. Because we do not have templates and we cannot provide type-based, compile-time polymorphism without literally writing a completely new function, cnc_mcstate_t has a function that will set its “assume valid” state. The proper = {} init of cnc_mcstate_t will keep it off as normal. But you can set it explicitly using the function, which helps us cover the “input is valid” bit.

Given all of this, we can demonstrate a small usage of the API here:

#include <ztd/cuneicode.h>

#include <ztd/idk/size.h>

#include <stdio.h>
#include <stdbool.h>
#include <string.h>

int main() {
	const char32_t input_data[] = U"Bark Bark Bark 🐕‍🦺!";
	char output_data[ztd_c_array_size(input_data) * 4] = {};
	cnc_mcstate_t state                                = {};
	// set the "do UB shit if invalid" bit to true
	cnc_mcstate_set_assume_valid(&state, true);
	const size_t starting_input_size  = ztd_c_string_array_size(input_data);
	size_t input_size                 = starting_input_size;
	const char32_t* input             = input_data;
	const size_t starting_output_size = ztd_c_array_size(output_data);
	size_t output_size                = starting_output_size;
	char* output                      = output_data;
	cnc_mcerror err                   = cnc_c32snrtomcsn_utf8(
	                       &output_size, &output, &input_size, &input, &state);
	const bool has_err          = err != cnc_mcerr_ok;
	const size_t input_read     = starting_input_size - input_size;
	const size_t output_written = starting_output_size - output_size;
	const char* const conversion_result_title_str = (has_err
		? "Conversion failed... \xF0\x9F\x98\xAD" // UTF-8 bytes for 😭
		: "Conversion succeeded \xF0\x9F\x8E\x89"); // UTF-8 bytes for 🎉
	const size_t conversion_result_title_str_size
		= strlen(conversion_result_title_str);
	// Use fwrite to prevent conversions / locale-sensitive-probing from
	// fprintf family of functions
	fwrite(conversion_result_title_str, sizeof(*conversion_result_title_str),
		conversion_result_title_str_size, has_err ? stderr : stdout);
	fprintf(has_err ? stderr : stdout,
		"\n\tRead: %zu %zu-bit elements"
		"\n\tWrote: %zu %zu-bit elements\n",
		(size_t)(input_read), (size_t)(sizeof(*input) * CHAR_BIT),
		(size_t)(output_written), (size_t)(sizeof(*output) * CHAR_BIT));
	fprintf(stdout, "%s Conversion Result:\n", has_err ? "Partial" : "Complete");
	fwrite(output_data, sizeof(*output_data), output_written, stdout);
	// The stream is (possibly) line-buffered, so make sure an extra "\n" is written
	// out; this is actually critical for some forms of stdout/stderr mirrors. They
	// won't show the last line even if you manually call fflush(…) !
	fwrite("\n", sizeof(char), 1, stdout);
	return has_err ? 1 : 0;
}

Which (on a terminal that hasn’t lost its mind⁶) produces the following output:

Conversion succeeded 🎉:
	Read: 19 32-bit elements
	Wrote: 27 8-bit elements
Complete Conversion Result:
Bark Bark Bark 🐕‍🦺! 

Of course, some readers may have a question about how the example is written. Two things, in particular…

fwrite? Huh??

The reason we always write out Unicode data using fwrite rather than fprintf/printf or similar is because on Microsoft Windows, the default assumption of input strings is that they have a locale encoding. In order to have that data reach specific kinds of terminals, certain terminal implementations on Windows will attempt to convert from what they suspect the encoding of the application’s strings are (e.g., from %s/%*.s) to the encoding of the terminal. In almost all cases, this assumption is wrong when you have a capable terminal (such as the new Windows Terminal, or the dozens of terminal emulators that run on Windows). The C Standard for fprintf and its %s modifier specifies no conversions, but it does not explicitly forbid them from doing this, either. They are also under no obligation to properly identify what the input encoding that goes into the terminal is either.

For example, even if I put UTF-8 data into fprintf("%s", (const char*)u8"🐈 meow 🐱");, it can assume that the data I put in is not UTF-8 but, in fact, ISO 8859-1 or Mac Cyrillic or GBK. This is, of course, flagrantly wrong for our given example. But, it doesn’t matter: it will misinterpret that data as one kind of encoding and blindly encode it to whatever the internal Terminal encoding is (which is probably UTF-16 or some adjacent flavor thereof).

The result is that you will get a bunch of weird symbols or a bunch of empty cells in your terminal, leading to confused users and no Unicode-capable output. So, the cross-platform solution is to use fwrite specifically for data that we expect implementations like Microsoft will mangle on various terminal displays (such as in VSCode, Microsoft Terminal, or just plain ol’ cmd.exe that is updated enough and under the right settings). This bypasses any internal %s decoding that happens, and basically shoves the bytes as-is straight to the terminal. Given it is just a sequence of bytes going to the terminal, it will be decoded directly by the display functions of the terminal and the shown cells, at least for the new Windows Terminal, will show us UTF-8 output.

It’s not ideal and it makes the examples a lot less readable and tangible, but that is (unfortunately) how it is.

What is with the "foo" string literal but the e.g. \xF0\x9F\x98\xAD sequence??

Let us take yet another look at this very frustrating initialization:

	// …

	const char* const conversion_result_title_str = (has_err
		? "Conversion failed... \xF0\x9F\x98\xAD" // UTF-8 bytes for 😭
		: "Conversion succeeded \xF0\x9F\x8E\x89"); // UTF-8 bytes for 🎉

	// …

You might look at this and be confused. And, rightly, you should be: why not just use a u8"" string literal? And, with that u8"" literal, why not just use e.g. u8"Blah blah\U0001F62D" to get the crying face? Well, unfortunately, I regret to inform you that

MSVC is At It Again!

Let’s start with just using u8"" and trying to put the crying face into the string literal, directly:

	// …

	const char very_normal[] = u8"😭";

	// …

This seems like the normal way of doing things. Compile on GCC? Works fine. Compile on Clang? Also works fine enough. Compile on MSVC? Well, I hope you brought tissues. If you forget to use the /utf-8 flag, this breaks in fantastic ways. First, it will translate your crying emoji into a sequence of code units that is mangled, at the source level. Then, as the compiler goes forward, a bunch of really fucked up bytes that no longer correspond to the sobbing face emoji (Unicode code point U+0001F62D) will then each individually be treated as its own Unicode code point. So you will get 4 code points, each one more messed up that the last, but it doesn’t stop there, because MSVC — in particular — has a wild behavior. The size of the string literal here won’t be 4 (number of mangled bytes) + 1 (null terminator) to have sizeof(very_normal) be 5. No, the sizeof(very_normal) here is NINE (9)!

See, Microsoft did this funny thing where, inside of the u8"", each byte is not considered as part of a sequence. Each byte is considered its own Unicode code point, all by itself. So the 4 fucked up bytes (plus null terminator) are each treated as distinct, individual code points (and not a sequence of code units). Each of these is then expanded out to their UTF-8 representation, one at a time. Since the high bit is set on all of these, each “code point” effectively translates to a double-byte UTF-8 sequence. Now, normally, that’s like… whatever, right? We didn’t specify /utf-8, so we’re getting garbage into our string literal at some super early stage in the lexing of the source. “Aha!”, you say. “I will inject each byte, one at a time, using a \xHH sequence, where HH is a 0-255 hexadecimal character.” And you would be right on Clang. And right on GCC. And right according to the C and C++ standard. You would even be correct if it was an L"" string literal, where it would insert one code unit corresponding to that sequence. But if you type this:

	// …

	const char very_normal[] = u8"\xF0\x9F\x98\xAD";

	// …

You would not be correct on MSVC.

The above code snippet is what folks typically reach for, when they cannot guarantee /utf-8 or /source-charset=.65001 (the Microsoft Codepage name for UTF-8). “If I can just inject the bytes, I can get a u8"" string literal, typed according with unsigned values, converted into my const char[] array.” This makes sense. This is what people do, to have cross-platform source code that will work everywhere, including places that were slow to adopt \U... sequences. It’s a precise storage of the bytes, and each \x fits for each character.

But it won’t work on MSVC.

The sizeof(very_normal) here, even with /utf-8 specified, is still 9. This is because, as the previous example shows up, it treats each code unit here as being a full code point. These are all high-bits-set values, and thus are treated as 2-byte UTF-8 sequences, plus the null terminator. No other compiler does this. MSVC does not have this behavior even for its other string literal types; it’s just with UTF-8 they pull this. So even if you can’t have u8"😭" in your source code — and you try to place it into the string in an entirely agnostic way that gets around bad source encoding — it will still punch you in the face. This is not standards-conforming. It was never standards-conforming, but to be doubly sure the wording around this in both C and C++ was clarified in recent additions to make it extremely clear this is not the right behavior.

There are open bug reports against MSVC for this. There were also bug reports against the implementation before they nuked their bug tracker and swapped to the current Visual Studio Voice. I was there, when I was not involved in the standard and code-young to get what was going on. Back when the libogonek author and others tried to file against MSVC about this behavior. Back when the “CTP”s were still a thing MSVC was doing.

They won’t fix this. The standard means nothing here; they just don’t give a shit. Could be because of source compatibility reasons. But even if it’s a source compatibility issue, they won’t even lock a standards conforming behavior behind a flag. /permissive- won’t fix it. /utf-8 won’t fix it. There’s no /Zc:unfuck-my-u8-literals-please flag. Nothing. The behavior will remain. It will screw up every piece of testing code you have written to test for specific patterns in your strings while you’re trying to make sure the types are correct. There is nothing you can do but resign yourself to not using u8"" anymore for those use cases.

Removing the u8 in front gets the desired result. Using const char very_normal[] = u8"\U0001F62D"; also works, except that only applies if you’re writing UTF-8 exactly. If you’re trying to set up e.g. an MUTF-8 null terminator (to work with Android/Java UTF-8 strings) inside of your string literal to do a quick operation? If you want to insert some “false” bytes in your test suite’s strings to check if your function works? …

Hah.

Stack that with the recent char8_t type changes for u8"", and it’s a regular dumpster fire on most platforms to program around for the latest C++ version.

Nevertheless!

This manages to cover the canonical conversions between most of the known encodings that come out of the C standard:

Anything else has the special suffix added, but ultimately it is not incredibly satisfactory. After all, part of the wonderful magic of ztd.text and libogonek is the fact that — at compile-time — they could connect 2 encodings together. Now, there’s likely not a way to fully connect 2 encodings at compile-time in C without some of the most disgusting macros I would ever write being spawned from the deepest pits of hell. And, I would likely need an extension or two, like Blocks or Statement Expressions, to make it work out nicely so that it could be used everywhere a normal function call/expression is expected.

Nevertheless, not all is lost. I promised an interface that could automatically connect 2 disparate encodings, similar to how ztd::text::transcode(…) can give to the ability to convert between a freshly-created Shift-JIS and UTF-8 without writing that specific conversion routine yourself. This is critical functionality, because it is the step beyond what Rust libraries like encoding_rs offer, and outstrips what libraries like simdutf, utf8cpp, or Standard C could ever offer. If we do it right, it can even outstrip libiconv, where there is a fixed set of encodings defined by the owner of the libiconv implementation that cannot be extended without recompiling the library and building in new routines. ICU includes functionality to connect two disparate encodings, but the library must be modified/recompiled to include new encoding routines, even if they have the ucnv_ConvertEx function that takes any 2 disparate encodings and transcodes through UChars (UTF-16). Part of the promise of this article was that we could not only achieve maximum speed, but allow for an infinity of conversions within C.

So let’s build the C version of all of this.

General-Purpose Interconnected Conversions Require Genericity

The collection of the cuneicode functions above are both strongly-typed and the encoding is known. In most cases (save for the internal execution and wide execution encodings, where things may be a bit ambiguous to an end-user (but not for the standard library vendor)), there is no need for any intermediate conversion steps. They do not need any potential intermediate storage because both ends of the transcoding operation are known. libiconv provides us with a good idea for what the input and output needs to look like, but having a generic pivot is a different matter. ICU and a few other libraries have an explicit pivot source; other libraries (like encoding_rs) want you to coordinate the conversion from the disparate encoding to UTF-8 or UTF-16 and then to the destination encoding yourself (and therefore provide your own UTF-8/16 pivot). Here’s how ICU does it in its ucnv_convertEx API:

U_CAPI void ucnv_convertEx(
	UConverter *targetCnv, UConverter *sourceCnv, // converters describing the encodings
	char **target, const char *targetLimit, // destination
	const char **source, const char *sourceLimit, // source data
	UChar *pivotStart, UChar **pivotSource, UChar **pivotTarget, const UChar *pivotLimit, // ❗❗ the pivot ❗❗
	UBool reset, UBool flush, UErrorCode *pErrorCode); // error code out-parameter

The buffers have to be type-erased, which means either providing void*, aliasing-capable⁷ char*, or aliasing-capable unsigned char*. (Aliasing is when a pointer to one type is used to look at the data of a fundamentally different type; only char and unsigned char can do that, and std::byte if C++ is on the table.) After we type-erase the buffers so that we can work on a “byte” level, we then need to develop what ICU calls UConverters. Converters effectively handle converting between their desired representation (e.g., SHIFT-JIS or EUC-KR) and transport to a given neutral middle-ground encoding (such as UTF-32, UTF-16, or UTF-8). In the case of ICU, they convert to UChar objects, which are at-least 16-bit sized objects which can hold UTF-16 code units for UTF-16 encoded data. This becomes the Unicode-based anchor through which all communication happens, and why it is named the “pivot”.

Pivoting: Getting from A to B, through C

ICU is not the first library to come up with this. Featured in libiconv, libogonek, my own libraries, encoding_rs (in the examples, but not the API itself), and more, libraries have been using this “pivoting” technique for coming up on a decade and a half now. It is effectively the same platonic ideal of “so long as there is a common, universal encoding that can handle the input data, we will make sure there is an encoding route from A to this ideal intermediate, and then go to B through said intermediate”. Let’s take a look at ucnv_convertEx from ICU again:

U_CAPI void ucnv_convertEx (UConverter *targetCnv, UConverter *sourceCnv,
	char **target, const char *targetLimit,
	const char **source, const char *sourceLimit,
	UChar *pivotStart, UChar **pivotSource, UChar **pivotTarget, // pivot / indirect
	const UChar *pivotLimit,
	UBool reset, UBool flush, UErrorCode *pErrorCode);

The pivot is typed as various levels of UChar pointers, where UChar is a stand-in for a type wide enough to hold 16 bits (like uint_least16_t). More specifically, the UChar-based pivot buffer is meant to be the place where UTF-16 intermediate data is stored when there is no direct conversion between two encodings. The iconv library has the same idea, except it does not expose the pivot buffer to you. Emphasis mine:

It provides support for the encodings…

… [huuuuge list] …

It can convert from any of these encodings to any other, through Unicode conversion.

— GNU version of libiconv, May 21st, 2023

In fact, depending on what library you use, you can be dealing with a “pivot”, “substrate”, or “go-between” encoding that usually ends up being one of UTF-8, UTF-16, or UTF-32. Occasionally, non-Unicode pivots can be used as well but they are exceedingly rare as they often do not accommodate characters from both sides of the equation, in a way that Unicode does (or gives room to). Still, just because somebody writes a few decades-old libraries and frameworks around it, doesn’t necessarily prove that pivots are the most useful technique. So, are pivots actually useful?

When I wrote my previous article about generic conversions, we used the concept of a UTF-32-based pivot to convert between UTF-8 and Shift-JIS, without either encoding routine knowing about the other. Of course, because this is C and we do not have ugly templates, we cannot use the compile-time checking to make sure the decode_one of one “Lucky 7” object and the encode_one of the other “Lucky 7” object lines up. So, we instead need a system where encodings pairs identify themselves in some way, and then identify that as the pivot point. That is, for this diagram:

And make the slight modification that allows for this:

The “something” is our indirect encoding, and it will also be used as the pivot. Of course, we still can’t know what that pivot will be, so we will once again use a type-erased bucket of information for that. Ultimately, our final API for doing this will look like this:

#include <stddef.h>

typedef enum cnc_mcerror {
	cnc_mcerr_ok = 0,
	cnc_mcerr_invalid_sequence = 1,
	cnc_mcerr_incomplete_input = 2,
	cnc_mcerr_insufficient_output = 3
} cnc_mcerror;

struct cnc_conversion;
typedef struct cnc_conversion cnc_conversion;

typedef struct cnc_pivot_info {
	size_t bytes_size;
	unsigned char* bytes;
	cnc_mcerror error;
} cnc_pivot_info;

cnc_mcerror cnc_conv_one_pivot(cnc_conversion* conversion,
	size_t* p_output_bytes_size, unsigned char** p_output_bytes,
	size_t* p_input_bytes_size, const unsigned char** p_input_bytes,
	cnc_pivot_info* p_pivot_info);

cnc_mcerror cnc_conv_pivot(cnc_conversion* conversion,
	size_t* p_output_bytes_size, unsigned char** p_output_bytes,
	size_t* p_input_bytes_size, const unsigned char** p_input_bytes,
	cnc_pivot_info* p_pivot_info);

The _one suffixed function does one-by-one conversions, and the other is for bulk conversions. We can see that the API shape here looks pretty much exactly like libiconv, with the extra addition of the cnc_pivot_info structure for the ability to control how much space is dedicated to the pivot. If p_pivot_info->bytes is a null pointer, or p_pivot_info is, itself, a null pointer, then it will just use some implementation-defined, internal buffer for a pivot. From this single function, we can spawn the entire batch of functionality we initially yearned for in libiconv. But, rather than force you to write nullptr/NULL in the exact-right spot of the cnc_conv_pivot function, we instead just provide you everything you need anyways:

// cnc_conv bulk variants
cnc_mcerror cnc_conv(cnc_conversion* conversion,
	size_t* p_output_bytes_size, unsigned char** p_output_bytes,
	size_t* p_input_bytes_size, const unsigned char** p_input_bytes);

cnc_mcerror cnc_conv_count_pivot(cnc_conversion* conversion,
	size_t* p_output_bytes_size,
	size_t* p_input_bytes_size, const unsigned char** p_input_bytes,
	cnc_pivot_info* p_pivot_info);
cnc_mcerror cnc_conv_count(cnc_conversion* conversion,
	size_t* p_output_bytes_size,
	size_t* p_input_bytes_size, const unsigned char** p_input_bytes);

bool cnc_conv_is_valid_pivot(cnc_conversion* conversion,
	size_t* p_input_bytes_size, const unsigned char** p_input_bytes,
	cnc_pivot_info* p_pivot_info);
bool cnc_conv_is_valid(cnc_conversion* conversion,
	size_t* p_input_bytes_size, const unsigned char** p_input_bytes);

cnc_mcerror cnc_conv_unbounded_pivot( cnc_conversion* conversion,
	unsigned char** p_output_bytes,
	size_t* p_input_bytes_size, const unsigned char** p_input_bytes,
	cnc_pivot_info* p_pivot_info);
cnc_mcerror cnc_conv_unbounded(cnc_conversion* conversion,
	unsigned char** p_output_bytes,
	size_t* p_input_bytes_size, const unsigned char** p_input_bytes);

// cnc_conv_one single variants
cnc_mcerror cnc_conv_one(cnc_conversion* conversion,
	size_t* p_output_bytes_size, unsigned char** p_output_bytes,
	size_t* p_input_bytes_size, const unsigned char** p_input_bytes);

cnc_mcerror cnc_conv_one_count_pivot(cnc_conversion* conversion,
	size_t* p_output_bytes_size,
	size_t* p_input_bytes_size, const unsigned char** p_input_bytes,
	cnc_pivot_info* p_pivot_info);
cnc_mcerror cnc_conv_one_count(cnc_conversion* conversion,
	size_t* p_output_bytes_size,
	size_t* p_input_bytes_size, const unsigned char** p_input_bytes);

bool cnc_conv_one_is_valid_pivot(cnc_conversion* conversion,
	size_t* p_input_bytes_size, const unsigned char** p_input_bytes,
	cnc_pivot_info* p_pivot_info);
bool cnc_conv_one_is_valid(cnc_conversion* conversion,
	size_t* p_input_bytes_size, const unsigned char** p_input_bytes);

cnc_mcerror cnc_conv_one_unbounded_pivot(cnc_conversion* conversion,
	unsigned char** p_output_bytes, size_t* p_input_bytes_size,
	const unsigned char** p_input_bytes, cnc_pivot_info* p_pivot_info);
cnc_mcerror cnc_conv_one_unbounded(cnc_conversion* conversion,
	unsigned char** p_output_bytes,
	size_t* p_input_bytes_size, const unsigned char** p_input_bytes);

It’s a lot of declarations, but wouldn’t you be surprised that the internal implementation of almost all of these is just one function call!

bool cnc_conv_is_valid(cnc_conversion* conversion,
	size_t* p_bytes_in_count,
	const unsigned char** p_bytes_in)
{
	cnc_mcerror err = cnc_conv_pivot(conversion,
		NULL, NULL,
		p_bytes_in_count, p_bytes_in,
		NULL); 
	return err == cnc_mcerr_ok;
}

It is mostly for convenience to provide these functions. Since the implementation is so simple, it warrants giving people exactly what they want. This is so they can have named functions which communicate what they’re doing in as normal a way, as opposed to using NULL/nullptr splattering that does not communicate anything to an external user why exactly someone is doing that with the function. Still, for as much as I talk these functions up, there’s two very important bits I’ve been sort of skirting around:

• How the heck do we get a cnc_conversion* handle?
• How do we make sure we provide generic connection points between random encodings?

Well, strap in, because we are going to be crafting a reusable, general-purpose encoding library that allows for run time extension of the available encodings (without loss of speed).

cuneicode and the Encoding Registry

As detailed in Part 1 and hinted at above, libiconv — and many other existing encoding infrastructures — do not provide a way to expand their encoding knowledge at run time. They ship with a fixed set of encodings, and you must either directly modify the library or directly edit data files in order to coax more encodings out of the interface. In the case of Standard C, sometimes that means injecting more files into the system locale files, or other brittle/non-portable things. We need a means of loading up and controlling a central place where we can stuff all our encodings. Not only that, but we also:

• need to allow for controlling all allocations made; and,
• need to allow for loading up an encoding registry with any “defaults” the library may have ready for us.

So, we need to come up with a new entity for the API that we will term a cnc_conversion_registry. As the name implies, it’s a place to store all of our conversions, and thusly a brief outline of the most important parts of the API should look like this:

typedef void*(cnc_allocate_function)(size_t requested_size, size_t alignment,
	size_t* p_actual_size, void* user_data);
typedef void*(cnc_reallocate_function)(void* original, size_t requested_size,
	size_t alignment, size_t* p_actual_size, void* user_data);
typedef void*(cnc_allocation_expand_function)(void* original, size_t original_size,
	size_t alignment, size_t expand_left, size_t expand_right, size_t* p_actual_size,
	void* user_data);
typedef void*(cnc_allocation_shrink_function)(void* original, size_t original_size,
	size_t alignment, size_t reduce_left, size_t reduce_right, size_t* p_actual_size,
	void* user_data);
typedef void(cnc_deallocate_function)(
	void* ptr, size_t ptr_size, size_t alignment, void* user_data);

typedef struct cnc_conversion_heap {
	void* user_data;
	cnc_allocate_function* allocate;
	cnc_reallocate_function* reallocate;
	cnc_allocation_expand_function* shrink;
	cnc_allocation_shrink_function* expand;
	cnc_deallocate_function* deallocate;
} cnc_conversion_heap;

cnc_conversion_heap cnc_create_default_heap(void);

typedef enum cnc_open_error {
	cnc_open_err_ok = 0,
	cnc_open_err_no_conversion_path = -1,
	cnc_open_err_insufficient_output = -2,
	cnc_open_err_invalid_parameter = -3,
	cnc_open_err_allocation_failure = -4
} cnc_open_error;

typedef enum cnc_registry_options {
	cnc_registry_options_none = 0,
	cnc_registry_options_empty = 1,
	cnc_registry_options_default = cnc_registry_options_none,
} cnc_registry_options;

struct cnc_conversion_registry;
typedef struct cnc_conversion_registry cnc_consversion_registry;

cnc_open_error cnc_open_registry(cnc_conversion_registry** p_out_registry, cnc_conversion_heap* p_heap,
	cnc_registry_options registry_options);
cnc_open_error cnc_new_registry(cnc_conversion_registry** p_out_registry,
	cnc_registry_options registry_options);