There are some things that I really enjoy in FP and that are more difficult to replicate in imperative languages that don’t enforce them.
One thins is the “everything is an expression” approach — as in, every operation produces a value of a known type. The unit type in OCaml and Rust may seem like noise at first but it saved me from many situations when I unintentionally tried to ignore outputs of functions.
Another annoying thing about the ability to ignore output to functions, is that libraries sometimes return useless error codes, and you don’t know for sure whether you should ignore it or not. Take for instance libsodium’s generic hash function:
int crypto_generichash(unsigned char *out, size_t outlen,
const unsigned char *in, unsigned long long inlen,
const unsigned char *key, size_t keylen);
Why is this function returning an int? Whyyyy? The documentation doesn’t even state it!! What am I supposed to do, check the value, and… do something if it fails? It’s a hash function for heaven’s sake, it is total, it works on every possible input!!
Wait.
Actually it doesn’t. This is C we’re talking about, if we give this bad boy a NULL pointer we’re fucked. Maybe? Maybe the function checks for NULL and kindly returns an error instead of crashing my program or corrupting my memory? I don’t know, the documentation is silent there. Sigh… I guess I’ll have to compensate for the documenter’s failure and use the source(as Linux kernel devs seem to require of user-space programmers whenever they’re using anything but basic system calls — such as Netlink sockets).
int
crypto_generichash(unsigned char *out, size_t outlen, const unsigned char *in,
unsigned long long inlen, const unsigned char *key,
size_t keylen)
{
return crypto_generichash_blake2b(out, outlen, in, inlen, key, keylen);
}
Okay, so if the key and hash lengths are too big, my program just crashes. In debug mode at least, I guess in release mode it’s just Undefined Behaviour™. Hey, but wait a minute, BLAKE2b hashes are at most 64 bytes long, not 255! Same for the key, though it could conceivably go up to 128 bytes (one BLAKE2b block) without bothering most implementations. So there’s a check, but it’s incomplete, and it still doesn’t affect the return value.
You know, I was almost hoping for a macro here. But nah, it’s just a function that crashes my program. A good thing, considering it this one will presumably still hold in production. Now the correct lengths are checked, which is somewhat redundant with the assert() from the calling function, but no big deal. And we also check for NULL, terrific! Note though that the NULL check is not very useful in practice: unless you’re in an embedded environment that tends to be too small for libsodium to begin with, the OS will cleanly crash your program anyway. Though here we do get rid of the potential Nasal Demons. Most importantly though, libsodium does not, can not, check for wrong non-null pointers, or buffers that are too short, errors that are arguably just as likely as using a NULL pointer. Though to their credit, this tend to be less true when it is embedded in a garbage collected language, where all pointers tend to be either NULL or safe.
Anyway, I have just wasted 10 minutes of my time investigating the possible values of a return code, only to find the only code is fucking zero. Oh well, at least it matches the amount of information I get out of it.
This is a problem for static analysis tools and conscientious programmers: each use of this function will either require a useless check with dead code no test can trigger, or will trigger a warning because I’m ignoring the return value. Or maybe there will be no warning, but then I worry about other functions with a return value I should not ignore. No matter how we cut it, this is not ideal.
And this is good code we’re talking about: libsodium isn’t just a good cryptographic library with excellent performance and good security track record, it’s also fairly good C code in general. And yet, there’s this basic error of giving an error code to a function, and then not using it.
Edit: though to be honest, I still disagree with the conflation of API misuse and actual failures: failures are expected and should totally go to an error code. For instance, checking a corrupt signature or trying to decrypt a forged message should return an error. For misuses however, I prefer asserts and panics, at least by default. Had libsodium adopted that model, its hash functions would return void, and properly denote that they cannot fail when used correctly.
This is a problem for static analysis tools and conscientious programmers: each use of this function will either require a useless check with dead code no test can trigger, or will trigger a warning because I’m ignoring the return value. Or maybe there will be no warning, but then I worry about other functions with a return value I should not ignore. No matter how we cut it, this is not ideal.
(void) blah;
And this is good code we’re talking about: libsodium isn’t just a good cryptographic library with excellent performance and good security track record, it’s also fairly good C code in general. And yet, there’s this basic error of giving an error code to a function, and then not using it.
I’m blaming C for this.
C has plenty of problems. A lot of problems, one might say. A goddamn monumental shitload of problems, even. But it does let you declare a function that returns nothing. C hasn’t forced anyone to return a meaningless int since 1989.
This is why higher-level types are useful and why I find C an unusable language these days. All of the dynamic checks in your example are things that could be represented in the type system in even a slightly higher-level language (including C++). Keys should be a specific type. If the output buffer is a fixed size then it could be a type of that size. In C++, I’d write that API as something like this:
using HashResult = std::array<uint8_t, HashLength>;
using HashKey = std::array<uint8_t, KeyLength>;
HashResult crypto_generichash(const std::ranges::range<uint8_t> input, const HashKey &key);
By construction, it’s now guaranteed to pass all of the dynamic checks in the C API. If you need to be able to handle dynamic sizes, then you can implement a function in the header that does the dynamic checks and returns the error result and then everything in your library can assume well-formed inputs.
In a functional language with dependent types, you can express even more useful things, such as ‘the output buffer for this authenticated construction must be the length of the input array plus the space for the integrity tags’. Doing this kind of thing in C++ is much more clunky but doing it in C is not possible.
Yeah I was disappointed when I saw the libsodium API was using untyped pointers all over the place. I had got the vague impression from reading discussions about it that it was supposed to be easier to use than typical cryptographic APIs – which to me implied better type safety to make it harder to misuse. But it seems “easier” meant a careful selection of primitives (good) with straightforward APIs (could be safer) but with cute names like box and secretbox which all look just the same (since the primitives are fixed I would prefer to name the API after the primitive so I can tell them apart!).
Blake2b is awkward for C because all its arrays are variable length (which I think means C++ std::array is the wrong type) but the various curve25519 and chacha ciphers ought to use types like
C99 allows a function declaration to specify the relationship between size arguments and array arguments but as far as I know it doesn’t help much for safety except perhaps with a good test suite using FORTIFY_SOURCE=3 and __builtin_dynamic_object_size.
I believe I vaguely considered something like this when I designed Monocypher, but ended up deciding against it, mostly in the name of idiomaticity. But realistically, how many errors a stricter API would actually prevent? C does not check bounds by default (or ever…), so what’s preventing me from accessing the array in the struct out of bound?
Well, turns out compilers are now fairly good at catching the most egregious errors, but then they’re equally good at catching out of bound accesses when we write something like void foo(uint8_t bar[32]): if the compiler catches me accessing more than 32 bytes inside the function I get a warning, and I think I also get a warning if I’m calling the function a buffer the compiler can see is too small — I’m not sure.
The most important argument for using bog standard uint8_t * though is interfacing with other languages. FFIs tend to be a major hassle whenever we deal with anything more complicated than basic types and pointers. Giving them the raw uint8_t * pointers instead makes them easier to write. You’re only safe once you’ve crossed the border, so we might as well make it easier to cross.
David’s right though, C is really really weak on this front. I can kinda compensate with a paranoid test suite and all sanitisers I can get my hands on, but a more expressive, safer type system would really go a long way. Just give me fat pointers and automatic bounds checking (warnings & errors at compile time, panics at runtime), and I will sleep much easier.
Turns out the function does return an error code after all. Though I confess I still don’t like the design, because as a user, I can guarantee I’m calling the function correctly, and when I do it is guaranteed not to fail. In my opinion a better outcome for failing those checks is to panic with a stack trace right then and there.
Languages that do not embrace “everything is an expression” make REPLs much more awkward and complicated, too, when they have them at all. Not having a simple way of trying out fragments of code piece-by-piece then produces additional knock-on complexity in tools for testing and debugging.
This is what I find so frustrating about Python. On top of having a weird split between statements and expressions, syntactic whitespace and botched lambdas make REPL-oriented development much harder. Some little changes could make it much nicer. Coming from Scheme or Julia makes it a serious downgrade in that regard.
Using dependent types one can represent state machines as follows:
StateMachine = record {
State : Type
Input : State -> Type
Output : (s : State) -> Input s -> Type
transition : (s : State) (i : Input s) -> Output s i -> State
}
As you can see the input and output types depend on the state, so one can disallow transitions (e.g. return only a valid subset of inputs in each state, or even the empty type if no inputs are allowed in some state).
(Dependent records are merely syntactic sugar for Sigma types, i.e. dependent product types, and (x : A) is a Pi type, i.e. a dependent function type.)
A slight variant of these state machines, where we move the State from being a field inside the record to being a parameter, give raise to predicate transformers:
Predicate A = A -> Type
[[_]] : StateMachine State -> (Predicate State -> Predicate State)
These predicate transformers form a monad where bind is the equivalent of the sequence rule (;) in Hoare logic, which means we can compose operations whose “protocol” is caputred by the state machine.
I have written a typestate program in rust but unfortunately it is unbelievably unwieldy if an operation transitions to one of several possible states.
isn’t it objectively better to get a finite and predictable error value from a function than an unspecified exception that may or may not happen that you still have to guard against?
Returning a specific error type is akin to the reviled checked exceptions in Java: clearer typing, yes, but you’re now on the hook for mapping any errors in functions you call — if your function signature says you only return a LogicError, then if you call something that returns IOError you have to somehow map that to a LogicError or broaden your return error type, which is of course viral. In practice I mostly see functions return some kind of very broad Error type (viz. Go, Objective-C.)
Exceptions “may or may not happen”, but the same is true of errors. Exceptions are like implicitly wrapping every call site with a zero-cost version of “if (error) return error”, the kind of boilerplate that fills Go code (and C code that bothers to handle errors.) Conversely, returned errors are like wrapping every throwing function call with a try/catch block that sets an error variable … whether or not there’s any need to catch the error.
In practice, only a small number of call sites need to do specific error handling (assuming you’ve got something like RAII, or at least GC) so having to wrap explicit checks around all of them adds a ton of noise. And only a small number of calls result in errors, so the overhead of checking each one is unfortunate.
I think what people are against is the implicit nature of exceptions — that a call might exit the function even though there’s nothing written there saying that. They have a point, and I really like the way newer languages require syntax like “try” on such a call. It isn’t necessary, it just serves as an annotation so the programmer knows this might occur.
What I’ve found is that in general you’re right, most call sites don’t need specific error handling, and some black box “general” error is perfectly fine (which is arguably covered by both a catch-all error type or exceptions). However, for the cases where you do want exhaustive error handling, and ensuring that you cover all the edge-cases, then being able to pattern match on all of the possible errors is super handy for preventing bugs.
I think Rust surprisingly gets it right, despite the error handling being rather complicated once you get into the nitty-gritty details. For applications, you can use anyhow for a catch-all error type, and for libraries you use explicit enums (generally using something like thiserror to help facilitate wrapping other errors). What this means is that for the application developer, you can pattern match where you need, but otherwise just bubble up effortlessly.
I think Zig gets it pretty close to right, with a potential error from a function being declared with an error union type and error set inference. The compiler will infer the smallest possible error set, which is suitable for ~95% of end-user cases. (Libraries and the stdlib often like to declare their error types explicitly, for clarity and to ensure they’re not propagating any errors they don’t intend to, and (mutually) recursive functions need at least one of them to declare their error set explicitly.)
The big omission is that there’s no payload for errors; there’s only the error tag value itself.
They have a point, and I really like the way newer languages require syntax like “try” on such a call. It isn’t necessary, it just serves as an annotation so the programmer knows this might occur.
It’s kind of necessary — without it, how do you handle the error (or hold onto the value, error or not) without propagating? One can think of other ways to do it, but you do need something differentiating propagate-vs-not.
There are some things that I really enjoy in FP and that are more difficult to replicate in imperative languages that don’t enforce them.
One thins is the “everything is an expression” approach — as in, every operation produces a value of a known type. The unit type in OCaml and Rust may seem like noise at first but it saved me from many situations when I unintentionally tried to ignore outputs of functions.
Another annoying thing about the ability to ignore output to functions, is that libraries sometimes return useless error codes, and you don’t know for sure whether you should ignore it or not. Take for instance libsodium’s generic hash function:
Why is this function returning an
int? Whyyyy? The documentation doesn’t even state it!! What am I supposed to do, check the value, and… do something if it fails? It’s a hash function for heaven’s sake, it is total, it works on every possible input!!Wait.
Actually it doesn’t. This is C we’re talking about, if we give this bad boy a NULL pointer we’re fucked. Maybe? Maybe the function checks for NULL and kindly returns an error instead of crashing my program or corrupting my memory? I don’t know, the documentation is silent there. Sigh… I guess I’ll have to compensate for the documenter’s failure and use the source (as Linux kernel devs seem to require of user-space programmers whenever they’re using anything but basic system calls — such as Netlink sockets).
Dammit.
Okay, so if the key and hash lengths are too big, my program just crashes. In debug mode at least, I guess in release mode it’s just Undefined Behaviour™. Hey, but wait a minute, BLAKE2b hashes are at most 64 bytes long, not 255! Same for the key, though it could conceivably go up to 128 bytes (one BLAKE2b block) without bothering most implementations. So there’s a check, but it’s incomplete, and it still doesn’t affect the return value.
Dammit!
Okay, loads of checks there, but still nothing more than a
return 0. Fuck!!!You know, I was almost hoping for a macro here. But nah, it’s just a function that crashes my program. A good thing, considering it this one will presumably still hold in production. Now the correct lengths are checked, which is somewhat redundant with the
assert()from the calling function, but no big deal. And we also check for NULL, terrific! Note though that the NULL check is not very useful in practice: unless you’re in an embedded environment that tends to be too small for libsodium to begin with, the OS will cleanly crash your program anyway. Though here we do get rid of the potential Nasal Demons. Most importantly though, libsodium does not, can not, check for wrong non-null pointers, or buffers that are too short, errors that are arguably just as likely as using a NULL pointer. Though to their credit, this tend to be less true when it is embedded in a garbage collected language, where all pointers tend to be either NULL or safe.Anyway, I have just wasted 10 minutes of my time investigating the possible values of a return code, only to find the only code is fucking zero. Oh well, at least it matches the amount of information I get out of it.
This is a problem for static analysis tools and conscientious programmers: each use of this function will either require a useless check with dead code no test can trigger, or will trigger a warning because I’m ignoring the return value. Or maybe there will be no warning, but then I worry about other functions with a return value I should not ignore. No matter how we cut it, this is not ideal.
And this is good code we’re talking about: libsodium isn’t just a good cryptographic library with excellent performance and good security track record, it’s also fairly good C code in general. And yet, there’s this basic error of giving an error code to a function, and then not using it.
I’m blaming C for this.
I think you skipped over the first if statement in
crypto_generichash_blake2bwhere the inputs are checked and a non-zero value is returned. It’s a bit hidden in the docs, but these return values are actually documented https://doc.libsodium.org/quickstart#how-do-i-check-if-a-function-call-succeededOops 🤭
Edit: though to be honest, I still disagree with the conflation of API misuse and actual failures: failures are expected and should totally go to an error code. For instance, checking a corrupt signature or trying to decrypt a forged message should return an error. For misuses however, I prefer asserts and panics, at least by default. Had libsodium adopted that model, its hash functions would return
void, and properly denote that they cannot fail when used correctly.Wait, what?
(void) blah;C has plenty of problems. A lot of problems, one might say. A goddamn monumental shitload of problems, even. But it does let you declare a function that returns nothing. C hasn’t forced anyone to return a meaningless int since 1989.
Correct, and it is fairly low overhead. Still a bummer though, when I can guaranteed the function I’m calling will not fail.
This is why higher-level types are useful and why I find C an unusable language these days. All of the dynamic checks in your example are things that could be represented in the type system in even a slightly higher-level language (including C++). Keys should be a specific type. If the output buffer is a fixed size then it could be a type of that size. In C++, I’d write that API as something like this:
By construction, it’s now guaranteed to pass all of the dynamic checks in the C API. If you need to be able to handle dynamic sizes, then you can implement a function in the header that does the dynamic checks and returns the error result and then everything in your library can assume well-formed inputs.
In a functional language with dependent types, you can express even more useful things, such as ‘the output buffer for this authenticated construction must be the length of the input array plus the space for the integrity tags’. Doing this kind of thing in C++ is much more clunky but doing it in C is not possible.
Yeah I was disappointed when I saw the libsodium API was using untyped pointers all over the place. I had got the vague impression from reading discussions about it that it was supposed to be easier to use than typical cryptographic APIs – which to me implied better type safety to make it harder to misuse. But it seems “easier” meant a careful selection of primitives (good) with straightforward APIs (could be safer) but with cute names like box and secretbox which all look just the same (since the primitives are fixed I would prefer to name the API after the primitive so I can tell them apart!).
Blake2b is awkward for C because all its arrays are variable length (which I think means C++
std::arrayis the wrong type) but the various curve25519 and chacha ciphers ought to use types likeC99 allows a function declaration to specify the relationship between size arguments and array arguments but as far as I know it doesn’t help much for safety except perhaps with a good test suite using
FORTIFY_SOURCE=3and__builtin_dynamic_object_size.I believe I vaguely considered something like this when I designed Monocypher, but ended up deciding against it, mostly in the name of idiomaticity. But realistically, how many errors a stricter API would actually prevent? C does not check bounds by default (or ever…), so what’s preventing me from accessing the array in the struct out of bound?
Well, turns out compilers are now fairly good at catching the most egregious errors, but then they’re equally good at catching out of bound accesses when we write something like
void foo(uint8_t bar[32]): if the compiler catches me accessing more than 32 bytes inside the function I get a warning, and I think I also get a warning if I’m calling the function a buffer the compiler can see is too small — I’m not sure.The most important argument for using bog standard
uint8_t *though is interfacing with other languages. FFIs tend to be a major hassle whenever we deal with anything more complicated than basic types and pointers. Giving them the rawuint8_t *pointers instead makes them easier to write. You’re only safe once you’ve crossed the border, so we might as well make it easier to cross.David’s right though, C is really really weak on this front. I can kinda compensate with a paranoid test suite and all sanitisers I can get my hands on, but a more expressive, safer type system would really go a long way. Just give me fat pointers and automatic bounds checking (warnings & errors at compile time, panics at runtime), and I will sleep much easier.
This should be its own submission!
It could be, if I didn’t make the critical mistake of missing this:
Turns out the function does return an error code after all. Though I confess I still don’t like the design, because as a user, I can guarantee I’m calling the function correctly, and when I do it is guaranteed not to fail. In my opinion a better outcome for failing those checks is to panic with a stack trace right then and there.
Languages that do not embrace “everything is an expression” make REPLs much more awkward and complicated, too, when they have them at all. Not having a simple way of trying out fragments of code piece-by-piece then produces additional knock-on complexity in tools for testing and debugging.
This is what I find so frustrating about Python. On top of having a weird split between statements and expressions, syntactic whitespace and botched lambdas make REPL-oriented development much harder. Some little changes could make it much nicer. Coming from Scheme or Julia makes it a serious downgrade in that regard.
At this point “make invalid states unrepresentable” is a bit old-hat. Great, sure, but what about making invalid state transitions unrepresentable?
Using dependent types one can represent state machines as follows:
As you can see the input and output types depend on the state, so one can disallow transitions (e.g. return only a valid subset of inputs in each state, or even the empty type if no inputs are allowed in some state).
(Dependent records are merely syntactic sugar for Sigma types, i.e. dependent product types, and (x : A) is a Pi type, i.e. a dependent function type.)
A slight variant of these state machines, where we move the
Statefrom being a field inside the record to being a parameter, give raise to predicate transformers:These predicate transformers form a monad where bind is the equivalent of the sequence rule (
;) in Hoare logic, which means we can compose operations whose “protocol” is caputred by the state machine.That’s what the typestate pattern is for.
I am getting deja vu :) And almost exactly a year to the day!
https://nitter.poast.org/tomjaguarpaw/status/1727434384104387055
I have written a typestate program in rust but unfortunately it is unbelievably unwieldy if an operation transitions to one of several possible states.
Half of these seem to be more about strict typing than FP. FP exists outside of Haskell and OCaml.
Returning a specific error type is akin to the reviled checked exceptions in Java: clearer typing, yes, but you’re now on the hook for mapping any errors in functions you call — if your function signature says you only return a LogicError, then if you call something that returns IOError you have to somehow map that to a LogicError or broaden your return error type, which is of course viral. In practice I mostly see functions return some kind of very broad Error type (viz. Go, Objective-C.)
Exceptions “may or may not happen”, but the same is true of errors. Exceptions are like implicitly wrapping every call site with a zero-cost version of “if (error) return error”, the kind of boilerplate that fills Go code (and C code that bothers to handle errors.) Conversely, returned errors are like wrapping every throwing function call with a try/catch block that sets an error variable … whether or not there’s any need to catch the error.
In practice, only a small number of call sites need to do specific error handling (assuming you’ve got something like RAII, or at least GC) so having to wrap explicit checks around all of them adds a ton of noise. And only a small number of calls result in errors, so the overhead of checking each one is unfortunate.
I think what people are against is the implicit nature of exceptions — that a call might exit the function even though there’s nothing written there saying that. They have a point, and I really like the way newer languages require syntax like “try” on such a call. It isn’t necessary, it just serves as an annotation so the programmer knows this might occur.
What I’ve found is that in general you’re right, most call sites don’t need specific error handling, and some black box “general” error is perfectly fine (which is arguably covered by both a catch-all error type or exceptions). However, for the cases where you do want exhaustive error handling, and ensuring that you cover all the edge-cases, then being able to pattern match on all of the possible errors is super handy for preventing bugs.
I think Rust surprisingly gets it right, despite the error handling being rather complicated once you get into the nitty-gritty details. For applications, you can use anyhow for a catch-all error type, and for libraries you use explicit enums (generally using something like thiserror to help facilitate wrapping other errors). What this means is that for the application developer, you can pattern match where you need, but otherwise just bubble up effortlessly.
Rust absolutely does not get that right.
The correct solution here are uniontypes in combination with type-inference.
You then might still want to wrap errors in certain cases, but you don’t have to anymore. Few languages support that, unfortunately.
I think Zig gets it pretty close to right, with a potential error from a function being declared with an error union type and error set inference. The compiler will infer the smallest possible error set, which is suitable for ~95% of end-user cases. (Libraries and the stdlib often like to declare their error types explicitly, for clarity and to ensure they’re not propagating any errors they don’t intend to, and (mutually) recursive functions need at least one of them to declare their error set explicitly.)
The big omission is that there’s no payload for errors; there’s only the error tag value itself.
It’s kind of necessary — without it, how do you handle the error (or hold onto the value, error or not) without propagating? One can think of other ways to do it, but you do need something differentiating propagate-vs-not.
Now how can I do daily affirmations when functional style data deemphasizes identity?
Immutability. Purity. Functions as values and combinators.