The title looks like clickbait, but I read it anyway and found that this post has a lot of content. It’s about the (in)stability of internet P2P systems in general.
I might be missing something, but instead of building a binary in the test and then run it, I’d rather structure main in a way that allows me to simply unit test it. For example, instead of passing os.Args into the CLI flag parser there’d be a function like the following in the main package:
The main function would just call run(os.Args, os.Stdin, os.Stdout). This way I can test run like any other function, and cover the same code as if I’d be doing an integration test as shown in the article.
But you also need to make sure that the main function works correctly. E.g. if you use meaningful exit codes, you’d want to make sure you test that. There’s a few things that are just different enough between a binary and a function call that a true end-to-end test should use the binary.
GOD is bound as a variable, just like the match arm below it. Rust does not enforce any casing rules on variables vs const. it will warn though that the match arm below it is entirely unreachable.
It’s because the thing on the left hand side is a “pattern”. If the pattern is a constant (for example, “12”), then this is a comparison: the pattern matches if the thing on the right hand side is equal to 12.
But if the pattern is the name of some variable (for example “foo”) then this becomes an assignment instead: the pattern always matches, and the variable is assigned the matching value. So the semantics depend crucially on whether the pattern is the name of a constant or of a variable, and that’s not necessarily obvious at the site of the match expression.
That raises the question: Is otherwise a keyword or just a convention?
I don’t think it’s even convention, otherwise is just a variable name the author chose to use. If anything I’d say it’s convention to use a semantically meaningful name.
If you have a catchall case where you don’t need to bind the value to a variable, you use _, which will communicate that to both the reader of the code and the compiler (and effects borrow checking and drop order).
This has been flagged as spam—to help me learn to make better submissions, could someone explain why that might be? Nominally, this tag is for “links that promote a commercial service”—I don’t see that here, unless it’s the fact that it’s a free post from a paid Substack?
Well, I’m not an AI expert, but I think I understood most of it, more or less. It’s a starkly critical review of a paper about an experiment trying to use AI to generate new scientific research ideas. The writer points out that the results are poor, and the techniques used have some potential dangers.
Honestly, this doesn’t sound like it’s written by an “AI cultist”; the reverse, I’d say. It’s effectively skewering some over-grand claims about what current AI models are useful for. It’s certainly about computing.
I’m not here to fight about it (it’s a garden party, not a debate club)—just to solicit feedback about what makes a good submission. Thanks for providing it.
I guess it can be read both ways. I read it as the author freaking out that the researchers of the papers didn’t try to sandbox their LLM models and is genuinely worried it will “escape”. But the text is so littered with in-jokes and textual tics that it’s hard to tell.
All other major programming languages and ecosystems have internalized this fact and provide solid and secure foundations to their developers. Go is paving the way, with deno and zig following closely.
The extended standard library, let’s call it stdx, should be independent of the language so it can evolves at its own pace and make breaking changes when it’s required.
The real stdx was a project by Brian Anderson (brson), who was the (de facto?) leader of the Rust Project for some years after Graydon Hoare left.
Oh no, please please don’t teach users how to do unauthenticated encryption. I know later on in the article you introduce authentication and that “now let’s make it better” is a fun narrative tool, but users will copy the first example you provide, and they will get hurt. “Eh, I don’t need authentication anyway, just confidentiality” and then surprise they have a padding oracle. Please just don’t. No one should choose CBC in 2024 (or 2014). We don’t teach people to first install live exposed wires and then how to insulate them.
AES-GCM is kinda easier to use than AES-CBC anyway!
P.S. You should also mention that random nonces are dangerous if you encrypt more than 2^32 messages with one key.
Yes, I take the point (and indeed make it to others on a regular basis).
The book, though, is precisely and deliberately “let’s learn why strong cryptography is the way it is, by writing weak cryptography and then breaking it, step by step, until we have something we can’t easily break“. It’s not intended to be a manual of “here’s how to securely use the stdlib crypto packages”; others have done that better than I could, including your good self.
After 2^32 messages with 2^96 nonces, there is a 2^-32 chance of nonce collision, per the birthday paradox. https://c2sp.org/XAES-256-GCM has some links you might find relevant.
I’ll restate what a sibling comment said in a few more words to see if my understanding maps correctly too. Most of this information comes from cryptographers like Soatok who has quite a lot of information about AES-GCM since they are not particularly a fan of it’s design as well as entries specifically about this topic (i.e. symmetric wear out) but also from NIST guidelines.
One invariant that must be held up with AES-GCM is that you absolutely must never re-use a nonce. Never. Since AES-GCM uses a 96 bit (12 byte) nonce, this technically gives a maximum of 2^96 nonces before you run out and must re-key. However, most don’t use sequential nonces because it turns out it’s really, really hard to reliably monotonically update a number and be guaranteed a number does not accidentally repeat (especially in multi-threaded or distributed computing). So randomly generating a nonce is often use. However, if you have 2^96 buckets and you randomly throw a ball in each one, eventually you’ll throw a ball into a used bucket by chance (a birthday collision). For obvious reasons that chance goes up with each nonce generated (ball thrown into a bucket). So cryptographers tend to (AFAIK) use a 50% rule and say after 2^48 nonces generated you’ll probably collide. However, 50% is still too high for many cryptographers so they lower that to 2^32 as an additional safety factor.
This is also often confused with several other “wear out” numbers associated with AES and more specifically AES-GCM such as a maximum message length of 2^36 - 256 bytes, or maximum amount of data safely encrypted under a single (key, nonce) pair of 2^68 bytes (which I believe stems from AES’s 128 bit, or 16 byte, block size regardless of key size). There is also a “H re-use” which is also tied to the block size and AFAIU says that you can only safely generate around 2^48 distinct keys.
Am I correct that that’s shell injection? That tpl.Execute just pastes the text of the argument in, and then shell.Fields would happily parse over that?
Maybe I’m not following, but shell.Fields is a function (from a separate Go library) that expands a string as if were double-quoted in a shell (using Go’s standard library to add variables from the environment). I don’t think that Fields gives things directly to a shell. But maybe I’m misunderstanding the danger you have in mind.
18:27:12|~/tmp
λ bat main.go
package main
import "github.com/bitfield/script"
func main() {
script.ListFiles("*").ExecForEach("touch {{.}}").Wait()
}
18:27:16|~/tmp
λ exa -l
.rw-r--r-- 218 matklad 20 Aug 18:27 go.mod
.rw-r--r-- 3.2k matklad 20 Aug 18:27 go.sum
.rw-r--r-- 0 matklad 20 Aug 18:27 hello world
.rw-r--r-- 124 matklad 20 Aug 18:27 main.go
18:27:18|~/tmp
λ go run main.go
18:27:20|~/tmp
λ exa -l
.rw-r--r-- 218 matklad 20 Aug 18:27 go.mod
.rw-r--r-- 3.2k matklad 20 Aug 18:27 go.sum
.rw-r--r-- 0 matklad 20 Aug 18:27 hello
.rw-r--r-- 0 matklad 20 Aug 18:27 hello world
.rw-r--r-- 124 matklad 20 Aug 18:27 main.go
.rw-r--r-- 0 matklad 20 Aug 18:27 world
It’s technically not a shell injection, because we indeed don’t call into the shell and don’t immediately get arbitrary code execution. Still, we can touch arbitrary files on the file system in the current directory that way, and it could be worse for something more powerful than touch.
I think this highlights that this shell Go package isnt much better than using the shell directly. The downside to building complex shell pipe commands is it can be hard for someone else to understand all the intricacies. But here we have Go functions that are also doing unintuitive things which means the reader is still required to have intimate knowledge of now a new thing, the shell package.
I’ve used Go for “scripting” by doing the long version using what’s in the stdlib. Albeit the verbosity, which grates at the soul, it works fine, and there less footguns. I think if I was to use Go for more scripting I’d keep reaching for the stdlib, and using things like os.File, etc. Any Go developer will understand it.
I think if it’s your library, you get a good speed advantage because you already know how it works and what the abstractions and limitations are. There’s a lot less advantage to using someone else’s convenience library.
And then undo that good choice by parsing the string into fields for convenience. It’s a design decision that I wouldn’t have made, but it’s an unfortunately common one.
I think, in the context of Go, the design might be defensible — I don’t think Go can express this in an ergonomic and safe way, like Rust, Zig, or TypeScript.
What I do find objectionable is not mentioning this in the docs for a function!
we indeed don’t call into the shell and don’t immediately get arbitrary code execution
Got it. I wanted to make sure I understood what you were worried about.
Still, we can touch arbitrary files on the file system in the current directory that way, and it could be worse for something more powerful than touch.
#!/usr/bin/env bash
# Don't do this.
for file in $(ls *)
do
touch $file
done
And you can (sort of? maybe?) fix it with quotes in your Go code.
package main
import "github.com/bitfield/script"
func main() {
// Alternatively: script.ListFiles("*").ExecForEach("touch '{{.}}'").Wait()
// Nope: the alternative is not good. See matklad's comment below.
script.ListFiles("*").ExecForEach(`touch "{{.}}"`).Wait()
}
Based on this discussion (and in particular this comment), the author of the library is aware of the general question. He seems to view it as “working as intended but beware the sharp edges of the shell.” Based on that same discussion, you’re not the only one who isn’t happy about that part of the design.
Actually, I think that’s worse than shell? In shell, if you write "$file", that’s ok for any file, even if includes spaces or quotes. Here, if you write '{{.}}', I’ll name my file Bobby' tables'
In a larger sense, I don’t think I disagree with you. Shell programs are brittle (especially because of word splitting), and insofar as this library imitates the shell, it has similar problems.
Thanks for the very thoughtful discussion. It’s perhaps worth saying that script is mainly intended to let people write as much logic as possible in Go, rather than executing external commands.
That said, it’ll always be useful to execute external commands, so I wonder if there’s anything we can do about the spaces-in-filenames issue. As you mentioned, this is actually nothing to do with script at all, which gladly hands off all that unpleasantness to mvdan’s wonderful sh package: https://github.com/mvdan/sh
To solve this problem in script, in other words, would mean solving it in sh. Should we think about opening an issue there to discuss possible solutions?
I was going to open an issue about this, but I see you’ve read this whole thread now.
I am not sure whether the problem concerns only the sh package. It’s definitely not only about spaces. matklad’s examples suggest that there’s at least three kinds of problems (spaces, single quotes, and double quotes).
The more general issue seems to be about shell escaping or sanitizing and properly splitting arguments for a shell command. At an earlier point in the history of script, you were using https://bitbucket.org/creachadair/shell to handle problems with shell quoting. I think the issues in this discussion are related to the problems you solved that way, but I’m not sure. You’re no longer using shell, and you may have a good reason for that. Also, the code matklad pointed to here involves templates, and after a handful of trial runs, I wonder if the templating code is (itself?) stripping away quotes in an unhelpful way.
$ ls -l
total 24
-rw-r--r-- 1 telemachus wheel 0 Aug 21 07:14 Bobby" Tables"
-rw-r--r-- 1 telemachus wheel 0 Aug 21 07:14 Bobby' Tables'
-rw-r--r-- 1 telemachus wheel 210 Aug 21 07:17 go.mod
-rw-r--r-- 1 telemachus wheel 3221 Aug 21 07:17 go.sum
-rw-r--r-- 1 telemachus wheel 0 Aug 21 07:19 hello world
-rw-r--r-- 1 telemachus wheel 125 Aug 21 07:16 main.go
$ cat main.go
package main
import "github.com/bitfield/script"
func main() {
script.ListFiles("*").ExecForEach("echo {{.}}").Stdout()
}
$ go run main.go
.DS_Store
Bobby Tables
Bobby Tables
go.mod
go.sum
hello world
main.go
I’ll open an issue (somewhere) if you think it would help, but I can no longer tell whether the problem is sh or script or both.
Yes, please do open an issue at script and we’ll continue the investigation there. As a baseline, you could post a script program that produces some unexpected output or effect, and say what you think the output or effect should have been. That should give me enough to help me figure out what’s going on.
The fix I would make is to add a quote helper to the template functions, so then you would write touch {{.|quote}} to get a well escaped quote. Going further, you could fork the template library to get autoquoting, like html/template has, but that seems like too much work.
Counterpoint: shouldn’t the library itself properly quote and sanitize the arguments (as far as possible)? First, users may not know when to take extra care about quoting. Second, when using external programs, isn’t it best to always quote things properly?
Perhaps tag this ‘historical’?
The title looks like clickbait, but I read it anyway and found that this post has a lot of content. It’s about the (in)stability of internet P2P systems in general.
Generally all entries from the domain <computer.rip> are high quality and informative.
So I guess it worked?
Video of the gameplay, just in case you don’t remember playing this: https://youtu.be/_G1-ruFYNGc?si=dMF9UpKP-bk7iXOn
I didn’t know you could run
govulncheckon binaries—that’s useful.gosecwas also new to me.I might be missing something, but instead of building a binary in the test and then run it, I’d rather structure
mainin a way that allows me to simply unit test it. For example, instead of passingos.Argsinto the CLI flag parser there’d be a function like the following in the main package:The
mainfunction would just callrun(os.Args, os.Stdin, os.Stdout). This way I can testrunlike any other function, and cover the same code as if I’d be doing an integration test as shown in the article.Worth mentioning
testscripthere too, as it’s based on the library used to test the Go toolchain itself.But you also need to make sure that the
mainfunction works correctly. E.g. if you use meaningful exit codes, you’d want to make sure you test that. There’s a few things that are just different enough between a binary and a function call that a true end-to-end test should use the binary.I lost it at the
GOD => println!("input was 1"),step. Why is there no error since GOD does not exist?(No pun intended)
GOD is bound as a variable, just like the match arm below it. Rust does not enforce any casing rules on variables vs const. it will warn though that the match arm below it is entirely unreachable.
It’s because the thing on the left hand side is a “pattern”. If the pattern is a constant (for example, “12”), then this is a comparison: the pattern matches if the thing on the right hand side is equal to 12.
But if the pattern is the name of some variable (for example “foo”) then this becomes an assignment instead: the pattern always matches, and the variable is assigned the matching value. So the semantics depend crucially on whether the pattern is the name of a constant or of a variable, and that’s not necessarily obvious at the site of the
matchexpression.Now I got it. Thanks!
That raises the question: Is
otherwisea keyword or just a convention? Seems to be just convention:This prints “Behold: 5”.
I don’t think it’s even convention,
otherwiseis just a variable name the author chose to use. If anything I’d say it’s convention to use a semantically meaningful name.If you have a catchall case where you don’t need to bind the value to a variable, you use
_, which will communicate that to both the reader of the code and the compiler (and effects borrow checking and drop order).otherwisemakes me think the author likes Haskell (see guards, guards!)@gpm is correct,
otherwiseis definitely not a convention. In fact, if you don’t use theotherwisevariable, you get an unused variable error!This has been flagged as spam—to help me learn to make better submissions, could someone explain why that might be? Nominally, this tag is for “links that promote a commercial service”—I don’t see that here, unless it’s the fact that it’s a free post from a paid Substack?
I’d flip the question around - why did you think this was a good submission to lobste.rs?
I found it almost unreadable, as it’s full of references that aren’t explained, or that refer to a specific community that I am not familiar with.
Well, I’m not an AI expert, but I think I understood most of it, more or less. It’s a starkly critical review of a paper about an experiment trying to use AI to generate new scientific research ideas. The writer points out that the results are poor, and the techniques used have some potential dangers.
Honestly, this doesn’t sound like it’s written by an “AI cultist”; the reverse, I’d say. It’s effectively skewering some over-grand claims about what current AI models are useful for. It’s certainly about computing.
I’m not here to fight about it (it’s a garden party, not a debate club)—just to solicit feedback about what makes a good submission. Thanks for providing it.
I guess it can be read both ways. I read it as the author freaking out that the researchers of the papers didn’t try to sandbox their LLM models and is genuinely worried it will “escape”. But the text is so littered with in-jokes and textual tics that it’s hard to tell.
Okay, that’s helpful, thanks.
stdxexists.As does https://blessed.rs/crates which is a hand curated list of libraries for common tasks.
Blessed looks great! I’ve used Rust for a long while but hadn’t come across that before
Man oh man, that would have saved a lot of time.
Sweet sweet abandonware. Truly stable
Yep. You also have
near-stdx,awesome-rust, and probably a bunch more that I couldn’t find in various states of abandonment.Turns out that curating a stdlib is a lot of work. Who knew!
For more context:
This is responding to:
The real
stdxwas a project by Brian Anderson (brson), who was the (de facto?) leader of the Rust Project for some years after Graydon Hoare left.https://lib.rs/std is an up-to-date set.
Oh no, please please don’t teach users how to do unauthenticated encryption. I know later on in the article you introduce authentication and that “now let’s make it better” is a fun narrative tool, but users will copy the first example you provide, and they will get hurt. “Eh, I don’t need authentication anyway, just confidentiality” and then surprise they have a padding oracle. Please just don’t. No one should choose CBC in 2024 (or 2014). We don’t teach people to first install live exposed wires and then how to insulate them.
AES-GCM is kinda easier to use than AES-CBC anyway!
P.S. You should also mention that random nonces are dangerous if you encrypt more than 2^32 messages with one key.
Yes, I take the point (and indeed make it to others on a regular basis).
The book, though, is precisely and deliberately “let’s learn why strong cryptography is the way it is, by writing weak cryptography and then breaking it, step by step, until we have something we can’t easily break“. It’s not intended to be a manual of “here’s how to securely use the stdlib crypto packages”; others have done that better than I could, including your good self.
You can perhaps get a better idea of the flavour of the book here: https://bitfieldconsulting.com/books/crypto
Source?
After 2^32 messages with 2^96 nonces, there is a 2^-32 chance of nonce collision, per the birthday paradox. https://c2sp.org/XAES-256-GCM has some links you might find relevant.
I’ll restate what a sibling comment said in a few more words to see if my understanding maps correctly too. Most of this information comes from cryptographers like Soatok who has quite a lot of information about AES-GCM since they are not particularly a fan of it’s design as well as entries specifically about this topic (i.e. symmetric wear out) but also from NIST guidelines.
One invariant that must be held up with AES-GCM is that you absolutely must never re-use a nonce. Never. Since AES-GCM uses a 96 bit (12 byte) nonce, this technically gives a maximum of 2^96 nonces before you run out and must re-key. However, most don’t use sequential nonces because it turns out it’s really, really hard to reliably monotonically update a number and be guaranteed a number does not accidentally repeat (especially in multi-threaded or distributed computing). So randomly generating a nonce is often use. However, if you have 2^96 buckets and you randomly throw a ball in each one, eventually you’ll throw a ball into a used bucket by chance (a birthday collision). For obvious reasons that chance goes up with each nonce generated (ball thrown into a bucket). So cryptographers tend to (AFAIK) use a 50% rule and say after 2^48 nonces generated you’ll probably collide. However, 50% is still too high for many cryptographers so they lower that to 2^32 as an additional safety factor.
This is also often confused with several other “wear out” numbers associated with AES and more specifically AES-GCM such as a maximum message length of
2^36 - 256bytes, or maximum amount of data safely encrypted under a single(key, nonce)pair of2^68bytes (which I believe stems from AES’s 128 bit, or 16 byte, block size regardless of key size). There is also a “Hre-use” which is also tied to the block size and AFAIU says that you can only safely generate around 2^48 distinct keys.https://github.com/bitfield/script/blob/d4778d8e0fad7a0a8cebc354c4ddd32454b29e1c/script.go#L417-L421
Am I correct that that’s shell injection? That
tpl.Executejust pastes the text of the argument in, and thenshell.Fieldswould happily parse over that?Maybe I’m not following, but
shell.Fieldsis a function (from a separate Go library) that expands a string as if were double-quoted in a shell (using Go’s standard library to add variables from the environment). I don’t think thatFieldsgives things directly to a shell. But maybe I’m misunderstanding the danger you have in mind.That’s what I mean:
It’s technically not a shell injection, because we indeed don’t call into the shell and don’t immediately get arbitrary code execution. Still, we can
toucharbitrary fileson the file systemin the current directory that way, and it could be worse for something more powerful than touch.That sucks but to be fair any tool relying on system instead of execve has the same problem. It’s a common trade off of convenience vs robustness.
I think this highlights that this
shellGo package isnt much better than using the shell directly. The downside to building complex shell pipe commands is it can be hard for someone else to understand all the intricacies. But here we have Go functions that are also doing unintuitive things which means the reader is still required to have intimate knowledge of now a new thing, theshellpackage.I’ve used Go for “scripting” by doing the long version using what’s in the stdlib. Albeit the verbosity, which grates at the soul, it works fine, and there less footguns. I think if I was to use Go for more scripting I’d keep reaching for the stdlib, and using things like os.File, etc. Any Go developer will understand it.
I think if it’s your library, you get a good speed advantage because you already know how it works and what the abstractions and limitations are. There’s a lot less advantage to using someone else’s convenience library.
They use execve though ;-)
And then undo that good choice by parsing the string into fields for convenience. It’s a design decision that I wouldn’t have made, but it’s an unfortunately common one.
I think, in the context of Go, the design might be defensible — I don’t think Go can express this in an ergonomic and safe way, like Rust, Zig, or TypeScript.
What I do find objectionable is not mentioning this in the docs for a function!
Go supports varargs, which is what the standard library uses.
Got it. I wanted to make sure I understood what you were worried about.
Yup, that could be bad. The problem is of course inherited from the shell itself.
And you can (sort of? maybe?) fix it with quotes in your Go code.
Based on this discussion (and in particular this comment), the author of the library is aware of the general question. He seems to view it as “working as intended but beware the sharp edges of the shell.” Based on that same discussion, you’re not the only one who isn’t happy about that part of the design.
Actually, I think that’s worse than shell? In shell, if you write
"$file", that’s ok for any file, even if includes spaces or quotes. Here, if you write'{{.}}', I’ll name my fileBobby' tables'Yup, the version with single quotes is no good. This one (which is parallel to
"$file") works fine even onBobby' Tables'.In a larger sense, I don’t think I disagree with you. Shell programs are brittle (especially because of word splitting), and insofar as this library imitates the shell, it has similar problems.
But it’ll break on
Bobby" tables", no?Yup.
Thanks for the very thoughtful discussion. It’s perhaps worth saying that
scriptis mainly intended to let people write as much logic as possible in Go, rather than executing external commands.That said, it’ll always be useful to execute external commands, so I wonder if there’s anything we can do about the spaces-in-filenames issue. As you mentioned, this is actually nothing to do with
scriptat all, which gladly hands off all that unpleasantness to mvdan’s wonderfulshpackage: https://github.com/mvdan/shTo solve this problem in
script, in other words, would mean solving it insh. Should we think about opening an issue there to discuss possible solutions?I was going to open an issue about this, but I see you’ve read this whole thread now.
I am not sure whether the problem concerns only the
shpackage. It’s definitely not only about spaces. matklad’s examples suggest that there’s at least three kinds of problems (spaces, single quotes, and double quotes).The more general issue seems to be about shell escaping or sanitizing and properly splitting arguments for a shell command. At an earlier point in the history of
script, you were using https://bitbucket.org/creachadair/shell to handle problems with shell quoting. I think the issues in this discussion are related to the problems you solved that way, but I’m not sure. You’re no longer usingshell, and you may have a good reason for that. Also, the code matklad pointed to here involves templates, and after a handful of trial runs, I wonder if the templating code is (itself?) stripping away quotes in an unhelpful way.I’ll open an issue (somewhere) if you think it would help, but I can no longer tell whether the problem is
shorscriptor both.Yes, please do open an issue at
scriptand we’ll continue the investigation there. As a baseline, you could post ascriptprogram that produces some unexpected output or effect, and say what you think the output or effect should have been. That should give me enough to help me figure out what’s going on.The fix I would make is to add a
quotehelper to the template functions, so then you would writetouch {{.|quote}}to get a well escaped quote. Going further, you could fork the template library to get autoquoting, like html/template has, but that seems like too much work.Counterpoint: shouldn’t the library itself properly quote and sanitize the arguments (as far as possible)? First, users may not know when to take extra care about quoting. Second, when using external programs, isn’t it best to always quote things properly?
I agree in principle, but I don’t know how feasible it would be to get it to work.
See https://github.com/bitfield/script/issues/184
That’s more like convergent evolution, rather than inheritance, but yeah..