SSH might surprise you — in all the wrong ways

Martin Kjellstrand

·

Follow

Published in

DevOps.dev

·

6 min read

·

Jul 5, 2023

--

5

Listen

Share

OpenSSH has some very peculiar handling around command line arguments.

It was only after nearly 23 years of using OpenSSH on an almost daily basis that I encountered this issue.

TL;DR

If you just want the short summary, I suggest you skip to “A few simple examples where this really bites you” section.

Background

A while back I was assisting a mentee with a script to automate some tasks and we were using ssh to execute commands on the remote server.

I was just feeding him commands off the top of my head without testing them myself, and he got back to me and said that they didn’t work.

Surely he was mistaken? Turns out, he wasn’t.

Intrigued? I certainly was; Keep reading.

Before we dig into the gory details, let’s examine how processes are actually executed and how command line arguments are passed to the respective program.

Let’s talk about how a program gets executed

When you type a command in the shell, the following description provides a simplified explanation of the underlying process:

1. The shell breaks what you wrote into distinct arguments, observing well-defined rules around what is a unique argument and what isn’t.
2. The shell searches the directories specified in the $PATH environment variable to locate the requested program.
3. The shell employs the exec family of POSIX system calls (such as execl(), execv(), and others) to execute the desired program. In essence, a system call is a mechanism by which an application solicits the kernel to perform an action on its behalf, and the exec family of system calls are used to execute another program.

Example:

$ figlet foobar ‘bar baz’

You shell will search the $PATH environment variable for a program named figlet, and then calls fork() followed by execve() with the arguments /usr/bin/figlet, figlet, foobar and bar baz as show on the below strace output (strace is a Linux-specific application that shows which system calls an application performs, but similar tools exist for other unices):

execve(“/usr/bin/figlet”, [“figlet”, “foobar”, “bar baz”], …

This output from strace shows us that our shell did a system call to the execve routine in the kernel (In the rest of this article, I will interleave output from invocations of execve() calls with the command line illustrations. If you want to try this out yourself you can use the strace -f -e trace=execve command to observe which calls to execve are actually made, and which arguments are passed to them)

As you can see, the quotes around “bar baz” are not passed to execve() because they are part of the shell’s parsing process.

We could have expressed this in a number of functionally equivalent ways using various quoting and expansion primitives, e.g:

$ figlet foobar bar\ baz
execve("/usr/bin/figlet", ["figlet", "foobar", "bar baz"], …
$ figlet foobar "bar baz"
execve("/usr/bin/figlet", ["figlet", "foobar", "bar baz"], …
$ BAZ='bar baz' figlet foobar "${BAZ}"
execve("/usr/bin/figlet", ["figlet", "foobar", "bar baz"], …

As long as we maintain proper quoting, we can nest invocations of sub-shells etc as much as we want, and still get the same result. For example:

$ sh -c 'sh -c "figlet foobar bar\ baz"'
execve("/usr/bin/sh", ["sh", "-c", "sh -c \"figlet foobar bar\\ baz\""], …
execve("/usr/bin/sh", ["sh", "-c", "figlet foobar bar\\ baz"], …
execve("/usr/bin/figlet", ["figlet", "foobar", "bar baz"], …

In each step the quoting/separating into individual arguments works exactly as expected.

After some time, the nesting of quotes and escape characters becomes second nature to us, and we seldom devote much thought to it.

Which is why the next part was so confusing.

Enter OpenSSH

The ssh man page says the following in the SYNOPSIS: (I’ve removed some of the options for brevity)

NAME

ssh — OpenSSH remote login client

SYNOPSIS

ssh [-…] destination [command [argument …]]

This looks awfully similar to the man page of for example the dash shell:

NAME

dash — command interpreter (shell)

SYNOPSIS

dash [-aCefnuvxIimqVEbp] [+aCefnuvxIimqVEbp] [-o option_name]
[+o option_name] [command_file [argument …]]

At this point, I’d argue that most people expect ssh to behave in a transparent fashion (i.e. like a regular sub-shell or any other command), and that the command line arguments are passed to the remote shell according to regular shell expansion rules. Much like rsh does; After all — the concept behind the SSH client command line utility was to create a secure drop-in replacement for utilities like rsh, offering enhanced security features. rsh in turn, was an extension to run sh commands on a remote host.

In other words, we would expect the following commands to be functionally equivalent:

$ figlet foobar bar\ baz
execve("/usr/bin/figlet", ["figlet", "foobar", "bar baz"], …
$ ssh localhost figlet foobar bar\ baz
execve("/usr/bin/ssh", ["ssh", "localhost", "figlet", "foobar", "bar baz"], …
execve("/usr/bin/figlet", ["figlet", "foobar", "bar", "baz"], …

WAT? Apparently our single argument “bar baz” which was passed to ssh as a single argument, was broken up by ssh into two distinct arguments, namely “bar” and “baz”.

Let’s nest it one level deeper, using standard shell expansion rules:

$ sh -c 'figlet foobar bar\ baz'
execve("/usr/bin/sh", ["sh", "-c", "figlet foobar bar\\ baz"], …
execve("/usr/bin/figlet", ["figlet", "foobar", "bar baz"], …
$ ssh localhost sh -c 'figlet foobar bar\ baz'
execve("/usr/bin/ssh", ["ssh", "localhost", "sh", "-c", "figlet foobar bar\\ baz"], …
execve("/usr/bin/sh", ["sh", "-c", "figlet", "foobar", "bar baz"], …
execve("/usr/bin/figlet", ["figlet"], …

Now surely, we’ve even embedded the “bar baz” argument inside single quotes — no way is ssh messing with that, right?

Wrong.

You’ll notice that /usr/bin/sh in the ssh case suddenly was invoked with [“sh”, “-c”, “figlet”, “foobar”, “bar baz”] and not [“sh”, “-c”, “figlet foobar bar\\ baz”] as we would expect.

Since we can see on the following line, ssh is invoked with properly quoted arguments:

execve("/usr/bin/ssh", ["ssh", "localhost", "sh", "-c", "figlet foobar bar\\ baz"], …

but we still end up with the wrong arguments executed on the remote server. (In this case, figlet gets called without any arguments, making it seemingly “hang” since it’s expecting input on stdin).

The ssh manpage vaguely alludes to this behaviour with the following sentence:

If a command is specified, it will be executed on the remote host instead
of a login shell. A complete command line may be specified as command,
or it may have additional arguments. If supplied, the arguments will be
appended to the command, separated by spaces, before it is sent to the
server to be executed.

The sentence I’m referring to is If supplied, the arguments will be appended to the command, separated by spaces, but to my mind it is very unclear and fails to convey the fact that ssh does its own expansion of command line arguments containing spaces themselves.

A few simple examples where this really bites you

$ ssh localhost 'cd /tmp && pwd'
/tmp
$ ssh localhost sh -c 'cd /tmp && pwd'
/home/mad

$ ls documentation\ directory
readme.txt
$ ssh localhost ls documentation\ directory
ls: cannot access 'documentation': No such file or directory
ls: cannot access 'directory': No such file or directory

In conclusion:

How anyone could think this was a good idea is beyond me, but at this point I’m going to assume that it was a either unintentional or intentional design decision taken long ago and at this point openssh can’t fix/change it with the risk of breaking existing code and installations.

Do you have any insights into this? Know the history behind this behavior? Feel like spending a few hours digging through historical openssh releases to uncover the back story behind this?

I’d love to hear about it!