Common Lisp for shell scripting

2021/04/11

... let's use SBCL instead of bash!.

Objective

Shell scripts are... just programs. Written in a rather ugly programming language, that was nevertheless designed to be as similar to actual UNIX command invocations as possible (given how it is actual UNIX command invocations for the most part). Using any other language (e.g. Python) definitely comes with some downsides: if most of what you're doing is invoking commands, there will be some ugliness overhead.

On the other hand, you'll win by using a neat programming language. Which is part of the reason Python took off for sysadmin tasks: it's fairly ubiquitous (just like bash), it's easy to write, no compile times, etc.

Common Lisp is great for expressiveness and interactivity. Easier to debug, too. Except... how do you even invoke commands from it?

This article is mostly a dictionary between shell scripts and Common Lisp code. The main goal is not to use Lisp as an actual shell (as in: typing commands interactively); it's about launching processes from Lisp code itself. So, no fancy autocompletion there.

Running Lisp files as scripts

... just put


#!/usr/bin/sbcl --script

in front, give it a chmod a+x, done!

... well, except for the part where if you expect quicklisp / ASDF / UIOP to be available right away (... because it's in your startup script), that one won't run. As a really ugly hack, you could just replace it with...


#!/usr/bin/sbcl --load /home/youruser/.sbclrc --script

... with the obvious caveat of "it only works on your machine with your username". However, if you're giving this script someone else, you should probably pack it up in a neater way anyway; see this stackoverflow answer about this (e.g. if you pack up everything into a binary, you'll get startup times that are a lot better).

Pathnames

A lot of what is happening in shell scripts is manipulating pathnames. So the shell is really easy to use when it comes to this, right? ... right?


OUT_DIR=$(dirname "/some/dir/file.name")  # sets it to /some/dir
FILENAME=$(basename "/some/dir/file.name") # ... it's file.name
EXTENSION="${filename##*.}" # name
FILENAME="${filename%.*}" # file

(I presume people who are writing shell scripts a lot don't have to look up shell parameter expansion syntax every single time. I'm not one of those people.

Meanwhile in Common Lisp, there is a path object. (They might even contain spaces.) So:


(defparameter *my-path* #P"/some/dir/file.name")

(describe *my-path*)
;; #P"/some/dir/file.name"
;;  [structure-object]
;;
;; Slots with :INSTANCE allocation:
;;   HOST       = #<:unix-host>
;;   DEVICE     = NIL
;;   DIRECTORY  = (:ABSOLUTE "some" "dir")
;;   NAME       = "file"
;;   TYPE       = "name"
;;   VERSION    = :NEWEST
;;

(pathname-name *my-path*)
;; "file"

(pathname-type *my-path*)
;; "name"

;; ... etc.

You can also construct pathnames. Sanely. With possible defaults. Want to keep everything but swap out only the extension?


(make-pathname :type "new-extension" :defaults *my-path*)

;;#P"/some/dir/file.new-extension"

This is also the point where reasonable pathname functions mysteriously run out. However, a few missing ones can be found in UIOP, the compatibilty lib used by ASDF; as a result, it's already available basically everywhere.


(uiop:ensure-pathname "/some/random/string")
;; #P"/some/random/string"

(uiop:pathname-parent-directory-pathname (uiop:ensure-pathname "/some/deep/hierarcy/"))
;; #P"/some/deep/"

They might seem a little bit verbose at first, but probably a lot less error-prone and readable than the bash equivalents. There are even gems like


(uiop:enough-pathname #P"/very/long/path/leading/somewhere.txt" #P"/very/long/path/")
;; #P"leading/somewhere.txt"

Launching programs

As it turns out, the requirements for a "build system" and a "shell script" are fairly similar; thus, UIOP doesn't disappoint here, either. Just look at uiop:run-program. We can just run the program and have it dump its output to stdout, just like a simple invocation in a shell script:


(uiop:run-program "ls" :output t :error-output t)

;; doesn't return anything, but will show output from the process

which is, nevertheless, still piped through the lisp process (e.g. if you're using SLIME, it'll come out through the repl, not the inferior-lisp process). However, there is also :interactive, which makes all this fairly close to being an actual shell replacement:


(uiop:run-program "htop" :output :interactive)

(... yes this actually launches a working htop and not random escaped garbage, at least if you run it from an actual console.)

Alternatively, we can request the output as a string:


(uiop:run-program "hostname" :output :string)
;; "our-hostname
;; "
;; NIL
;; 0

... which is evaluated by the shell... as in, actual bash, in case we'd need anything from it:


(uiop:run-program "ls |grep x" :output :string)
;; "a.txt
;; b.txt
;; tmp_list.txt
;; "
;; NIL
;; 0

... but we can also go the "we know exactly what we'd like to run, without passing it to a shell" route. Note the difference between passing in a string as a command, which will run in a shell:


(uiop:run-program "pstree" :output t :error-output t)
;;      (...)
;;      |-sshd-+-sshd---sshd---fish---sbcl-+-sh---pstree
;;      (...)

and the same with passing args as a list, in which case we're launching it directly:


(sb-ext:run-program "program_name" '(arg1 arg2 arg3))
;;      (...)
;;      |-sshd-+-sshd---sshd---fish---sbcl-+-pstree
;;      (...)

For additional convenience, there is ":lines" for ":output"; I don't repeat the docs here, the main point about this is that it exists. You can also launch async processes; we won't detail those here, either.

There are some caveats. Typing "run-program" is kinda tedious even if you do import symbols from the uiop package. Also, by default, everything is going to /dev/null, which is fairly annoying if you're trying to debug a "shell script"; none of is is impossible to fix by a 3 to 5 line function or macro you can write yourself after you're annoyed enough with all of this.

However, on the plus side: all this, just like with pathnames, works on many operating systems. See, for example, Windows-style pathnames! (... and not just by faking /c/some/path; there is an actual "device" component in pathnames).

Replacing pipes, grep, UNIX tools

... um... why are doing this again?

Ohh yes. You think bash is a horrible programming language and you prefer using lisp instead. However, there is a decent chance that you still have your UNIX tools available, including an actual shell instance, which are perfectly capable of grepping, piping, etc., if you just give them the right command line (see the grep example above). Even on Windows, you might have a bash implementation, if you figure out how to launch it from the standard Win command line instead of a fancy MINGW launcher.

As in: if you're porting a really complex shell script, you can still use bash for parts of it. You might replace grepping with some Lispy filtering later; you might not. If you only know how to do something in the shell, you do it in the shell. And by the time we're all back to using Lisp Machines, these problems will have other solutions anyway.

In the meantime, enjoy neat loop constructs, reasonable pathname handling, space-containing pathnames, debuggers, interactivity and shell one-liners together!

This is post no. 14 for Kev Quirk's #100DaysToOffload challenge.

... comments welcome, either in email or on the (eventual) Mastodon post on Fosstodon.