Table of Contents

•  
  • Warning to early readers
    • Current Status
    • What remains to be written
    • Additional ideas
    • Thank you!
  • About
  • Introduction
  • Prerequisites
  • Typographic conventions
• Part One: Development environment
  • Chapter 1: Roswell
    • A quick word about Lisp implementations
    • Installing Roswell
    • Installing Lisp implementations
    • A quick primer on Scripting with Roswell
  • Chapter 2: Lisp libraries
    • ASDF: Another System Definition Facility
    • Quicklisp
    • Local projects
    • Project generators
    • Commonly used libraries
  • Chapter 3: Editing Lisp
    • Emacs
    • Slime
    • Sly
• Part Two: Introduction to web development with Lisp
  • Chapter 4: Caveman2
    • The web stack
    • Setting up a simple project
    • Configuration
    • Views
    • Routing
    • A few words about Clack/Lack
    • Running the app
    • Conclusion
  • Chapter 5: Datafly and SXQL
    • Database access
    • The Wiki Model
    • User model
    • Page model
    • Playing with the model in the repl

Warning to early readers

This is a very early work in progress version of Full Stack Lisp. It is incomplete and possibly contains incorrect information. I’m releasing it as a status update and to get early feedback. Constructive criticism is extremely welcome. I hang out in irc on freenode in #lisp and #lispweb and on twitter @pavelpenev. The book also has a dedicated twitter account @fullstacklisp and a web page. Source code is on github. If you find anything wrong with it please open an issue.

Current Status

Each section of the book is in one of the following states(ordered by proximity to completion):

• To be written
• Early draft
• Late draft
• Needs editing
• Done

The current status of book is as follows

• Frontmatter (Needs editing)
• Part I
  • Chapter 1: Roswell (Late Draft)
  • Chapter 2: Lisp libraries (Early draft)

    Some parts are missing and will possibly be expanded beyond that at a later stage

  • Chapter 3: Editing Lisp (Early draft)

    Many sections are TBW, some parts might get rewritten. Additionally I’ll eventually cover the Atom editor.

• Part II
  • Chapter 4: Caveman2 (Early draft)

    Very early draft, possibly subject to a complete rewrite. Many missing parts.

What remains to be written

Current plans include:

• A chapter walking the reader through developing a simple wiki app

  Will include detailed information on Caveman2’s features, including a detailed Djula tutorial, more Envy examples, and an introduction to it’s DB layer.

• A chapter on alternatives

  Caveman uses Djula for templating and datafly/sxql for DB access. We’ll introduce alternatives, such as s-expression based HTML generation and various database libraries

• A basic Ironclad tutorial

  Introduction to ironclad and how to properly store passwords, API keys, ect.

• A chapter on frontend tools.

  Tools to help with CSS and writing JavaScript such as paredit

• A chapter on deployment

  From a simple Nginx setup to using docker with common lisp

Additional ideas

I have some other ideas that might not make it into the book, such as:

• A chapter on Restful services
• Building desktop apps in a webview
• Packaging executables

If you have other ideas and they seem like they might be in the scope of the book, I’d love to hear from you!

Thank you!

Thank you for reading this early draft! Feedback is extremely crucial at this phase of the project. I might be painting myself into a corner and the only way to know is if somebody tells me. Thank you especially if you bought the book through leanpub. I’m currently self-funding this project, which is a fancy way of saying I’m burning through my savings to write and I don’t know how much I’ll be able to finish before they run out. Your contribution is greatly appreciated and is of great help!

About

Lisp aliens have infected the terrestrial memetosphere with the meme complex of Common Lisp. The mindless human hosts find themselves helpless spreaders of the glory and joy of Lisp Development. Join us and become a great Lisp Programmer! Learn how to build and deploy modern Lisp applications in the booming ecosystem of modern software inhabited by lesser technologies. In the name of alien memetic symbiosis, Go Forth And Hack!

Introduction

After a couple of years of playing around with Scheme and later Clojure around 2008-2010 I made the fateful decision to learn Common Lisp. Originally the plan was to learn just enough Lisp to be able to read “On Lisp” by Paul Graham with proficiency and increase my mad Clojure skills, which was my favorite language at the time. My first reaction to this mutable and downright archaic dialect was a mix between fascination and revulsion. Uppercase symbols, uuuugghhhh! Somehow this mess of a language grew on me and I’ve been in love ever since. I’ve half joked about it infecting my brain, and if you think of a programming language as a memetic complex, it obviously is capable of infecting human hosts, so I’m not sure the joke isn’t an actual fact. Send Help!

When I look back on the last five years I’ve been involved in this horrible mess, the amount of change is astonishing. Lisp might be the mother of all late bloomers, well into its fifties it grew into a viable platform for actual applications written by mortal programmers. The Common Lisp ecosystem is growing into a very usable state, it’s a great time to explore how to actually use this language in a practical manner from top to bottom and deliver modern applications.

This is a book about application development. That means we’ll explore run-times, servers, storage, frameworks and deployment, as well as various ways to use Lisp as part of a client application. In addition tools, libraries and practices the author finds useful will find their way into the book.

This book assumes only basic Lisp knowledge, complete newbies should read an introductory tutorial before reading this book.

The following is a quick outline of some of the topics this book will cover:

• Setting up a development environment
• Web servers, Clack HTTP library and the various frameworks built on top of it.
• Client side lisp ** Lispy HTML/CSS/JavaScript ** Using 3rd party APIs ** Desktop apps
• DBs and ORMs
• Very basic security and crypto
• Deployment and management

Prerequisites

I’m asuming you’re at least somewhat familiar with lisp. If not, you should read a tutorial or an introductory book such as “Practical Common Lisp” or “Land of Lisp”. This book isn’t meant for complete beginners, but you don’t need to know much to get started and pick things as you go along.

You should also have a basic understanding of web development. Terms like HTTP, HTML, web servers, etc. shouldn’t be foreign words to you.

You should have a familiarity with Unix-like operating systems, like Linux, OS X or the BSDs.

Some database knowledge will be extremely useful in later chapters.

As always, your google-fu is your greatest friend.

Typographic conventions

These are the typographic conventions used in this book.

Code appearing inline in text paragraphs looks like this:

This code is inlined: (lambda () (format t "Hello World")).

This is a code block in a file.:

1 (defun hello-world ()
2   (format t "Hello World"))

The following characters represent various prompts:

A * represents a Lisp REPL(Read Eval Print Loop, the lisp interactive interpreter), => marks the returned result:

1  * (format nil "Hello World")
2  => "Hello World"

$ is a unix shell, # is a root shell, or code executed with sudo:

1  # apt-get install foo
2  $ foo --bar baz

> is a windows cmd.exe prompt:

1  > dir C:\

Part One: Development environment

Source

In this part of the book we’ll talk about everything you need in order to set up and use a basic Lisp and Emacs environment. We’ll cover installing Lisp, writing scripts, editing lisp and interacting with the REPL as well as an introduction to some utilities and commonly used libraries that you should know about.

Chapter 1: Roswell

Roswell is a command line interface to common lisp. It’s a way for you to manage your lisp setup from the unix shell. For example it can be used to install lisp implementations, write and run scripts written in Lisp and produce executables among other things. It’s fairly new, but it’s unquestionably one of the most useful tools to come out recently.

A quick word about Lisp implementations

Because Lisp is an ANSI standard anybody is free to implement it themselves, and over the decades a significant number of implementations have appeared. In this section I’ll list some of the more important implementations and why you might choose it over the others.

The most popular open source implementation is SBCL, it has a native compiler and generates very efficient code. With proper optimizations it is not unheard of to get C-like performance out of Lisp programs. SBCL happens to be the authors personal favorite.

CCL is probably the second most popular implementation. Like SBCL it generates native code, although it has a faster compiler(meaning compiling code takes less time). For this reason it might be a better choice for use during development, while SBCL might be better suited for production, although both work well in both cases. An additional feature of CCL is that on OS X it comes with it’s own IDE and a cocoa bridge for GUI programming. Both are useful and very high-quality so in this book we’ll show you how to set them both up.

Some honorable mentions:

ABCL is Common Lisp running on the JVM. If you need to interface to a java system or are familiar with that ecosystem, ABCL deserves a serious look.

ECL or Embeddable Common Lisp is a very small implementation which as the name suggests supports embedding in a C/C++ application. In addition to it’s bytecode compiler it supports native compilation through an intermediate translation to C. It has a much smaller footprint that the other implementations and is generally very cool.

There are also the two major commercial implementations: LispWorks and Allegro Common Lisp. Both with native compilers and their own IDEs, GUI frameworks, and many other libraries. Both are rather expensive, but have limited free versions for trial purpose and might be worth checking out.

In summary, unless you’re doing something specific or just like to play around with interesting systems, SBCL and/or CCL are the implementations you should care about for the moment.

Installing Roswell

The best way to install Roswell is from source:

First let’s install its dependencies:

1  # apt-get install git build-essentials automake libcurl4-openssl-dev

Next we clone the Roswell repo and build it:

1  $ git clone -b release https://github.com/roswell/roswell.git
2  $ cd roswell
3  $ sh bootstrap
4  $ ./configure
5  $ make
6  # make install

Let’s verify that we’ve installed it:

1  $ ros version
2  roswell 0.0.3.56(e0d9c43)
3  build with gcc (Ubuntu 4.9.2-10ubuntu13) 4.9.2
4  libcurl 7.38.0
5  lispdir='/usr/local/share/common-lisp/source/roswell/'
6  configdir='/home/pav/.roswell/'

Type ros help to get a complete list of commands and options.

Installing Lisp implementations

Let’s install SBCL and CCL. This is done with the ros install <implementation>/[<version>] command. You can have multiple versions of the same implementation installed at the same time. At any one time roswell will use one as the default. Let’s see what’s available for installation:

 1  $ ros help install
 2  Usage: ros install impl [OPTIONS]
 3 
 4  For more details on impl specific options, type:
 5   ros help install impl
 6 
 7  Candidates impls for installation are:
 8  ccl-bin
 9  sbcl
10  sbcl-bin
11  clisp
12  ecl

Ok, we have two candidates for SBCL: sbcl and sbcl-bin. The difference is that sbcl-bin downloads and installs a pre-build binary, while installing sbcl will download the source and build it. You should preferably install sbcl from source since on some platforms the pre-build binary was build without threading support and we threads:

1  $ ros install sbcl

When it’s done we can try to run a repl to see if the installation was successful:

1  $ ros run
2  * (+ 1 2)
3 
4  3

It appears it is. Now let’s install CCL:

1  $ ros install ccl-bin

If we have more than one implementation installed we can choose which one we want to use by suplying the -L option:

1  $ ros -L ccl-bin run
2  Welcome to Clozure Common Lisp Version 1.11-r16635  (LinuxX8664)!
3 
4  CCL is developed and maintained by Clozure Associates. For more information
5  about CCL visit http://ccl.clozure.com.  To enquire about Clozure's Common Lisp
6  consulting services e-mail [email protected] or visit http://www.clozure.com.
7 
8  ? (+ 1 2)
9  3

It appears to work. If you want to change the default installation you can use the use command. First let’s list the installed implementations and then change the default:

1  $ ros list installed
2  Installed implementations:
3  ccl-bin
4  ccl-bin/1.11
5  sbcl
6  sbcl/1.3.2
7  $ ros use ccl-bin

Now when we type ros run we’ll get a CCL repl by default.

A quick primer on Scripting with Roswell

Roswell installable scripts

Roswell has the ability to install scripts other people have written and shared through Quicklisp(the Lisp package manager, we’ll take a detailed look at it in the next chapter).

By default roswell stores its scripts in ~/.roswell/bin, let’s add it to the path:

1  $ echo "export PATH=\"\$HOME/.roswell/bin:\$PATH\"" >> ~/.bashrc

Installing new scripts is done with the install command:

1  $ ros install <script name>

For a list of currently available scripts see here

Writing your own scripts

We’ll write a very simple example of an echo script that prints everything you give it back at you. Here’s the code:

 1 #!/bin/sh
 2 #|-*- mode:lisp -*-|#
 3 #|
 4 exec ros -Q -- $0 "$@"
 5 |#
 6 
 7 #|
 8 An echo script
 9 |#
10 
11 (defun main (&rest args)
12   (format t "~{~A ~}~%" args))

Now we must make the script executable:

1  $ chmod +x ~/.roswell/bin/ros-echo.ros

Let’s try it out:

1  $ ros-echo.ros hello
2  hello

How it works

The first five lines basically say how the script is to be executed. First it is run as a normal shell script, but then the script is executed with the ros command with the name of the script as a parameter and the rest of the arguments passed to it.

1 #!/bin/sh
2 #|-*- mode:lisp -*-|#
3 #|
4 exec ros -Q -- $0 "$@"
5 |#

Roswell loads the code and executes the main function. In this case it must take arbitrary number of arguments so we define it with an argument list of (&rest args). The format directive simply prints the each element of the args list with spaces between them and a blank line at the end. It’s literally one of the simplest possible scripts.

11 (defun main (&rest args)
12   (format t "~{~A ~}~%" args))

Compiling scripts

What about performance though? Lets see how fast our script is:

1  $ time ros-echo.ros hello
2  hello
3  ros-echo.ros hello 0,36s user 0,01s system 98% cpu 0,374 total

That isn’t very good. Fortunately Roswell has a way build our script into a binary executable:

1  $ ros build ~/.roswell/bin/ros-echo.ros

This will create a binary executable with the same name, but without the .ros extension. Let’s see how well it performs:

1  $ time ros-echo hello
2  hello
3  ros-echo hello 0,00s user 0,01s system 70% cpu 0,011 total

That’s a sizable improvement.

Dealing with arguments

Let’s modify the script to give it commands. Say we want to extend the script to capitalize, upcase or downcase it’s input. It will look something like this:

 1  $ ros-echo.ros hello world
 2  hello world
 3 
 4  $ ros-echo.ros --capitalize hello world
 5  Hello World 
 6 
 7  $ ros-echo.ros --upcase hello world
 8  HELLO WORLD 
 9 
10  $ ros-echo.ros --downcase HELLO WORLD
11  hello world 

Here’s how the script would look like now:

 1 #!/bin/sh
 2 #|-*- mode:lisp -*-|#
 3 #|
 4 exec ros -Q -- $0 "$@"
 5 |#
 6 
 7 #|
 8 An echo script
 9 |#
10 
11 (defun main (&optional $1 &rest args)  
12   (let ((args (cond ((string-equal $1 "--capitalize") (mapcar #'string-capitaliz\
13 e args))
14                     ((string-equal $1 "--upcase") (mapcar #'string-upcase args))
15                     ((string-equal $1 "--downcase") (mapcar #'string-downcase ar\
16 gs))
17                     (t (cons $1 args)))))
18     (format t "~{~A ~}~%" args)))

First, note the arguments to the main function. The first argument to the script gets bound to $1 and the rest to args. In the cond form, if $1 is one of the valid commands, we transform args appropriately, otherwise we simply stick $1 at the head of the list, since in that case it isn’t a command and we want to echo it as well. This is a very simple example, but it gives an idea of what you could do.

If you want to see more fancy ways to write scripts, check out the source of the Roswell installable scripts here. For more information about Roswell check out it’s wiki.

Chapter 2: Lisp libraries

In this chapter we’ll introduce the ASDF build system, Quicklisp the Lisp package manager as well as several commonly used libraries you should know about.

ASDF: Another System Definition Facility

Lisp code is organized in something called a system. A system is a directory containing all of your source code and a system definition file(with the .asd extension) which tells ASDF how to build and load your project. Almost all modern lisp projects use it.

Let’s look at a very minimal ASDF project to get an idea of what we’re talking about:

1 example-project/
2   README.md
3   example-project.asd
4   src/
5     example-project.lisp

In the example-project directory there are several files, the README.md is self-explanatory. The src directory contains all of our source files and the example-project.asd tells lisp how to build it. Let’s take a look inside

 1 (in-package :cl-user)
 2 (defpackage example-project-asd
 3   (:use :cl :asdf))
 4 (in-package :example-project-asd)
 5 
 6 (defsystem example-project
 7   :version "0.1"
 8   :author "Pavel Penev"
 9   :mailto "[email protected]"
10   :license "MIT"
11   :depends-on (:iterate :alexandria)
12   :components ((:module "src"
13                 :components
14                 ((:file "example-project"))))
15   :description "Example project")

As you can see this is just ordinary lisp code. Let’s look at the defsystem form in more detail, it defines information about the project such as it’s version, author, license, etc. The interesting parts are :depends-on and :components. The :depends-on clause lists the libraries our project depends on, obviously, and the components tells where to find the source code, in this case the file example-project(note it is written without the .lisp extension) in the src directory. More complicated set-ups exist and we’ll look at them in time. For now this will suffice. In the next section we’ll take a look at Quicklisp and how we can load such systems into Lisp.

Quicklisp

Quicklisp is a package manager for lisp. It handles downloading and installation of lisp code. If you’re using roswell it’s already installed on your system. Let’s try it out, Open a lisp repl with ros run and type the following to install the alexandria(a collection of useful utilities) library:

1  * (ql:quickload "alexandria") 
2  => ("alexandria")

Alexandria includes a function which takes a nested list and “flattens it” into a non-nested one, let’s see if we installed alexandria by trying it out

1  * (alexandria:flatten '((1 2) (3 4) (5 6)))
2  => (1 2 3 4 5 6)

Here are some other useful Quicklisp commands:

ql:system-apropos takes a string and searches the names of available systems with it, for example if we search for “json” we get:

 1  * (ql:system-apropos "json")
 2 => #<SYSTEM cl-json / cl-json-20141217-git / quicklisp 2016-02-08>
 3    #<SYSTEM cl-json.test / cl-json-20141217-git / quicklisp 2016-02-08>
 4    #<SYSTEM com.gigamonkeys.json / monkeylib-json-20120208-git / quicklisp 2016-\
 5 02-08>
 6    #<SYSTEM define-json-expander / define-json-expander-20140713-git / quicklisp\
 7  2016-02-08>
 8    #<SYSTEM graph-json / graph-20150407-git / quicklisp 2016-02-08>
 9    #<SYSTEM json-responses / json-responses-20151031-hg / quicklisp 2016-02-08>
10    #<SYSTEM json-responses-test / json-responses-20151031-hg / quicklisp 2016-02\
11 -08>
12    #<SYSTEM json-streams / json-streams-20140713-git / quicklisp 2016-02-08>
13    #<SYSTEM json-streams-tests / json-streams-20140713-git / quicklisp 2016-02-0\
14 8>
15    #<SYSTEM json-template / cl-json-template-20110418-hg / quicklisp 2016-02-08>
16    #<SYSTEM st-json / st-json-20150608-git / quicklisp 2016-02-08>

All of these systems can be loaded with ql:quickload.

The function ql:update-client will update your installation of Quicklisp.

Quicklisp libraries are usually updated once a month, occasionally you’ll have to run (ql:update-all-dists) to get the newest versions.

The function ql:uninstall can be used to remove an installed system.

That about covers the basic Quicklisp usage.

Local projects

Quicklisp isn’t only useful for installing remote libraries. It can also be used to load local ASDF systems. By default when you use ql:quickload Quicklisp searches the directories defined in the variable ql:*local-project-directories* for systems with the name you want to install, if they are not found there it tries to download them off the internet. If you use roswell among these directories will be ~/.roswell/local-projects. This is where you can put your lisp code if you want it to be quickload-able.

TBW about asdf:*central-registry*

Project generators

TBW

Commonly used libraries

Below are a number of commonly used libraries and tools for Common Lisp. Some of them you might find useful, but mostly you should familiarize yourself with them, since you are likely to have to read code that uses them.

Quickdocs

Quickdocs is a website that contains documentation for a vast majority of active Lisp projects. You can browse it to find interesting libraries.

Alexandria

Website API

Alexandria is possibly the most popular Lisp library. It’s conservative collection of utilities meant to be portable across implementations. Quickdocs reports that 289 projects in Quicklisp use it. You will have to read code that uses it fairly often and it’s way too useful not to know. I highly recommend reading it’s documentation and becoming familiar with it. Here’s some of the stuff in it:

• Hash table utilities
• Data and Control Flow
• Utilities to work with cons cell based data structures
• Sequence utilities
• Utilities to read files into strings or vectors
• Macro writing utilities
• Utilities for dealing with Symbols, Arrays, Types and Numbers

SPLIT-SEQUENCE

API

The split-sequence library is very simple. It contains only 3 functions: split-sequence, split-sequence-if and split-sequence-if-not. Like alexandria, it’s very commonly used.

Iterate

Website API

iterate is an iteration construct similar to the built in loop, but arguably better. It’s very popular and even if you don’t use it yourself you’ll find yourself reading code that uses it very often, so it would be valuable if you get at least a basic understanding of its syntax.

Bordeaux-threads

Website API

Bordeaux-threads is a portable threading library for CL. Although the standard doesn’t say anything about threads, most Lisp implementations provide them, unfortunately all hove minor differences. BT takes care of the problem by providing a common interface. If you do anything with threads, this library is a must.

Other useful library lists

cliki.net has a list of currently recommended libraries

There is also Fernando Borretti’s opinionated CL State of the Union article which lists some recommendations

Finally you can look at Quicklisp download statistics to get an idea of what are some other commonly used libraries. The latest is from 2015-08-23

Chapter 3: Editing Lisp

Because Lisp is a very interactive language in order to make full use of it, you need an interactive IDE. Users of the commercial Lisps(acl and LispWorks) can enjoy full feature IDEs, but in the Open Source world we are currently limited. The best options are Emacs and Vim with Slime and slimv(Superior Lisp interaction mode for Emacs/Superior Lisp interaction mode for Vim) or the new Emacs mode Sly(a fork of Slime with many improvements). A port of Slime is in progress for Atom, but it isn’t very feature complete yet and it’s github page claims it’s still in Beta. If you don’t want to use Emacs or Vim you might find this project interesting. Other than that there is the CCL IDE, but it only works on OS X. The last and least effective option is to write it like you would any other popular language. Write code in any editor you chose, compile manually, run and edit again. This option essentially nullifies a good chunk of Lisps advantages, at that point you might as well be writing in Blub. If you go with this option you will still have the power of the Common Lisp language, but not of the Common Lisp system. Lisp code wants to be developed interactively. Never the less, if you are a beginner and already use an editor that doesn’t have an interactive lisp mode and you don’t want to switch just yet, you can choose this option, at least for now. Eventually though you’ll want to use the best tools available.

I highly recommend Emacs though. In this chapter I’ll introduce it and Slime, as well as spend a bit of time discussing Sly as an alternative option. If Emacs isn’t your thing you could skip this chapter.

Emacs

More than a mere text editor, Emacs is a rabbit hole of power. On the pyramid of editors, Emacs stands on top as the One True Editor. And the reason for that is very simple, having been born in the Lisp world it embodies the idea that a system should be completely modifiable, extensible and inspectable. Pretty much all of Emacs can be modified while it’s running, it’s self-documenting, you can pull up the documentation for every key binding, every interactive command and you can jump to the source of pretty much every part of the system and edit it(at least the parts written on Lisp, the core of Emacs is in C). If there was ever a tool designed to adapt to you, rather than you to it, it’s Emacs. Throughout the decades millions of lines of Emacs Lisp code has been written in Emacs extensions. Everything from simple syntax high-lighting modes for programming languages to a full-blown text based web browser, email client, IRC client and who knows what else. People sometimes joke that Emacs is an OS and not an editor, but that joke is at least partially true. Other than common lisp itself, Emacs is one of the few bits of 80s Lisp Machines tech that survived into the modern world.

Getting started

TBW: Installation on various platforms

A quick note about notation: Ctrl and Alt is usually written as C and M in most Emacs tutorials, so Ctrl-s would be written as C-s and Alt-x would be written as M-x. The reason M is used is because on old systems Alt used to be called Meta and all emacs literature still calls it that. When you read the word Meta anywhere in relation to Emacs it simply means Alt. Sometimes to trigger a command more than one combination is used, for example saving a file is done with the key chord C-x C-s, which would mean you press the Ctrl key at the same time as x, and then you press Ctrl at the same time as s, alternatively, you can hold down Ctrl and then press x followed by s. This is a different command than C-x s in which case you type C-x followed by s. Emacs includes an interactive tutorial that will get you started with the most common editing commands as well as most of the relevant terminology. This is a prerequisite for the rest of the chapter. Once you start Emacs you can access it by typing C-h t(you press C-h, let go and then you type t). Spend some time with it and come back to this book. In the next section I’ll show you how to customize Emacs.

A few more notes

As you’ve noticed in the tutorial the way you move around a file in Emacs is with the C-p, C-n, C-b and C-f keys. These do the same things as the arrow keys. It’s a trade-off, unlike the arrow keys they are key combinations rather than a single stroke, but unlike the arrow keys they are on the keyboard itself rather than on the side, in other words you don’t have to remove your hand from the typing position. Touch-typists will find the emacs key-bindings faster, especially with a little practice. Some users even disable the arrow keys. What I did wasn’t as dramatic. I simply put a sticky note above my arrow keys and practiced using the emacs keys.

Another point to make is that the key-chords are in fact a bit awkward. Pressing Ctrl all the time could hurt your pinkie finger. There are two solutions to this problem. One is to rebind Caps Lock to Ctrl. Many emacs users do this since Caps Lock is closer to the home row of the keyboard and it makes tying key chords much easier and doesn’t stress the pinkie as much. The other option is to swap the Alt and Ctrl keys. On the keyboards on which emacs was developed the Ctrl keys were on either side of the Space key and the user could press them with their thumbs rather than use their pinkies. This is the better option in my opinion. My own personal set up binds the Alt keys to Ctrl and I use the Win keys(with the Windows logo) to Alt. This is done in the KDE keyboard settings(the desktop environment I use).

TBW: Write an explanation on how to do these things for various platforms

A basic Emacs setup

The entry point for most Emacs customization is the file ~/.emacs. I’ve created an example .emacs file for your use based on my own customizations. You can find the file here. I’ve included it below. It’s heavily commented and I urge you to read it carefully before you decide to use it. It’s part of this tutorial:

  1 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
  2 ;; .emacs                                                                    ;;
  3 ;; Author: Pavel Penev                                                       ;;
  4 ;; Licence: MIT                                                              ;;
  5 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
  6 
  7 ;; This file describes a basic Emacs setup. Feel free to modify it to
  8 ;; your liking.
  9 
 10 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 11 ;; Section I: Generic settings                                               ;;
 12 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 13 
 14 ;; The following 3 lines disable unnecessary GUI elements, in this case the
 15 ;; menu bar, the tool bar and the scroll bar. If you wish, you can comment out
 16 ;; the menu-bar and keep it, but eventually I recommend you disable it.
 17 
 18 (if (fboundp 'menu-bar-mode) (menu-bar-mode -1))
 19 (if (fboundp 'tool-bar-mode) (tool-bar-mode -1))
 20 (if (fboundp 'scroll-bar-mode) (scroll-bar-mode -1))
 21 
 22 ;; Sometimes the mini-buffer becomes multi-line, and it can be a bit annoying as
 23 ;; you type in it. This makes it stay one line.
 24 
 25 (setq resize-mini-windows nil)
 26 
 27 ;; We don't need the Emacs splash screen. You can keep it on if you're into
 28 ;; that sort of thing
 29 
 30 (setq inhibit-splash-screen t)
 31 
 32 ;;; global-linum-mode adds line numbers to all open windows, In my opinion
 33 ;;; there is no reason not to have this enabled.
 34 
 35 (global-linum-mode)
 36 
 37 ;;; Emacs comes with a built-in text based browser. Since I use the browse-url
 38 ;;; function most often to browse documentation, I've set it to eww, the Emacs
 39 ;;; Web Browser. It works well for that purpose. If you would prefer to use a
 40 ;;; graphical browser, you can change this line.
 41 
 42 (setq browse-url-browser-function 'eww-browse-url)
 43 
 44 ;;; I prefer to make the default font slightly smaller.
 45 
 46 (set-face-attribute 'default nil :height 90)
 47 
 48 ;; Show matching parentecies globaly.
 49 
 50 (show-paren-mode 1)
 51 
 52 ;; Use UTF-8 by default
 53 (set-language-environment "UTF-8")
 54 
 55 ;; Don't ring the bell. It saves some annoyance
 56 
 57 (setq ring-bell-function 'ignore)
 58 
 59 ;; If you're one of the heathens who prefers tabs over spaces, you should
 60 ;; remove the following line. It makes indentation use spaces.
 61 (setq-default indent-tabs-mode nil)
 62 
 63 ;;; A simple backup setup. Makes sure I don't foo~ and #.foo files in
 64 ;;; directories with files you edit.
 65 
 66 (setq
 67  backup-by-copying t      ; don't clobber symlinks-
 68  backup-directory-alist
 69  '(("." . "~/.saves"))    ; don't litter my fs tree
 70  auto-save-file-name-transforms
 71  `((".*" ,temporary-file-directory t))
 72  delete-old-versions t
 73  kept-new-versions 6
 74  kept-old-versions 2
 75  version-control t)       ; use versioned backups
 76 
 77 
 78 ;; Set up emacs server. This allows you to run emacs in the background and
 79 ;; connect to it with emacs client. It reduces startup time significantly. If
 80 ;; the server is not running, it starts it.
 81 
 82 (load "server")
 83 
 84 (unless (server-running-p)
 85   (server-start))
 86 
 87 ;;; ido-mode, or Interactively DO mode, adds lots of improvements when working
 88 ;;; with buffers of files. You can read more about it at:
 89 ;;; https://www.emacswiki.org/emacs-test/InteractivelyDoThings
 90 (require 'ido)
 91 (ido-mode t)
 92 
 93 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 94 ;; Section II: Packages                                                      ;;
 95 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 96 
 97 ;; Elpa, the default package repository for emacs is fairly conservative, so
 98 ;; we'll add the melpa and marbaraie repositories
 99 (require 'package)
100 (add-to-list 'package-archives
101 	     '("marmalade" . "http://marmalade-repo.org/packages/"))
102 
103 (add-to-list 'package-archives
104              '("melpa" . "http://melpa.org/packages/") t)
105 
106 ;; You can install packages by typing M-x package-install <package-name>. I
107 ;; recomend you install the following packages: smex, which adds an improved
108 ;; version of M-x. I highly recomend this. You can read more about smex at:
109 ;; https://github.com/nonsequitur/smex/
110 
111 ;; Another ofthen used mode is magit, which is an interface to git, allowing
112 ;; you to manage your repos through emacs. You can read more about it at:
113 ;; http://magit.vc/ It is one of the most useful modes available for emacs
114 
115 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
116 ;; Section III: Global Key Bindings                                          ;;
117 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
118 
119 ;; By default C-x o is bound to 'other window, but I find I use it much more
120 ;; ofther than open-line, which is bound to C-o, so I swap their definitions
121 (global-set-key (kbd "C-o") 'other-window)
122 (global-set-key (kbd "C-x o") 'open-line)
123 
124 ;; M-0..3 are bound to 'digit-argument. To be used with C-u. I don't use them
125 ;; ofthen, so I prefer to rebind them to the window commands, since M-1 is
126 ;; easyer to type than C-x 1. 
127 (global-set-key (kbd "M-1") 'delete-other-windows)
128 (global-set-key (kbd "M-2") 'split-window-vertically)
129 (global-set-key (kbd "M-3") 'split-window-horizontally)
130 (global-set-key (kbd "M-0") 'delete-window)
131 
132 ;; Set the enter key to newline-and-indent which inserts a new line and then
133 ;; indents according to the major mode. This is very convenient. 
134 (global-set-key (kbd "<RET>") 'newline-and-indent)
135 
136 ;; If you have installed smex, you can uncomment the following lines. To
137 ;; activate it without restarting Emacs select the lines and type M-x eval-regio\
138 n:
139 
140 ;; (require 'smex)
141 
142 ;; (global-set-key (kbd "M-x") 'smex) 
143 ;; (global-set-key (kbd "M-X") 'smex-major-mode-commands) 
144 ;; (global-set-key (kbd "C-c C-c M-x") 'execute-extended-command)

Slime

The Superior Lisp Interaction Mode for Emacs is the most widely used IDE for Common Lisp development. It augments Emacs with an editing mode, a REPL, a debugger and an object inspector, as well as referencing facility allowing you to jump to various points in Lisp source code.

Installation

The easiest way to install slime is to use Quicklisp and Roswell. From the command line do this:

1  $ ros -Q -e '(ql:quickload :quicklisp-slime-helper)' -q

This line will tell roswell to start Lisp with Quicklisp loaded and evaluate the expression (ql:quickload :quicklisp-slime-helper) which installs Slime and then quit.

Slime has two parts, The emacs part called slime and the Common Lisp part called swank. In order for Emacs to know where slime is, we must add the following line to our .emacs file:

1 (load (expand-file-name "~/.roswell/impls/ALL/ALL/quicklisp/slime-helper.el"))

Now we must tell slime how to start a Lisp repl with the following code

1 (setf slime-lisp-implementations
2       `((sbcl ("ros" "-Q" "-l" "~/.sbclrc" -L "sbcl" "run"))
3         (ccl  ("ros" "-Q" "-l" "~/.ccl-init.lisp" "-L" "ccl-bin" "run"))))
4 
5 (setf slime-default-lisp 'sbcl)
6 
7 (setq slime-net-coding-system 'utf-8-unix)

The first form is a list of installed Lisp implementations and how to start them. By default Roswell doesn’t load the implementation init files(.sbclrc and .ccl-init.lisp in this case) so we include them with the -l options. With the -L option we specify which implementation we want to start. The second form sets SBCL as the default. When you start slime this is the one that will be used. In order to start slime with a different implementation you could do M-- M-x slime ccl and a repl running ccl will start up. The last form tells slime to use utf-8 when it’s communicating with the Lisp process.

Next Let’s modify the way buffers with Lisp code behave:

 1 (defun lisp-hook-fn ()
 2   (interactive)
 3   ;; Start slime mode
 4   (slime-mode)
 5   ;; Some useful key-bindings
 6   (local-set-key [tab] 'slime-complete-symbol)
 7   (local-set-key (kbd "M-q") 'slime-reindent-defun)
 8   ;; We set the indent function. common-lisp-indent-function 
 9   ;; will indent our code the right way
10   (set (make-local-variable lisp-indent-function) 'common-lisp-indent-function)
11   ;; We tell slime to not load failed compiled code
12   (setq slime-load-failed-fasl 'never))
13 
14 ;; Finally we tell lisp-mode to run our function on startup
15 (add-hook 'lisp-mode-hook 'lisp-hook-fn)

The function lisp-hook-fn is set up to be called whenever lisp-mode is activated in emacs, it starts slime-mode sets up a few key bindings and alters a few settings. This code makes the Tab key use the slime auto-completion function instead of the default one. It also binds M-q to the lisp-indent-function Whenever you type M-q while the point is in a lisp block of code it will re-indent it according to the lisp convention. For that reason we set the lisp-indent-function to common-lisp-indent-function. We also tell slime to never load code that failed compilation.

The Slime repl

The Slime repl is one of it’s most central features. You might have noticed that SBCL and CCL both have very simple repls, and are generally unsuited for interactive work. The slime repl on the other hand is much more feature complete. It starts up automatically when you type M-x slime.

This is how it looks like when you start it:

A new buffer called *slime-repl sbcl* is created where the repl lives. Notice the repl prompt is CL-USER>. This signifies that by default you are in the CL-USER package. If you change the package, for example with in-package the prompt will change to reflect that.

In the repl you have features like auto-completion bound to the Tab character, and a full history. You can move up and down the history with the arrow keys or with M-p and M-n. You can also search the history up and down with M-s and M-r which function very similarly to the Emacs built-in commands C-s and C-r.

Slime also displays the argument lists for functions and macros in the mini-buffer, for example if you type (mapcar in the repl, the mini-buffer will display (mapcar function list &rest more-lists) telling you how to use the function.

Slime also includes a bunch of built in shortcuts you can activate by pressing the ,(comma) key, when you press it you will be prompted in the mini-buffer for a command. One such a command is !p which sets the current package, for example , !p foo-package will set the current package to foo-package. Note that the package must be defined. You also get auto-complete with Tab in this prompt. For a complete list of commands see here

Editing Lisp code

Paredit

TBW

Navigation

Slime includes a function to go to the definition of a symbol and edit it called slime-edit-definition, by default it’s bound to M-.. You can use it as many times as you wish, and when you are done you can go back to the previous location you were editing with the command slime-pop-find-definition-stack bound to M-,.

Compilation and evaluation.

When you are editing lisp code you would want to run your code. There are several ways to do this. The first one you already know about, you can use ql:quickload to load your project into Lisp, but while you’re editing code you’ll want to re-compile individual files or even individual code blocks. Slime includes functions to help you with this.

First let’s talk briefly about running lisp code. There are two ways you can run lisp code, the first is to evaluate it, the second is to compile and load it. When you tell slime to evaluate your code it sends it to Lisp and calls eval on it. Depending on how eval is implemented your implementation might directly interpret the code, or it might compile it first. For example by default SBCL compiles all code unless you enable the interpreter explicitly. When you tell slime to compile your code it sends it to Lisp and runs compile on it.

Here are some of the available evaluation commands:

• C-x C-e will evaluate the last expression before the point and display the result in the mini-buffer.
• C-M-x will evaluate the top-level form where the point is in, for example if you’re in the middle of editing a defun form it will evaluate it. Again the result will be displayed in the mini-buffer.
• C-c C-r will do the same to a selected region.

Generally you would use these commands if you want to see the result of evaluation in the mini-buffer. For example evaluating (+ 1 2) would display 3, but compiling it will simply tell you that compilation was finished and not the result.

Additionally another useful evaluation command is C-c C-j it evaluates the last expression by copying it to the repl and displaying the result there.

Compilation works slightly differently. When your code is compiled and run, you will only see a “compilation finished” message as well as a summary of how many notes, warnings and errors you got.

The most common compilation commands are:

• C-c C-c compiled the top-level form. It works similarly to C-M-x. You’ll be using this one the most often when working with lisp code.
• C-c C-k compiles the whole file.

Generally when you’re working with lisp code you will be recompiling functions as you edit them and testing the results in the repl. When you compile code you get the result of the compilation printed in the *slime-compilation* buffer. This is where all of the compiler notes(errors and warnings) will be displayed. Let’s look at an example, suppose I have a file with the following lisp intentionally wrong function:

1 (defun foo (foo)
2   (let (nil)))

Now I put the point anywhere in that form and type C-c C-c the *slime-compilation* buffer will pop-up with the following contents:

 1 cd /home/pav/example/
 2 1 compiler notes:
 3 
 4 foo.lisp:1:1:
 5   style-warning: 
 6     The variable FOO is defined but never used.
 7     --> PROGN SB-IMPL::%DEFUN SB-IMPL::%DEFUN SB-INT:NAMED-LAMBDA 
 8     ==>
 9       #'(SB-INT:NAMED-LAMBDA FOO
10             (FOO)
11           (BLOCK FOO
12             (LET (())
13               )))
14     
15 
16 foo.lisp:2:3:
17   error: NIL names a defined constant, and cannot be used as a local variable.
18 
19 Compilation failed.

We see 1 style warning and 1 error. The style warning states that the variable foo is never used even though we defined it. This is valid code, but could be a sign of a bug. The error states that the symbol NIL cannot be used as a local variable in let. Notice the warnings and notes also include the file name and the position, those lines are links to the file position where the note occurred. For example you could click with the mouse on them and they will bring you to the offending code.

The compilation buffer includes custom keyboard commands to navigate it. The commands M-p and M-n will go through all of the notes. If you typed C-c C-c in this buffer it will bring you to the offending code as if you clicked on the link. Notice also in the code buffer that the wrong code is marked with colored underlines: Yellow for a warning and red for an error. Once you’re done with the compiler buffer you can quit it with the q key.

The Slime debugger

TBW

The Slime inspector

Slime includes an inspector witch which you can inspect values in your running system. Suppose we have the following code:

1 (defparameter *list* (list "Hello" 'world))

Our list contains a string and a symbol. If you compiled this code you can inspect the variable by moving the point to the name *list* and typing C-c I. The inspector buffer will pop up and it will look something like this:

1 #<CONS {10037829B7}>
2 --------------------
3 A proper list:
4 0: "Hello"
5 1: WORLD

This tells you that the object you’re inspecting is a cons cell and is a proper list, followed by it’s contents. You can use the Tab key to move between the inspectable values in the buffer and descent into them. For example if you type Tab the point will move to the value "Hello" if you now type Return you will inspect that value. It will look like this:

 1 #<(SIMPLE-ARRAY CHARACTER (5)) {100378293F}>
 2 --------------------
 3 Dimensions: (5)
 4 Element type: CHARACTER
 5 Total size: 5
 6 Adjustable: NIL
 7 Fill pointer: NIL
 8 Contents:
 9 0: #\H
10 1: #\e
11 2: #\l
12 3: #\l
13 4: #\o

This tells us that the string is a simple array of characters, it has 5 elements and we can do the same again. Type Tab until the point is on the H character and press Return:

1 #<STANDARD-CHAR {4849}>
2 --------------------
3 Char code: 72
4 Lower cased: #\h
5 Upper cased: @2=#\H

You can go on like this, but how do you go back? The l key will pop you back up. Pressing it a couple of times will get you back to the inspector for *list*. You can quit the inspector with q. ### Other useful information

Here are some other useful features you might want to know about:

• C-c C-d d will describe a symbol. A buffer with a description of the symbol will pop-up.
• C-c C-d f will do the same but for functions.
• C-c C-d a will run slime-apropos, it will search the defined symbols with the string you supply it.
• C-c C-d h will look-up a symbol in the hyperspec and display the result in a web browser. You’ll be using this one a lot.
• C-c C-m and C-c M-m are bound to slime-macroexpand-1 and slime-macroexpand-all respectively. A buffer will open with the macro expansion.

Sly

TBW

Part Two: Introduction to web development with Lisp

In this part of the book I’ll introduce basic web development tools for Lisp. These include the Caveman2 framework and it’s assosiated libraries. We’ll cover creating a caveman project and we’ll walk trough it’s basic functionality.

Chapter 4: Caveman2

The web stack

Caveman2 is a rather simple but feature complete framework for web development. It includes most of what you’d expect such a tool to have, such as a template language, database access, configuration, url routing, ect.

Caveman is based on ningle, a micro-framework which handles only the most basic functionality, such as url routing. Ningle itself is based on the Clack HTTP library and server interface. Because common lisp apps can support many different web servers such as hunchentoot and woo, you need a common interface between them, clack fills that role. Other than abstracting the server backend it also provides request and response objects and handler middleware. We’ll take a deeper look into these features in due time. For now let’s take a closer look at caveman itself.

In this chapter we’ll take a look at what caveman provides for us and we’ll setup a simple project. Here’s what we’ve got:

• Ningle based url routing
• Database access using the datafly and sxql libraries
• The Djula templating library, a port of the Django template language.
• A configuration system called ENVY that allows us to configure our app using environment variables.

Setting up a simple project

First let’s install caveman with quicklisp

1  * (ql:quickload "caveman2") 
2  => ("caveman2")

Caveman comes with a built in project generator. Let’s say we want to write our own wiki, here is how we can generate a project skeleton for it:

 1  * (caveman2:make-project #P"~/.roswell/local-projects/fullstackwiki"
 2                           :author "Pavel" 
 3                           :license "MIT")
 4 
 5 writing ~/.roswell/local-projects/fullstackwiki/fullstackwiki.asd
 6 writing ~/.roswell/local-projects/fullstackwiki/fullstackwiki-test.asd
 7 writing ~/.roswell/local-projects/fullstackwiki/app.lisp
 8 writing ~/.roswell/local-projects/fullstackwiki/README.markdown
 9 writing ~/.roswell/local-projects/fullstackwiki/.gitignore
10 writing ~/.roswell/local-projects/fullstackwiki/db/schema.sql
11 writing ~/.roswell/local-projects/fullstackwiki/src/config.lisp
12 writing ~/.roswell/local-projects/fullstackwiki/src/db.lisp
13 writing ~/.roswell/local-projects/fullstackwiki/src/main.lisp
14 writing ~/.roswell/local-projects/fullstackwiki/src/view.lisp
15 writing ~/.roswell/local-projects/fullstackwiki/src/web.lisp
16 writing ~/.roswell/local-projects/fullstackwiki/static/css/main.css
17 writing ~/.roswell/local-projects/fullstackwiki/t/fullstackwiki.lisp
18 writing ~/.roswell/local-projects/fullstackwiki/templates/index.html
19 writing ~/.roswell/local-projects/fullstackwiki/templates/_errors/404.html
20 writing ~/.roswell/local-projects/fullstackwiki/templates/layouts/default.html
21 
22  => T

What we did was we called the function caveman2:make-project with the path to our project as an argument. We put our project in local-projects, so ASDF and quicklisp can find it. We called the project “fullstackwiki” and we gave the author name and license as keyword arguments. Note that we must pass it a pathname object rather than a string.

As you can see from the printed output of make-project it generated quite a bit of files. Let’s take closer look.

The root directory of our project contains asd files for our app and an auto generated test system, a README file and a lisp specific .gitignore file.

Let’s take a closer look at fullstackwiki.asd

 1 (in-package :cl-user)
 2 (defpackage fullstackwiki-asd
 3   (:use :cl :asdf))
 4 (in-package :fullstackwiki-asd)
 5 
 6 (defsystem fullstackwiki
 7   :version "0.1"
 8   :author "Pavel"
 9   :license "MIT"
10   :depends-on (:clack
11                :lack
12                :caveman2
13                :envy
14                :cl-ppcre
15                :uiop
16 
17                ;; for @route annotation
18                :cl-syntax-annot
19 
20                ;; HTML Template
21                :djula
22 
23                ;; for DB
24                :datafly
25                :sxql)
26   :components ((:module "src"
27                 :components
28                 ((:file "main" :depends-on ("config" "view" "db"))
29                  (:file "web" :depends-on ("view"))
30                  (:file "view" :depends-on ("config"))
31                  (:file "db" :depends-on ("config"))
32                  (:file "config"))))
33   :description ""
34   :in-order-to ((test-op (load-op fullstackwiki-test))))

Here we see that the code to our app is in the “src” directory(called a module by ASDF), and it has the following componets:

• config.lisp is the main configuration, it uses ENVY.
• db.lisp defines our database layer
• view.lisp handles html templates
• web.lisp defines our routes
• main.lisp handles how our app is started and stopped

We’ll take a look at all of these files in the next sections.

Configuration

Envy is a configuration utility which allows you to define several different configurations and then switch between them based on the value of an environment variable. For example you can have a separate configuration on a development machine and one for production deployment. It also allows you to define a common configuration and have the other ones just define the differences, rather than duplicate code. Let’s take a closer look at src/config.lisp:

 1 (in-package :cl-user)
 2 (defpackage fullstackwiki.config
 3   (:use :cl)
 4   (:import-from :envy
 5                 :config-env-var
 6                 :defconfig)
 7   (:export :config
 8            :*application-root*
 9            :*static-directory*
10            :*template-directory*
11            :appenv
12            :developmentp
13            :productionp))
14 (in-package :fullstackwiki.config)
15 
16 (setf (config-env-var) "APP_ENV")
17 
18 (defparameter *application-root*   (asdf:system-source-directory :fullstackwiki))
19 (defparameter *static-directory*   (merge-pathnames #P"static/" *application-roo\
20 t*))
21 (defparameter *template-directory* (merge-pathnames #P"templates/" *application-\
22 root*))
23 
24 (defconfig :common
25   `(:databases ((:maindb :sqlite3 :database-name ":memory:"))))
26 
27 (defconfig |development|
28   '())
29 
30 (defconfig |production|
31   '())
32 
33 (defconfig |test|
34   '())
35 
36 (defun config (&optional key)
37   (envy:config #.(package-name *package*) key))
38 
39 (defun appenv ()
40   (uiop:getenv (config-env-var #.(package-name *package*))))
41 
42 (defun developmentp ()
43   (string= (appenv) "development"))
44 
45 (defun productionp ()
46   (string= (appenv) "production"))

First the line (setf (config-env-var) "APP_ENV") tells ENVY to switch configuration based on the APP_ENV environment variable.

Next the *application-root*, *static-directory* and *temlate-directory* variables are defined which contain our app, our static resources and our html templates respectively.

The macro defconfig is used to define configurations as the name suggests. A config has a name and a body. As you can see above the body itself is a plist of values. The :common name is special, because it is included in every other configuration. In this case it only defines that the database used by default is sqlite3 and it’s to be stored in memory, rather than create a file on disk.

The other defined configurations are |development|, |production| and |test|, which are all empty. The reason they are named with || is because that’s how common lisp symbols are written in lower case. Remember that a configuration is chosen based on the value of an environment variable. You could name your config production, but since lisp symbols are upcased by default, you’ll have to set your environment variable in upcase: export APP_ENV=PRODUCTION.

Finally the file defines a few utility functions: * config gives returns back the configuration, or if an optional key argument is given, it looks it up and returns it’s value. * appenv returns the value of the configured environment variable * developmentp and productionp simply tell you if you’re running in dev or prod respectively.

That’s it. We’ll take a closer look again at configuration when we set up a database for our project other than sqlite3, for now you get the idea.

Views

By default caveman apps use Djula for html templating. Djula is an implementation of the Django template engine. If you haven’t used a template language before it’s basically an ordinary text file with special tags in between which insert or transform data. For example this is a very simple Djula template:

1 {% if username %}
2 Hello {{username}}!
3 {% else %}
4 Please login!
5 {% endif %}

This template defines a variable called username, when we render our template we’ll pass it an argument with the value of username and it will use it to transform the text. For example it’s value is Pavel, well get the string “Hello Pavel”, but if it’s nil we’ll get the string “Please login!”. Although template languages aren’t specific to HTML, they are extremely useful for just that purpose and that’s mostly what we’ll be using them for in this book. We’ll go through the template language itself in a separate chapter later.

In a caveman app, the Djula templates are kept in the directory designated by *template-directory*, which in our case is fullstackwiki/templates/. A fresh project will have a few such templates already defined in that directory. Like the Django engine, Djula supports template inheritance, which means that means we can put common code in separate files and reuse it in other templates. Our project already has such a shared template defined in the directory fullstackwiki/templates/layout/. Let’s take a look at the file named default.html in that directory:

 1 <!DOCTYPE html>
 2 <html>
 3 <head>
 4   <meta charset="utf-8">
 5   <title>{% block title %}{% endblock %}</title>
 6   <link rel="stylesheet" type="text/css" media="screen" href="/css/main.css">
 7 </head>
 8 <body>
 9   {% block content %}{% endblock %}
10 </body>
11 </html>

As you can see, this is just an ordinary HTML document, but with Djula tags added. In this case the tags are two pairs of {% block $}/{% endblock %}. What this tag does is it allows other templates to insert data in these blocks when they extend the default.html file. In our case we’ve defined two named blocks: title and content. Let’s take a look at index.html, which extends base.html:

1 {% extends "layouts/default.html" %}
2 {% block title %}Welcome to Caveman2{% endblock %}
3 {% block content %}
4 <div id="main">
5   Welcome to <a href="http://8arrow.org/caveman/">Caveman2</a>!
6 </div>
7 {% endblock %}

The first line introduces the extends tag, we give it the base template as an argument. Now any time we use the block tag with the name of a block, anything in it will be inserted into the base template.

Other than that caveman also defines a errors/404.html page for us.

That’s it for now. We’ll take a look at more tags as we need them.

Routing

The main purpose of a web framework is to match urls to code that returns a response to the client. Let’s see how caveman does that. The important code is in fullstackwiki/src/web.lisp. Let’s look at some of the code in that file.

First up, caveman2 stores url routing rules in an object called an app, which is an instance of the <app> class. Here we create a subclass called <web> and instantiate it, store it in a variable called fullstackwiki.web:*web* and clear it’s routing rules:

19 (defclass <web> (<app>) ())
20 (defvar *web* (make-instance '<web>))
21 (clear-routing-rules *web*)

Now we can begin defining routes. A route is a mapping of url template and additional conditions to Common Lisp functions. Caveman2 gives us two ways to define them. The first is annotation syntax, and the other is the defroute macro. Notice on line 14 of the file we have the following code: (syntax:use-syntax :annot). This allows us to annotate functions using the @annotation syntax, this is how it might look like:

1 @route GET "/"
2 (defun index ()
3   (render #P"index.html"))

The first line is an annotation, followed by a function. This isn’t standard Common Lisp, but a reader syntax extension. Annotations are essentially functions that take a form and transform it in some way, in this case it creates a routing rule for the url template "/" and associates the function index as it’s handler. We won’t be using annotation syntax in this book, since it is equivalent to the defroute macro in functionality and I have a strong aesthetic preference, at least for my projects, you are free to choose annotations. Now let’s look at the equivalent code as a defroute as it was defined in fullstackwiki/src/web.lisp:

26 (defroute "/" ()
27   (render #P"index.html"))

This is essentially equivalent to the annotation code. It defines a handler for the "/" url. The body of the route simply renders the index.html template and returns it to the client. We’ll take a look at much more complicated examples of routes later in the book.

Finally at the bottom of the file is this interesting method:

32 (defmethod on-exception ((app <web>) (code (eql 404)))
33   (declare (ignore app))
34   (merge-pathnames #P"_errors/404.html"
35                    *template-directory*))

In case no route matches the url of the request, this method will handle it by sending back a 404 Not Found response.

A few words about Clack/Lack

TBW

Running the app

The way clack based apps are run is with the clack:clackup function. Other than a function or a component it can also take an app file that configures our application as an argument:

1 (clack:clackup #P"/path/to/app.lisp" args)

This is useful since we might want to add middleware and other options to our app and we need a place where we can do that. Also the file can be used to start our application from the command line. We’ll see how that might be done in later chapters. In our case, caveman defined one such file in the root of our project called app.lisp, let’s take a look:

 1 (ql:quickload :fullstackwiki)
 2 
 3 (defpackage fullstackwiki.app
 4   (:use :cl)
 5   (:import-from :lack.builder
 6                 :builder)
 7   (:import-from :ppcre
 8                 :scan
 9                 :regex-replace)
10   (:import-from :fullstackwiki.web
11                 :*web*)
12   (:import-from :fullstackwiki.config
13                 :config
14                 :productionp
15                 :*static-directory*))
16 (in-package :fullstackwiki.app)
17 
18 (builder
19  (:static
20   :path (lambda (path)
21           (if (ppcre:scan "^(?:/images/|/css/|/js/|/robot\\.txt$|/favicon\\.ico$\
22 )" path)
23               path
24               nil))
25   :root *static-directory*)
26  (if (productionp)
27      nil
28      :accesslog)
29  (if (getf (config) :error-log)
30      `(:backtrace
31        :output ,(getf (config) :error-log))
32      nil)
33  :session
34  (if (productionp)
35      nil
36      (lambda (app)
37        (lambda (env)
38          (let ((datafly:*trace-sql* t))
39            (funcall app env)))))
40  *web*)

First it uses quicklisp to load our project, since the file itself isn’t part of it. It then imports our app and its configuration and uses lack:builder to build a lack component. It’s setting up static file handling, session and other middleware based on our configuration and applying them to our app. When you call clack:clackup with that file as an argument, it gets evaluated and the result of the lack:builder form is run as our app.

For actually running the app, caveman generated a few convenience functions in fullstackwiki/src/main.lisp let’s take a look:

 1 (in-package :cl-user)
 2 (defpackage fullstackwiki
 3   (:use :cl)
 4   (:import-from :fullstackwiki.config
 5                 :config)
 6   (:import-from :clack
 7                 :clackup)
 8   (:export :start
 9            :stop))
10 (in-package :fullstackwiki)
11 
12 (defvar *appfile-path*
13   (asdf:system-relative-pathname :fullstackwiki #P"app.lisp"))
14 
15 (defvar *handler* nil)
16 
17 (defun start (&rest args &key server port debug &allow-other-keys)
18   (declare (ignore server port debug))
19   (when *handler*
20     (restart-case (error "Server is already running.")
21       (restart-server ()
22         :report "Restart the server"
23         (stop))))
24   (setf *handler*
25         (apply #'clackup *appfile-path* args)))
26 
27 (defun stop ()
28   (prog1
29       (clack:stop *handler*)
30     (setf *handler* nil)))

This file defines our main package fullstackwiki. It also defines our app file and a pair of start/stop functions. As you can see, start checks if the app is running, and if it is, it offers to restart it, otherwise it calls clack:clackup with the file and sets the result to the *handler* variable. Stop is pretty self explanatory.

Conclusion

TBW

Chapter 5: Datafly and SXQL

In this chapter we’ll take a look at Caveman2’s database access capability and define a simple model for our wiki app.

Database access

The default Caveman2 data layer uses a library called CL-DBI, which provides drivers for most popular RDBMSs out there, including PostgreSQL, SQLite and MySQL. Instead of using it directly though, most of the time we’ll be using another library called datafly to access our data. This library provides a thin wrapper over CL-DBI and gives us 3 main functions: EXECUTE executes an SQL statement, RETRIEVE-ONE and RETRIEVE-ALL are used to fetch data, again with an SQL statement. Instead of writing SQL by hand, we’ll be using a lispy SQL dsl called SXQL, which allows us to write queries in a lispy syntax. You can use pretty much any database libraries you wish with Caveman2, and later on we’ll take a look at some alternatives, but for now the default choices will work just fine.

Before we look at some examples, let’s first set up a database. This book will use PostgreSQL, so let’s install it, on debian-like Linux distros this will be

1  $ sudo apt-get install postgresql

I also advise you to install the excellent pgadmin3 package, which provides a nice GUI for database management:

1  $ sudo apt-get install pgadmin3

Next, the first thing we must do us set up a postgreSQL database and a user. We must do this with the postgres unix account, type the following to log in as the postgres user

1  $ sudo su - postgres

Now start the PostgreSQL CLI client psql:

1  $ psql

You will notice the new prompt:

1 postgres=# 

Let’s create a new user, type the following:

1 CREATE USER fswiki WITH PASSWORD 'fswikipass';

Now we must create a new database:

1 CREATE DATABASE fswikidb;

And finally, we must grant our user all the privileges to use the new database:

1 GRANT ALL PRIVILEGES ON DATABASE fswikidb TO fswiki;

That’s it, type \q to exist psql, and type exit to logout of the postgres account.

Now, let’s try it out. Open the src/config.lisp, and edit the (defconfig :common ...) section:

22 (defconfig :common
23     `(:databases ((:maindb :postgres
24                            :database-name "fswikidb"
25                            :username "fswiki"
26                            :password "fswikipass"))))

Now let’s take a look at the src/db.lisp file:

 1 (in-package :cl-user)
 2 (defpackage fullstackwiki.db
 3   (:use :cl)
 4   (:import-from :fullstackwiki.config
 5                 :config)
 6   (:import-from :datafly
 7                 :*connection*
 8                 :connect-cached)
 9   (:export :connection-settings
10            :db
11            :with-connection))
12 (in-package :fullstackwiki.db)
13 
14 (defun connection-settings (&optional (db :maindb))
15   (cdr (assoc db (config :databases))))
16 
17 (defun db (&optional (db :maindb))
18   (apply #'connect-cached (connection-settings db)))
19 
20 (defmacro with-connection (conn &body body)
21   `(let ((*connection* ,conn))
22      ,@body))

What we see here is a few utility functions defined for us. connection-settings pulls the current db configuration, db establishes a new connection, or pulles a an already existing one from a chache, and the macro with-connection simply binds it to the *connection* variable so that datafly functions know which connection to use when querying the database, and enter into this package

Now let’s start a fresh repl and load our project to look at some examples:

1  * (ql:quickload "fullstackwiki")
2  * (in-package :fullstackwiki.db)

For all our code we should use with-connection, but it’s a bit cumbersome to write it all the time in the repl, so let’s set a top-level connection:

1  * (apply #'datafly:connect-toplevel (connection-settings :maindb))

Now we can call datafly functions without wrapping them in calls to with-connection. Let’s create a simple table:

1  * (datafly:execute
2      (sxql:create-table (:test-table if-not-exists t)
3          ((id :type 'serial :primary-key t)
4           (name :type 'text))))

This creates a new table called test-table if it doesn’t already exist. Calling this code again will not do anything, since it already exists. The table has two fields, id, which is of type serial, a postgreSQL specific way to get auto-incremented integers, this field is our primary key, and name, which is a text field. The syntax for sxql:create-table is in general fairly simple:

1 (sxql:create-table <table-name> (<fields>*) <options>*)

Fairly similar to standard SQL, but in lisp syntax. The field definitions have various options, we’ve seen type, and primary-key, but others like not-null and unique are also supported.

Now let’s insert a row into the table:

1  * (datafly:execute
2      (sxql:insert-into :test-table
3        (sxql:set= :name "Hello World")))

Similar to standard SQL insert into statement, sxql:insert-into takes a table same and a list of clauses, in this case only one sxql:set= clause. We set the name to “Hello World”, the id is auto-incremented, so we don’t have to specify it.

Retrieving data works in a similar way:

1  * (datafly:retrieve-all (sxql:select :* (sxql:from :test-table)))
2 => ((:ID 1 :NAME "Hello World"))

Again, the statement sxql:select works like in regular SQL, It expects a list of fielsds and a list of clauses, in this case we have a single field :* equivalent to the SQL * operator, and a single from clause specifying the table. Instead of datafly:execute we use datafly:retrieve-all since we expect to get back results instead of just executing a statement on the server. The result is a list of plists, in this case just one, since we only have one record in the table. The function datafly:retrieve-one works in a similar way, but returns only a single result:

1  * (datafly:retrieve-one (sxql:select :* (sxql:from :test-table)))
2 => (:ID 1 :NAME "Hello World")

We’ll see more examples and I’ll go into more depth in a moment, this is just to get your feet wet. Now that we’ve seen the basics, let’s drop this table and disconnect, so we can get to some real work in the next section:

1  * (datafly:execute (sxql:drop-table :test-table))
2 
3  * (datafly:disconnect-toplevel)

The Wiki Model

Our wiki app has some pretty basic requirements. First we need to be able to create users and log them in. Next users have to be able to create and edit wiki pages. We also want pages to have different versions, so they have a history we can go back to and see what was edited. In this chapter we’ll see a very simple model that can do what we want. The model only has two tables, user and page, and a few functions to create and query the data. The first thing we’ll need to do though is update our fullstackwiki.asd file. We’ll need to add two lines, the first is adding a dependency to cl-pass a library for hashing passwords, since we don’t want to store them in plain text. The other one is to add a src/model.lisp file where our code will be. Here is how the updated asd file looks like. The lines we’ve changed are number 27 and number 34. Notice that model depends on db:

 1 (in-package :cl-user)
 2 (defpackage fullstackwiki-asd
 3   (:use :cl :asdf))
 4 (in-package :fullstackwiki-asd)
 5 
 6 (defsystem fullstackwiki
 7   :version "0.1"
 8   :author "Pavel"
 9   :license "MIT"
10   :depends-on (:clack
11                :lack
12                :caveman2
13                :envy
14                :cl-ppcre
15                :uiop
16 
17                ;; for @route annotation
18                :cl-syntax-annot
19 
20                ;; HTML Template
21                :djula
22 
23                ;; for DB
24                :datafly
25                :sxql
26                ;; Password hashing
27                :cl-pass)
28   :components ((:module "src"
29                 :components
30                 ((:file "main" :depends-on ("config" "view" "db"))
31                  (:file "web" :depends-on ("view"))
32                  (:file "view" :depends-on ("config"))
33                  (:file "db" :depends-on ("config"))
34                  (:file "model" :depends-on ("db")) ;; model
35                  (:file "config"))))
36   :description ""
37   :in-order-to ((test-op (load-op fullstackwiki-test))))

Another bit of houskeeping we must do is extend src/db.lisp with an additional macro. Our model will need to be able to execute SQL statements inside a database transaction, for this we’ll need to implement a with-transaction macro, add this to the bottom of the file:

19 (defmacro with-transaction (conn &body body)
20   `(let ((*connection* ,conn))
21      (cl-dbi:with-transaction *connection*
22        ,@body)))

This macro is similar to with-connection, but we additionally make a call to the low-level CL-DBI:with-transaction macro to make sure everything in the body is executed in the proper order and we don’t break our database. We also have to export it from the defpackage form at top of the file:

 1 (in-package :cl-user)
 2 (defpackage fullstackwiki.db
 3   (:use :cl)
 4   (:import-from :fullstackwiki.config
 5                 :config)
 6   (:import-from :datafly
 7                 :*connection*
 8                 :connect-cached)
 9   (:export :connection-settings
10            :db
11            :with-connection
12            :with-transaction)) ;; new macro

Now we’re almost all set. Let’s create a new file called src/model.lisp and create a package for it:

 1 ;;;; model.lisp
 2 
 3 (in-package :cl-user)
 4 
 5 (defpackage fullstackwiki.model
 6   (:use :cl :sxql)
 7   (:import-from :fullstackwiki.db
 8                 :db
 9                 :with-connection
10                 :with-transaction) 
11   (:import-from :datafly
12                 :execute
13                 :retrieve-all
14                 :retrieve-one)
15   (:export :create-tables
16            :find-user
17            :add-user
18            :authenticate-user
19            :add-page
20            :get-latest-page
21            :get-latest-pages-by-user
22            :get-sorted-pages
23            :count-pages
24            :nth-page-revision))
25 
26 (in-package :fullstackwiki.model)

This sets up a new package called fullstackwiki.model for us, and it imports all the symbols we’ll need from sxql, fullstackwiki.db and datafly. It also exports the package’s public API. I’ll explain what all these functions do in the next sections.

User model

Our user model is very simple, It consists of a user table, let’s write a function that creates it for us:

30 (defun create-user-table ()
31   "Create user table if it doesn't exist yet."
32   (with-connection (db)
33     (execute
34      (create-table (:user :if-not-exists t)
35          ((id :type 'serial :primary-key t)
36           (username :type 'text :not-null t :unique t)
37           (email :type 'text :not-null t :unique t)
38           (password :type 'text :not-null t))))))

The user table has four fields, an id, username, email and password. This is pretty self-explanatory. We make all fields :not-null and the email and username fields have to be :unique. Now a function to register a new user:

40 (defun add-user (username email password)
41   "add user record to database."
42   (with-connection (db)
43     (execute
44      (insert-into :user
45        (set= :username username
46              :email email
47              :password (cl-pass:hash password))))))

Again, nothing we haven’t seen before, we insert a new user row, but the password is hashed with the cl-pass:hash function. This is how it might look like in the repl:

1 * (cl-pass:hash "hello")
2 => "PBKDF2$SHA256:20000$307629b11dd0f4df0c55c2b78f2f97a8$5213c150869544fc0a96e95\
3 d10d4a3157cd398ddeb4b3f128ebcaa90e6e50482"
4 * (cl-pass:check-password "hello" *)
5 => T
6 * (cl-pass:check-password "wrong password" **)
7 => NIL

Next, we need a way to retrieve a user:

49 (defun find-user-by-username (username)
50   "lookup user record by username."
51   (with-connection (db)
52     (retrieve-one
53      (select :*
54        (from :user)
55        (where (:= :username username))))))

This is the first time we see a clause other that from in an SXQL statement, in this case we specify what the username must be with the where clause. Let’s do the same with the email field:

57 (defun find-user-by-email (email)
58   "lookup user record by email."
59   (with-connection (db)
60     (retrieve-one
61      (select :* (from :user)
62              (where (:= :email email))))))

Now, since we’ll want the user to be able to log in either with a username or with an email address, let’s combine the two functions:

64 (defun find-user (username-or-email)
65   "lookup user record by username or email."
66   (or (find-user-by-username username-or-email)
67       (find-user-by-email username-or-email)))

And finally, we must check to see if the user credentials are valid:

69 (defun authenticate-user (username-or-email password)
70   "Lookup user record and validate password. Returns two values:
71    1st value, was password correct T or NIL
72    2nd value, was user found, T or NIL
73 Example:
74    (VALUES NIL NIL) -> user not found
75    (VALUES NIL T) -> user found, but wrong password
76    (VALUES T T) -> password correct"
77   (let ((password-hash (getf (find-user username-or-email) :password)))
78     (if password-hash 
79         (values (cl-pass:check-password password password-hash) t)
80         (values nil nil))))

This is a bit more advanced, so let’s look a bit closer. First we find the user and bind the password field to the local variable password-hash, if the value is not nil, such a user exists and we can check it’s password, if not, no such user exists and the login fails. What’s a bit more interesting here is the use of multiple return values. If you’re new to lisp, this is one of the more unique features of the language. We have the ability to return more than one value. By default only the first is used, but since there is more than one way for this function to fail(wrong password and no such user) we want to add a bit of extra information in case we might need it to display an error message for example. Basically the first value tells you if the login succeeded or it failed, and the second value tells you if it failed because of a wrong password or a non-existent user.

And that’s it, that’s our user model. We have everything we need for now. Let’s move on.

Page model

The basic idea of the page model is fairly simple. A page is identified by it’s title, but it can have different versions. Only one of those versions is the latest though, and when a user edits a page, a new record is inserted into the database and is designated the latest version. This is a very simple model that will do just fine for our purposes. Let’s look at the table definition:

84 (defun create-page-table ()
85   "Create page table if it doesn't exist yet."
86   (with-connection (db)
87     (execute
88      (create-table (:page if-not-exists t)
89          ((id :type 'serial :primary-key t)
90           (title :type 'text :not-null t)
91           (author-id :type 'integer :not-null t)
92           (content :type 'text :not-null t)
93           (date :type 'timestamp :not-null t)
94           (latest :type 'boolean :not-null t)) ;; is this the latest version of \
95 the page
96        (foreign-key '(:author-id) :references '(:user :id))))))

The fields id, title and content should be obvious by now, the author-id field is a foreign key to the user id, so it must be an integer. The date is a timestamp, I’ll explain that in a bit. The last field latest is a bit more interesting. It’s a binary flag that designates if the page is the last revision. This is to make lookup a bit easier, since most of the time we only need to see the latest version of an article. I’ll explain that in a bit more detail in a bit as well. The last line is a table option that specifies that author-id is a foreign-key to the user table and the field id. now let’s add a new page:

 97 (defun add-page (title author content)
 98   "Add page to database. Mark it as \"latest: true\", mark previous \"latest: fa\
 99 lse\"."
100   (with-transaction (db)
101     (execute
102      (update :page
103        (set= :latest "false")
104        (where (:and
105                (:= :title title)
106                (:= :latest "true")))))
107     (execute
108      (insert-into :page
109        (set= :title title
110              :author-id (getf (find-user author) :id)
111              :content content
112              :date (local-time:now)
113              :latest "true")))))

This is where the with-transaction macro comes into play. Since we can only have one latest version of a page, we need some way to simultaneously set the previous value to "false", and insert a new “latest” version, without worrying that we’ll mess up the database and end up with two new versions, or no new versions. This is done with an SQL transaction, it ensures that two queries will be executed as one so we don’t have to worry about order. The first execute statement does an SQL update on the previous article with the same title and a latest value of "true", and sets it to "false". Next we insert a new row in the table, with a latest value of “true”. We’ll see how that’s useful in the next function. Another thing to note is that the date field is given a value of (local-time:now), this function creates a new timestamp for us. I’ll go into more detail on how to deal with dates and time in Common Lisp in another chapter.

Let’s look at the rest of the functions, next up we need to find the latest version of an article:

114 (defun get-latest-page (title)
115   "Get the latest version of a page by TITLE."
116   (with-connection (db)
117     (retrieve-one
118      (select :*
119        (from :page)
120        (where (:and (:= :title title)
121                     (:= :latest "true")))))))

The query is relatively simple, we select the page with the title we want, and we ask for the latest version. This is much simpler that the alternative approach of sorting by the date and selecting the first result.

Now for a more complex query, this one has an implicit join:

123 (defun get-latest-pages-by-user (username)
124   "Get the latest versions of all pages by USERNAME."
125   (with-connection (db)
126     (retrieve-all
127      (select (:page.id :title :username :content :date :latest)
128        (from :page :user)
129        (where (:and (:= :user.id :author-id)
130                     (:= :user.username username)
131                     (:= :latest "true")))))))

Here we see a few features of SXQL we haven’t seen before, namely selecting specific fields(instead of just using :*) and qualifying field names with their tables, for example, because user and page both have fields named id we need a way to distinguish between them, so we call them :page.id and :user.id respectively. This works automatically. We also select from more than one table. What we’re doing here is we’re telling PostgreSQL to join the tables together, and select those fields where the author-id and the user.id are equal for the user named with the variable username and we also want the latest versions of the articles. Notice how much more complex this query would have been if we had to sort all the articles and get the latest one ourselves. This is actually what the next function does, but in this case we want to get all the versions of a single article:

133 (defun get-sorted-pages (title)
134   "Get all versions of a page by TITLE, sorted by it's timestamp in descending o\
135 rder, newest first."
136   (with-connection (db)
137     (retrieve-all
138      (select :*
139        (from :page)
140        (where (:= :title title))
141        (order-by (:desc :date))))))

This is a fairly simple query, we select all the pages with the given title, and we order them by the date. The latest article will be of course the first result. If we want to know how many versions a page has without retrieving them all, we can use a query like this:

142 (defun count-pages (title)
143   "Count the number of versions a page has."
144   (with-connection (db)
145     (getf (retrieve-one
146            (select (:title (:count :*))
147              (from :page)
148              (where (:= :title title))
149              (group-by :title)))
150           :count)))

This one is a bit more complex. What we’re doing here is were calling the COUNT() sql function on all rows and we’re grouping them by title. Since there will be only one title, there will be one result returned. The result will look something like this: (:title "foo" :count 3) if the article named foo has 3 versions. We simply getf the count from that result and return it. Next up, if we want to get a specific revision of a page, we can now do this pretty easily:

152 (defun nth-page-revision (title n)
153   "Get the nth version of a page, sorted by its DATE."
154   (nth n (get-sorted-pages title)))

We simply get the nth element of the result returned by get-sorted-pages. The (nth-page-revision "foo" 0) will give us the latest version of course, even if that’s a bit inefficient.

Finally we want a convenience function co create all of our tables for us:

158 (defun create-tables ()
159   "Create all tables, if they don't exist already."
160   (create-user-table)
161   (create-page-table))

And that’s our database model. Now let’s play around with it.

Playing with the model in the repl

Go to the repl, load our project, go into the fullstackwiki.model package and create the tables:

1 * (ql:quickload "fullstackwiki")
2 
3 * (in-package :fullstackwiki.model)
4 
5 * (create-tables)

Now let’s add a user and play around with it:

 1 * (add-user "fullstackwiki" "[email protected]" "this-is-a-passwor\
 2 d")
 3 
 4 * (find-user "fullstackwiki")
 5 => (:ID 1 
 6     :USERNAME "fullstackwiki" 
 7     :EMAIL "[email protected]"
 8     :PASSWORD "PBKDF2$SHA256:20000$e57a9bef3bb97ed3aff8d1c9d69d4f9f$bed1f429fc76\
 9 ed95d8f6d729d10897ce4ea7a1b93cea93a6e4690af7305ccd23")
10 
11 * (find-user "[email protected]")
12 => (:ID 1 
13     :USERNAME "fullstackwiki" 
14     :EMAIL "[email protected]"
15     :PASSWORD "PBKDF2$SHA256:20000$e57a9bef3bb97ed3aff8d1c9d69d4f9f$bed1f429fc76\
16 ed95d8f6d729d10897ce4ea7a1b93cea93a6e4690af7305ccd23")

Next up we’ll try to authenticate it:

1 * (authenticate-user "fullstackwiki" "this-is-a-password")
2 => T
3 => T

Notice the two return values. Same works with the email:

1 * (authenticate-user "[email protected]" "this-is-a-password")
2 => T
3 => T

Now let’s fail a few times:

1 * (authenticate-user "No Such User" "this-is-a-password")
2 => NIL
3 => NIL

notice the second value is also nil, so no user was found.

1 * (authenticate-user "fullstackwiki" "wrong password")
2 => NIL
3 => T

Here the first value is nil, but the second value is T, because the username is correct, but the password is wrong.

Now let’s create a few pages:

1 * (add-page "page1" "fullstackwiki" "This is the first version")
2 
3 * (add-page "page1" "fullstackwiki" "This is the second versions")
4 
5 * (add-page "page1" "fullstackwiki" "This is the third version")
6 
7 * (add-page "page 2" "fullstackwiki" "This is the first version of page 2")
8 
9 * (add-page "page 2" "fullstackwiki" "This is the second version of page 2")

Now let’s query them a bit:

 1 * (get-latest-page "page1")
 2 => (:ID 3 
 3     :TITLE "page1" 
 4     :AUTHOR-ID 1 
 5     :CONTENT "This is the third version"
 6     :DATE 3673640374 
 7     :LATEST T)
 8 
 9 * (get-latest-page "page 2")
10 => (:ID 5 
11     :TITLE "page 2" 
12     :AUTHOR-ID 1 
13     :CONTENT "This is the second version of page 2" 
14     :DATE 3673640434 
15     :LATEST T)

Now let’s get all the pages by our user:

 1 * (get-latest-pages-by-user "fullstackwiki")
 2 =>((:ID 3 
 3     :TITLE "page1" 
 4     :USERNAME "fullstackwiki" 
 5     :CONTENT "This is the third version" 
 6     :DATE 3673640374 
 7     :LATEST T)
 8    (:ID 5 
 9     :TITLE "page 2" 
10     :USERNAME "fullstackwiki" 
11     :CONTENT "This is the second version of page 2" 
12     :DATE 3673640434 
13     :LATEST T))

And all versions of page1:

 1 * (get-sorted-pages "page1")
 2 => ((:ID 3 
 3      :TITLE "page1" 
 4      :AUTHOR-ID 1 
 5      :CONTENT "This is the third version" 
 6      :DATE 3673640374 
 7      :LATEST T)
 8     (:ID 2 
 9      :TITLE "page1" 
10      :AUTHOR-ID 1 
11      :CONTENT "This is the second versions" 
12      :DATE 3673640356 
13      :LATEST NIL)
14     (:ID 1 
15      :TITLE "page1" 
16      :AUTHOR-ID 1 
17      :CONTENT "This is the first version" 
18      :DATE 3673640338 
19      :LATEST NIL))

Notice only the first one is the latest. Let’s get the oldest one:

1 * (nth-page-revision "page1" 2)
2 => (:ID 1 
3     :TITLE "page1" 
4     :AUTHOR-ID 1 
5     :CONTENT "This is the first version" 
6     :DATE 3673640338 
7     :LATEST NIL)

And now let’s see how many are there:

1 * (count-pages "page1")
2 => 3
3 * (count-pages "page 2")
4 => 2

And that’s it. This is our model. In this chapter we learned how to use a relational database with Common Lisp and implemented this simple schema so now we can continue with building our app.