Harden shell scripts with improved robustness & verbosity (#233)

Merges manubot/rootstock#233 Hat tip to Michael Hoffman (https://hoffmanlab.org/) for feedback. * Apply shell best practices References: https://kvz.io/blog/2013/11/21/bash-best-practices/ https://www.davidpashley.com/articles/writing-robust-shell-scripts/ manubot/rootstock#233 (comment) * Specify bash (not just sh). Other shellcheck suggestions Properly use command substitution and parameter expansion in the appropriate places. * echo diagnostic messages to stderr Previously echo defaulted to stdout https://stackoverflow.com/a/41766832/4651668 Placement of stderr redirect to: "echo >&2" * Update bash options: xtrace specified by calling command * Quote variable names * deploy.sh: use Travis env vars for CI build URLs These environment variables were added to Travis in late 2018 travis-ci/travis-ci#8935 (comment)
greenelab · Jun 27, 2019 · ddb0288 · ddb0288
1 parent 2fd84f1
commit ddb0288
Show file tree

Hide file tree

Showing 6 changed files with 61 additions and 45 deletions.
diff --git a/.travis.yml b/.travis.yml
@@ -20,11 +20,11 @@ install:
   - conda list --name manubot
   - conda activate manubot
 script:
-  - sh build/build.sh
+  - bash build/build.sh
 cache:
   directories:
     - ci/cache
 after_success:
   - test $TRAVIS_BRANCH = "master" &&
     test $TRAVIS_EVENT_TYPE = "push" &&
-    sh ci/deploy.sh
+    bash -o xtrace ci/deploy.sh
diff --git a/README.md b/README.md
@@ -48,7 +48,7 @@ Then, you can build the manuscript on POSIX systems by running the following com
 conda activate manubot
 
 # Build the manuscript, saving outputs to the output directory
-sh build/build.sh
+bash build/build.sh
 
 # At this point, the HTML & PDF outputs will have been created. The remaining
 # commands are for serving the webpage to view the HTML manuscript locally.
@@ -65,7 +65,7 @@ Sometimes it's helpful to monitor the content directory and automatically rebuil
 The following command, while running, will trigger both the `build.sh` and `webpage.py` scripts upon content changes:
 
 ```sh
-sh build/autobuild.sh
+bash build/autobuild.sh
 ```
 
 ### Continuous Integration

diff --git a/build/README.md b/build/README.md
@@ -4,10 +4,10 @@
 `sh build/build.sh` should be executed from the root directory of the repository.
 By default, `build.sh` creates HTML and PDF outputs.
 However, setting the `BUILD_PDF` environment variable to `false` will suppress PDF output.
-For example, run local builds using the command `BUILD_PDF=false sh build/build.sh`.
+For example, run local builds using the command `BUILD_PDF=false bash build/build.sh`.
 
 To build a DOCX file of the manuscript, set the `BUILD_DOCX` environment variable to `true`.
-For example, use the command `BUILD_DOCX=true sh build/build.sh`.
+For example, use the command `BUILD_DOCX=true bash build/build.sh`.
 To export DOCX for all Travis builds, set a [Travis environment variable](https://docs.travis-ci.com/user/environment-variables/#Defining-Variables-in-Repository-Settings).
 Currently, equation numbers via `pandoc-eqnos` are not supported for DOCX output.
 There is varying support for embedding images in DOCX output.

diff --git a/build/autobuild.sh b/build/autobuild.sh
@@ -1,6 +1,9 @@
-# Automatically rebuild mansucript outputs and the webpage when content changes
-# Depends on watchdog https://github.com/gorakhargosh/watchdog
+#!/usr/bin/env bash
+
+## autobuild.sh: automatically rebuild mansucript outputs and the webpage when content changes
+## Depends on watchdog https://github.com/gorakhargosh/watchdog
+
 watchmedo shell-command \
   --wait \
-  --command='sh build/build.sh && python build/webpage.py' \
+  --command='bash build/build.sh && python build/webpage.py' \
   content
diff --git a/build/build.sh b/build/build.sh
@@ -1,12 +1,18 @@
-set -o errexit
+#!/usr/bin/env bash
+
+## build.sh: compile manuscript outputs from content using Manubot and Pandoc
+
+set -o errexit \
+    -o nounset \
+    -o pipefail
 
 # Set timezone used by Python for setting the manuscript's date
 export TZ=Etc/UTC
 # Default Python to read/write text files using UTF-8 encoding
 export LC_ALL=en_US.UTF-8
 
 # Generate reference information
-echo "Retrieving and processing reference metadata"
+echo >&2 "Retrieving and processing reference metadata"
 manubot process \
   --content-directory=content \
   --output-directory=output \
@@ -23,15 +29,15 @@ mkdir -p output
 
 # Create HTML output
 # http://pandoc.org/MANUAL.html
-echo "Exporting HTML manuscript"
+echo >&2 "Exporting HTML manuscript"
 pandoc --verbose \
   --from=markdown \
   --to=html5 \
   --filter=pandoc-fignos \
   --filter=pandoc-eqnos \
   --filter=pandoc-tablenos \
-  --bibliography=$BIBLIOGRAPHY_PATH \
-  --csl=$CSL_PATH \
+  --bibliography="$BIBLIOGRAPHY_PATH" \
+  --csl="$CSL_PATH" \
   --metadata link-citations=true \
   --include-after-body=build/themes/default.html \
   --include-after-body=build/plugins/table-scroll.html \
@@ -48,14 +54,14 @@ pandoc --verbose \
   --include-after-body=build/plugins/hypothesis.html \
   --include-after-body=build/plugins/analytics.html \
   --output=output/manuscript.html \
-  $INPUT_PATH
+  "$INPUT_PATH"
 
 # Return null if docker command is missing, otherwise return path to docker
 DOCKER_EXISTS="$(command -v docker || true)"
 
 # Create PDF output (unless BUILD_PDF environment variable equals "false")
-if [ "$BUILD_PDF" != "false" ] && [ -z "$DOCKER_EXISTS" ]; then
-  echo "Exporting PDF manuscript using WeasyPrint"
+if [ "${BUILD_PDF:-}" != "false" ] && [ -z "$DOCKER_EXISTS" ]; then
+  echo >&2 "Exporting PDF manuscript using WeasyPrint"
   if [ -L images ]; then rm images; fi  # if images is a symlink, remove it
   ln -s content/images
   pandoc \
@@ -66,25 +72,25 @@ if [ "$BUILD_PDF" != "false" ] && [ -z "$DOCKER_EXISTS" ]; then
     --filter=pandoc-fignos \
     --filter=pandoc-eqnos \
     --filter=pandoc-tablenos \
-    --bibliography=$BIBLIOGRAPHY_PATH \
-    --csl=$CSL_PATH \
+    --bibliography="$BIBLIOGRAPHY_PATH" \
+    --csl="$CSL_PATH" \
     --metadata link-citations=true \
     --webtex=https://latex.codecogs.com/svg.latex? \
     --include-after-body=build/themes/default.html \
     --output=output/manuscript.pdf \
-    $INPUT_PATH
+    "$INPUT_PATH"
   rm images
 fi
 
 # Create PDF output (unless BUILD_PDF environment variable equals "false")
-if [ "$BUILD_PDF" != "false" ] && [ -n "$DOCKER_EXISTS" ]; then
-  echo "Exporting PDF manuscript using Docker + Athena"
+if [ "${BUILD_PDF:-}" != "false" ] && [ -n "$DOCKER_EXISTS" ]; then
+  echo >&2 "Exporting PDF manuscript using Docker + Athena"
   if [ -d output/images ]; then rm -rf output/images; fi  # if images is a directory, remove it
   cp -R -L content/images output/
   docker run \
     --rm \
     --shm-size=1g \
-    --volume=`pwd`/output:/converted/ \
+    --volume="$(pwd)/output:/converted/" \
     --security-opt=seccomp:unconfined \
     arachnysdocker/athenapdf:2.16.0 \
     athenapdf \
@@ -94,21 +100,21 @@ if [ "$BUILD_PDF" != "false" ] && [ -n "$DOCKER_EXISTS" ]; then
 fi
 
 # Create DOCX output (if BUILD_DOCX environment variable equals "true")
-if [ "$BUILD_DOCX" = "true" ]; then
-  echo "Exporting Word Docx manuscript"
+if [ "${BUILD_DOCX:-}" = "true" ]; then
+  echo >&2 "Exporting Word Docx manuscript"
   pandoc --verbose \
     --from=markdown \
     --to=docx \
     --filter=pandoc-fignos \
     --filter=pandoc-eqnos \
     --filter=pandoc-tablenos \
-    --bibliography=$BIBLIOGRAPHY_PATH \
-    --csl=$CSL_PATH \
+    --bibliography="$BIBLIOGRAPHY_PATH" \
+    --csl="$CSL_PATH" \
     --metadata link-citations=true \
     --reference-doc=build/themes/default.docx \
     --resource-path=.:content \
     --output=output/manuscript.docx \
-    $INPUT_PATH
+    "$INPUT_PATH"
 fi
 
-echo "Build complete"
+echo >&2 "Build complete"
diff --git a/ci/deploy.sh b/ci/deploy.sh
@@ -1,26 +1,33 @@
-# Exit on errors
-set -o errexit
+#!/usr/bin/env bash
+
+## deploy.sh: run during a Travis CI build to deploy manuscript outputs to the output and gh-pages branches on GitHub.
+
+# Set options for extra caution & debugging
+set -o errexit \
+    -o nounset \
+    -o pipefail
 
 # Add commit hash to the README
-export OWNER_NAME=`dirname $TRAVIS_REPO_SLUG`
-export REPO_NAME=`basename $TRAVIS_REPO_SLUG`
+OWNER_NAME="$(dirname "$TRAVIS_REPO_SLUG")"
+REPO_NAME="$(basename "$TRAVIS_REPO_SLUG")"
+export OWNER_NAME REPO_NAME
 envsubst < webpage/README.md > webpage/README-complete.md
 mv webpage/README-complete.md webpage/README.md
 
 # Configure git
 git config --global push.default simple
-git config --global user.email `git log --max-count=1 --format='%ae'`
-git config --global user.name "`git log --max-count=1 --format='%an'`"
-git checkout $TRAVIS_BRANCH
-git remote set-url origin [email protected]:$TRAVIS_REPO_SLUG.git
+git config --global user.email "$(git log --max-count=1 --format='%ae')"
+git config --global user.name "$(git log --max-count=1 --format='%an')"
+git checkout "$TRAVIS_BRANCH"
+git remote set-url origin "[email protected]:$TRAVIS_REPO_SLUG.git"
 
 # Decrypt and add SSH key
 openssl aes-256-cbc \
   -K $encrypted_9befd6eddffe_key \
   -iv $encrypted_9befd6eddffe_iv \
   -in ci/deploy.key.enc \
   -out ci/deploy.key -d
-eval `ssh-agent -s`
+eval "$(ssh-agent -s)"
 chmod 600 ci/deploy.key
 ssh-add ci/deploy.key
 
@@ -33,24 +40,24 @@ git fetch origin gh-pages:gh-pages output:output
 python build/webpage.py \
   --no-ots-cache \
   --checkout=gh-pages \
-  --version=$TRAVIS_COMMIT
+  --version="$TRAVIS_COMMIT"
 
 # Generate OpenTimestamps
-ots stamp webpage/v/$TRAVIS_COMMIT/index.html
-if [ "$BUILD_PDF" != "false" ]; then
-  ots stamp webpage/v/$TRAVIS_COMMIT/manuscript.pdf
+ots stamp "webpage/v/$TRAVIS_COMMIT/index.html"
+if [ "${BUILD_PDF:-}" != "false" ]; then
+  ots stamp "webpage/v/$TRAVIS_COMMIT/manuscript.pdf"
 fi
 
 # Commit message
 MESSAGE="\
-`git log --max-count=1 --format='%s'`
+$(git log --max-count=1 --format='%s')
 
 This build is based on
 https://github.com/$TRAVIS_REPO_SLUG/commit/$TRAVIS_COMMIT.
 
 This commit was created by the following Travis CI build and job:
-https://travis-ci.com/$TRAVIS_REPO_SLUG/builds/$TRAVIS_BUILD_ID
-https://travis-ci.com/$TRAVIS_REPO_SLUG/jobs/$TRAVIS_JOB_ID
+$TRAVIS_BUILD_WEB_URL
+$TRAVIS_JOB_WEB_URL
 
 [ci skip]