Update for expanded manubot package (#48)

* Update for expanded manubot package * Create distinct caching directory * Deployment: references branch now output * Rewrite documentation * Rename CONTRIBUTING.md to USAGE.md Refs manubot/manubot#2
greenelab · Aug 9, 2017 · 084e30d · 084e30d
1 parent 64f3ddb
commit 084e30d
Show file tree

Hide file tree

Showing 22 changed files with 233 additions and 568 deletions.
diff --git a/.gitignore b/.gitignore
@@ -1,12 +1,11 @@
 # Generated manuscript output files
-output/index.html
-output/manuscript.pdf
-output/manuscript.docx
-output/*.ots
+output/*
+!output/README.md
 
-# Generated reference files
-references/generated/*
-!references/generated/README.md
+webpage/*.ots
+
+# Manubot cache directory
+ci/cache
 
 # Python
 __pycache__/

diff --git a/.travis.yml b/.travis.yml
@@ -20,7 +20,7 @@ script:
   - sh build/build.sh
 cache:
   directories:
-    - references/generated
+    - ci/cache
 after_success:
   - test $TRAVIS_BRANCH = "master" &&
     test $TRAVIS_PULL_REQUEST = "false" &&

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
diff --git a/README.md b/README.md
@@ -15,10 +15,36 @@ To see what's incoming, check the [open pull requests](https://github.com/greene
 Instructions for using Manubot Rootstock for your own manuscript are still evolving.
 The recommended approach is to clone this repository, as detailed [here](https://github.com/greenelab/manubot-rootstock/issues/6#issuecomment-314541837).
 
-## Source
+## Repository directories & files
 
-The manuscript source is located in [`content`](content).
-Text should be written in markdown files, with a digit prefix for ordering (e.g. `01.`, `02.`, etc.) and a `.md` suffix.
+See [`USAGE.md`](USAGE.md) for documentation on how to create or contribute to a manuscript.
+
+The directories are as follows:
+
++ [`content`](content) contains the manuscript source, which includes markdown files as well as inputs for citations and references.
++ [`output`](output) contains the outputs (generated files) from the manubot including the resulting manuscripts.
+  You should not edit these files manually, because they will get overwritten.
++ [`webpage`](webpage) is a directory meant to be rendered as a static webpage for viewing the HTML manuscript.
++ [`build`](build) contains commands and tools for building the manuscript.
++ [`ci`](ci) contains files necessary for deployment via continuous integration.
+  For the CI configuration, see [`.travis.yml`](.travis.yml).
+
+## Local execution
+
+To run the Manubot locally, install the [conda](https://conda.io) environment as described in [`build`](build).
+Then, you can build the manuscript on POSIX systems by running the following commands.
+
+```sh
+# Activate the manubot conda environment
+source activate manubot
+
+# Build the manuscript
+sh build/build.sh
+
+# View the manuscript locally at http://localhost:8000/
+cd webpage
+python -m http.server
+```
 
 ## Continuous Integration
 
@@ -28,7 +54,7 @@ When you make a pull request, Travis CI will test whether your changes break the
 The build process aims to detect common errors, such as invalid references.
 If your build fails, see the Travis CI logs for the cause of failure and revise your pull request accordingly.
 
-When a pull request is merged, Travis CI performs the build and writes the results to the [`gh-pages`](https://github.com/greenelab/manubot-rootstock/tree/gh-pages) and [`references`](https://github.com/greenelab/manubot-rootstock/tree/references) branches.
+When a pull request is merged, Travis CI performs the build and writes the results to the [`gh-pages`](https://github.com/greenelab/manubot-rootstock/tree/gh-pages) and [`output`](https://github.com/greenelab/manubot-rootstock/tree/output) branches.
 The `gh-pages` branch hosts the following URLs:
 
 + **HTML manuscript** at https://greenelab.github.io/manubot-rootstock/
@@ -49,7 +75,7 @@ All files matched by the following glob patterns are dual licensed under CC BY 4
 
 + `*.sh`
 + `*.py`
-+ `*.yml`
++ `*.yml` / `*.yaml`
 + `*.json`
 + `*.bib`
 + `*.tsv`
@@ -60,5 +86,6 @@ All other files are only available under CC BY 4.0, including:
 + `*.md`
 + `*.html`
 + `*.pdf`
++ `*.docx`
 
 Please open [an issue](https://github.com/greenelab/manubot-rootstock/issues) for any question related to licensing.
diff --git a/USAGE.md b/USAGE.md
@@ -0,0 +1,134 @@
+# Manubot usage guidelines
+
+This repository uses the [Manubot](https://github.com/greenelab/manubot) to automatically produce a manuscript from its source.
+
+## Manubot markdown
+
+Manuscript text should be written in markdown files, which should be located in [`content`](content) with a digit prefix for ordering (e.g. `01.`, `02.`, etc.) and a `.md` extension.
+
+For basic formatting, check out the [CommonMark Help](http://commonmark.org/help/) page for an introduction to the formatting options provided by standard markdown.
+In addition, manubot supports an extended version of markdown, tailored for scholarly writing.
+
+### Tables
+
+Manubot supports [markdown tables](https://help.github.com/articles/organizing-information-with-tables/ "GitHub Help: Organizing information with tables").
+
+```md
+| Column 1 | Column 2 | Column 3 |
+|----------|----------|----------|
+| value_a | 1 | 47 |
+| value_b | 2 | 56 |
+
+Table: Caption for this example table. {#tbl:example-id}
+```
+
+Support for table numbering and citation is provided by [`pandoc-tablenos`](https://github.com/tomduck/pandoc-tablenos).
+Above, `{#tbl:example-id}` sets the table ID, which creates an HTML anchor and allows citing the table like `@tbl:example-id`.
+For easy creation of markdown tables, check out the [Tables Generator](http://www.tablesgenerator.com/markdown_tables) webapp.
+
+### Figures
+
+Figures can be included with the following markdown:
+
+```md
+![Caption for the example figure.](url_or_path_to_figure){#fig:example-id}
+```
+
+Support for figure numbering and citation is provided by [`pandoc-fignos`](https://github.com/tomduck/pandoc-fignos).
+This figure can be cited in the text using `@fig:example-id`.
+In context, a figure citation may look like: `Figure {@fig:example-id}B shows …`.
+
+For images created by the manuscript authors that are hosted elsewhere on GitHub, we recommend using a [versioned](https://help.github.com/articles/getting-permanent-links-to-files/) GitHub URL to embed figures, thereby preserving exact image provenance.
+When embedding SVG images hosted on GitHub, passing the URL through [RawGit](https://rawgit.com/) is necessary.
+An example of a URL that has been passed through RawGit is:
+
+```
+https://cdn.rawgit.com/greenelab/scihub/572d6947cb958e797d1a07fdb273157ad9154273/figure/citescore.svg
+```
+
+Figures placed in the [`content/images`](content/images) directory can be embedded using their relative path.
+For example, we embed an [ORCID](https://orcid.org/) icon inline using:
+
+```md
+![ORCID icon](images/orcid.svg){height="13px"}
+```
+
+The bracketed text following the image declaration is interpreted by Pandoc's [`link_attributes`](http://pandoc.org/MANUAL.html#extension-link_attributes) extension.
+For example, the following will override the figure number to be "S1" and set the image width to 5 inches:
+
+```md
+{#fig:supplement tag="S1" width="5in"}
+```
+
+### Citations
+
+Manubot supports Pandoc [citations](http://pandoc.org/MANUAL.html#citations) via `pandoc-citeproc`.
+However, Manubot performs automated citation processing and metadata retrieval on raw citations.
+Therefore, citations must be of the following form: `@source:identifier`, where `source` is one of the options described below.
+When choosing which source to use for a citation, we recommend the following order:
+
+1. DOI (Digital Object Identifier), cite like `@doi:10.15363/thinklab.4`.
+2. PubMed ID, cite like `@pmid:26158728`.
+3. _arXiv_ ID, cite like `@arxiv:1508.06576v2`.
+4. URL / webpage, cite like `@url:http://openreview.net/pdf?id=Sk-oDY9ge`.
+
+Cite multiple items at once like:
+
+```md
+Here is a sentence with several citations [@doi:10.15363/thinklab.4; @pmid:26158728; @arxiv:1508.06576].
+```
+
+Note that multiple citations must be semicolon separated.
+
+#### Citation tags
+
+The system also supports citation tags, which are recommended for the following applications:
+
+1. A citation's identifier contains forbidden characters, such as `;` or ending with a non-alphanumeric character other than `/`.
+  In these instances, you must use a tag.
+2. A single reference is cited many times.
+  Therefore, it might make sense to define a tag, so if the citation updates (e.g. a newer version becomes available), only a single change is required.
+
+Tags should be defined in [`content/citation-tags.tsv`](content/citation-tags.tsv).
+If `citation-tags.tsv` defines the tag `study-x`, then this study can be cited like `@tag:study-x`.
+
+## Reference metadata
+
+The Manubot workflow requires the bibliographic details for references (the set of all cited works) as CSL (Citation Style Language) Items (also known as [citeproc JSON](http://citeproc-js.readthedocs.io/en/latest/csl-json/markup.html#csl-json-items)).
+The Manubot attempts to automatically retrieve metadata and generate valid citeproc JSON for references, which is exported to `output/references.json`.
+However, in some cases the Manubot fails to retrieve metadata or generates incorrect or incomplete citeproc metadata.
+Errors are most common for `url` references.
+For these references, you can manually specify the correct citeproc in [`content/manual-references.json`](content/manual-references.json), which will override the automatically generated reference data.
+To do so, create a new citeproc record that contains the field `"standard_citation"` with the appropriate reference identifier as its value.
+The identifier can be obtained from the `standard_citation` column of `citations.tsv`, which is located in the `output` branch or in the `output` subdirectory of local builds.
+As an example, `manual-references.json` contains:
+
+```json
+"standard_citation": "url:https://github.com/greenelab/manubot-rootstock"
+```
+
+For guidance on what CSL JSON should be like for different document types, refer to [these examples](https://github.com/aurimasv/zotero-import-export-formats/blob/a51c342e66bebd97b73a7230047b801c8f7bb690/CSL%20JSON.json).
+
+## Manuscript metadata
+
+[`content/metadata.yaml`](content/metadata.yaml) contains manuscript metadata that gets passed through to Pandoc, via a [`yaml_metadata_block`](http://pandoc.org/MANUAL.html#extension-yaml_metadata_block).
+`metadata.yaml` should contain the manuscript `title`, `authors` list, and `keywords`.
+Additional metadata, such as `date`, will automatically be created by the Manubot.
+
+We recommend authors add themselves to `metadata.yaml` via pull request (when requested by a maintainer), thereby signaling that they've read and approved the manuscript.
+The following YAML shows the supported key–value pairs for an author:  
+
+```yaml
+github: dhimmel  # strongly suggested
+name: Daniel S. Himmelstein  # mandatory
+initials: DSH  # strongly suggested
+orcid: 0000-0002-3012-7446  # mandatory
+twitter: dhimmel  # optional
+email: [email protected]  # suggested
+affiliations: Department of Systems Pharmacology and Translational Therapeutics, University of Pennsylvania  # strongly suggested
+funders: GBMF4552  # optional
+```
+
+## Manubot feedback
+
+If you experience any issues with the Manubot or would like to contribute to its source code, please visit [`greenelab/manubot`](https://github.com/greenelab/manubot) or [`greenelab/manubot-rootstock`](https://github.com/greenelab/manubot-rootstock).
diff --git a/references/pandoc-reference.docx → build/assets/pandoc-reference.docx b/references/pandoc-reference.docx → build/assets/pandoc-reference.docx
diff --git a/references/style.csl → build/assets/style.csl b/references/style.csl → build/assets/style.csl
diff --git a/build/build.sh b/build/build.sh
@@ -5,13 +5,17 @@ export LC_ALL=en_US.UTF-8
 
 # Generate reference information
 echo "Retrieving and processing reference metadata"
-(cd build && python references.py)
+manubot \
+  --content-directory=content \
+  --output-directory=output \
+  --cache-directory=ci/cache \
+  --log-level=INFO
 
 # pandoc settings
-CSL_PATH=references/style.csl
-DOCX_PATH=references/pandoc-reference.docx
-BIBLIOGRAPHY_PATH=references/generated/bibliography.json
-INPUT_PATH=references/generated/all-sections.md
+CSL_PATH=build/assets/style.csl
+DOCX_PATH=build/assets/pandoc-reference.docx
+BIBLIOGRAPHY_PATH=output/references.json
+INPUT_PATH=output/manuscript.md
 
 # Make output directory
 mkdir -p output
@@ -32,7 +36,7 @@ pandoc --verbose \
   --css=github-pandoc.css \
   --include-in-header=build/assets/analytics.js \
   --include-after-body=build/assets/anchors.js \
-  --output=output/index.html \
+  --output=output/manuscript.html \
   $INPUT_PATH
 
 # Create PDF output
@@ -44,13 +48,14 @@ wkhtmltopdf \
   --margin-bottom 17 \
   --margin-left 0 \
   --margin-right 0 \
-  output/index.html \
+  webpage/index.html \
   output/manuscript.pdf
 
 # Create DOCX output when user specifies to do so
 if [ "$BUILD_DOCX" = "true" ];
 then
     echo "Exporting Word Docx manuscript"
+    ln --symbolic content/images images
     pandoc --verbose \
     --smart \
     --from=markdown \
@@ -63,4 +68,5 @@ then
     --reference-docx=$DOCX_PATH \
     --output=output/manuscript.docx \
     $INPUT_PATH
+    rm --recursive images
 fi