Redo implementation in Bourne Shell
I have implemented the build system redo as designed by DJB in Bourne Shell. To understand how redo can be simpler, more flexible, more powerful and more reliable than make, read Introduction to redo and/or redo: a top-down software build system. The current version of redo is redo 4.0.6.
- redo
- the main program
- redo-always
- marks the current target as always needing to be rebuilt
- redo-dot
- prints redo dependency graph in DOT format (example output, example rendering)
- redo-ifchange
- adds dependencies for the current target (if a dependency changes, the target will be rebuilt)
- redo-ifcreate
- adds non-existence dependencies for the current target (if a non-existence dependency is created, the target will be rebuilt)
- redo-ood
- prints a list of all target files that are out of date
- redo-targets
- prints a list of all target files that exist (a target is a file redo can build)
- redo-sources
- prints a list of all source files that exist (a source is a dependency that is not a target)
- redo-stamp
- detects if the current target has changed (see apenwarr's documentation)
- redo-whichdo
- prints search paths for dofiles to build a redo target
- redo-gcc
- automatically tracks dependencies and non-existence dependencies when compiling using gcc.
Installation
To install redo, copy the redo executables and man pages (detached PGP signature) to a directory in your $PATH. For example, if /usr/local/bin exists, is listed in your $PATH and can be written to by the current user, the following commands will install redo:
wget http://news.dieweltistgarnichtso.net/bin/archives/redo-sh.tar.gz tar -moxzf redo-sh.tar.gz -C /usr/local
This website is hosted in Germany and German police is legally allowed to spread malware. I therefore strongly recommend to verify the PGP signature before installation to ensure that the redo release archive has not been tampered with. If you have gpg installed and trust my PGP signing key, the following commands will install redo only if the signature matches:
wget http://news.dieweltistgarnichtso.net/bin/archives/redo-sh.tar.gz http://news.dieweltistgarnichtso.net/bin/archives/redo-sh.tar.gz.sig gpg --verify redo-sh.tar.gz.sig redo-sh.tar.gz && tar -moxzf redo-sh.tar.gz -C /usr/local
This implementation of redo depends only on GNU Core Utilities or BusyBox.
Frequently Asked Questions
- How do I build a file with redo?
- What do the redo parameters $1, $2, $3 mean?
- How does the extension removal in parameter $2 work?
- How do I declare dependencies with redo?
- How does this redo implementation check dependencies?
- How do I declare non-existence dependencies with redo?
- What are command line options for this redo implementation?
- How can I use this redo implementation to build in parallel?
- How can I make redo build everything in a different directory?
- Why can a target be built twice during a run?
- Why does redo-ifchange not add a dependency if the invoked dofile does not generate a target?
- Did you write tests for your redo implementation?
- How does this implementation of redo compare with other implementations?
How do I build a file with redo?
To build a file target, you have to create a dofile target.do with shell commands that write the intended result of the build either to standard output or to its parameter $3. Then run redo target.
The default target is all: redo without arguments executes commands from all.do.
What do the redo parameters $1, $2, $3 mean?
When redo runs a dofile, it gives it three parameters:
| $1 | filename of target |
| $2 | basename of target, without extension if default dofile is used |
| $3 | filename of temporary output file that is renamed on build success |
How does the extension removal in parameter $2 work?
Redo removes the extension that it gets from the default dofile filename:
| Target | Dofile | Parameter $2 |
|---|---|---|
| a.b.c | a.b.c.do | a.b.c |
| a.b.c | default.do | a.b.c |
| a.b.c | default.c.do | a.b |
| a.b.c | default.b.c.do | a |
How do I declare dependencies with redo?
Use the redo-ifchange command in a dofile: redo-ifchange dependency inside target.do means: If the target is built, the dependency is built if it does not exist and recorded as a dependency. On subsequent builds, if dependency does not exist or has changed since the last build, both dependency and target are rebuilt.
How does this redo implementation check dependencies?
For dependency checking, this implementation of redo checks the dependencies' ctime against the stored ctime. If the ctime differs, it checks the dependencies' md5sum against the stored md5sum. This is arguably more useful than just using ctime.
How do I declare non-existence dependencies with redo?
Use the redo-ifcreate command in a dofile: redo-ifcreate ne_dependency inside target.do means: If target is built, the non-existing file ne_dependency is recorded as a non-existence dependency. If ne_dependency exists on subsequent builds, target is rebuilt.
What are command line options for this redo implementation?
| Short option | Long option | Effect |
|---|---|---|
| -d | --debug | print dependency checks as they happen |
| --debug-jobs | print messages about job management | |
| --debug-locks | print messages about file locking | |
| -h | --help | print usage instructions and exit |
| -j [n] | --jobs [n] | execute at most [n] dofiles in parallel |
| --version | print version information and exit | |
| -x | --xtrace | print commands as they are executed (variables expanded) |
How can I use this redo implementation to build in parallel?
If the -j or --jobs option is given followed by an integer, an invocation of redo or redo-ifchange with multiple arguments builds targets in parallel. The integer argument specifies the maximum number of targets allowed to be built in parallel. Targets are locked, so if targets built in parallel share a dependency, redo only builds it once, unless the dependency dofile uses redo-always to rebuild every time, or the dependency becomes out of date during a build in some other way.
How can I make redo build everything in a different directory?
You can use union mounts to combine multiple directories. For example, on Linux with unionfs-fuse, the command unionfs -o cow 'output=RW:source=RO' /tmp/build combines the output and source directories so that /tmp/build contains all files from the source directory, but any newly created file will show up only in the output directory.
Why can a target be built twice during a run?
Some build processes require several builds of the same target. Naïve implementors of build systems disregard this possibility and do not check the validity of a target again if it was built during a run.
- A target that has a dofile that invokes redo-always will be rebuilt several more times, if several targets invoke redo-ifchange for it.
- A dofile can use a loop involving inotifywait to wait for changes to dependencies and immediately rebuild targets if a dependency changes.
- TeX documents can contain internal references. Page and equation numbers are written to an external file needed for a second build.
- TemplateHaskell with profiling needs an executable compiled twice.
- The technique detailed in David A. Wheeler's Fully Countering Trusting Trust through Diverse Double-Compiling requires two builds of a target.
Why does redo-ifchange not add a dependency if the invoked dofile does not generate a target?
redo rebuilds target files when source files have changed. Files that do not exist can never change – therefore, a non-existent file is always up to date.
Did you write tests for your redo implementation?
Yes.
| Test Script | Description | Test Status |
|---|---|---|
| always_rebuild_1 | A target built with a dofile that invoked redo-always must always be rebuilt. | Passed |
| dependency_ne_dofile_1 | A target built with a default dofile in the same directory must be rebuilt if a more specific dofile appears. | Passed |
| dependency_ne_dofile_2 | A target built with a default dofile in the parent directory must be rebuilt if a more specific dofile appears. | Passed |
| dependency_ne_nobuild_1 | A target must not be rebuilt when a non-existence dependency does not exist. | Passed |
| dependency_ne_rebuild_1 | A target must be rebuilt when a non-existence dependency exists. | Passed |
| dependency_nobuild_1 | A target must not be rebuilt when a dependency does not change. | Passed |
| dependency_nobuild_2 | A target must not be rebuilt when a dependency of a dependency does not change. | Passed |
| dependency_nobuild_3 | A target must not be rebuilt when a dependency of a dependency of a dependency does not change. | Passed |
| dependency_rebuild_1 | A target must be rebuilt when a dependency changes. | Passed |
| dependency_rebuild_2 | A target must be rebuilt when a dependency of a dependency changes. | Passed |
| dependency_rebuild_3 | A target must be rebuilt when a dependency of a dependency of a dependency changes. | Passed |
| directory_dependency_1 | Directories can not be targets or dependencies. | Passed |
| dofile_generated_1 | A target built by a dofile built during the build must be built. | Passed |
| dofile_generated_2 | A target built by a dofile built during the build by a dofile built during the build must be built. | Passed |
| nothing_written_1 | A non-top-level build rule not writing to a target must yield a warning. | Passed |
| nothing_written_2 | Each non-top-level build rule not writing to a target must yield a warning. | Passed |
| ood_1 | When invoked, redo-ood must output targets for which a dependency changed. | Passed |
| ood_2 | When invoked, redo-ood must output targets for which a dependency changed and targets for which a dependency of a dependency changed. | Passed |
| ood_3 | When invoked, redo-ood must output targets for which a dependency changed and targets for which a dependency of a dependency changed and targets for which a dependency of a dependency of a dependency changed. | Passed |
| parallel_1 | Targets must be built in parallel if “-j” option is given. | Passed |
| parallel_2 | Targets must be built in parallel if “--jobs” option is given. | Passed |
| whichdo_absolute_1 | When invoked with an absolute path name as an argument, redo-whichdo must output non-existent dofiles in the same directory until it outputs one that exists. | Passed |
| whichdo_absolute_2 | When invoked with an absolute path name as an argument, redo-whichdo must output non-existent dofiles in the parent directory and the same directory until it outputs one that exists. | Passed |
| whichdo_absolute_default_1 | When invoked with an absolute path name with the prefix “default” as an argument, redo-whichdo must not output any dofile filename twice. | Passed |
| whichdo_absolute_default_2 | When invoked with an absolute filename for the file “default.do” as an argument, redo-whichdo must not output that filename as possible dofile. | Passed |
| whichdo_relative_1 | When invoked with a relative path name as an argument, redo-whichdo must output non-existent dofiles in the same directory until it outputs one that exists. | Passed |
| whichdo_relative_2 | When invoked with a relative path name as an argument, redo-whichdo must output non-existent dofiles in the parent directory and the same directory until it outputs one that exists. | Passed |
| whichdo_relative_default_1 | When invoked with a relative path name with the prefix “default” as an argument, redo-whichdo must not output any dofile filename twice. | Passed |
| whichdo_relative_default_2 | When invoked with a relative filename for the file “default.do” as an argument, redo-whichdo must not output that filename as possible dofile. | Passed |
How does this implementation of redo compare with other implementations?
The following table compares redo implementations. Differences from the DJB redo design and incompatibilities with existing dofiles are emphasized.
| Implementation | Programming Language | Modification Time Check | File Content Hash Check | Parallel Build | Notes, Bugs, and Incompatibilities |
|---|---|---|---|---|---|
| Nils Dagsson Moskopp's redo | Bourne Shell | ✓ Yes | ✓ Yes | ✓ Yes |
|
| Alan Grosskurth's redo | Bourne Shell | ✕ No | ✓ Yes | ✕ No |
|
| Avery Pennarun's do | Bourne Shell | ✕ No | ✕ No | ✕ No |
|
| Avery Pennarun's redo | Python | ✓ Yes | ✕ No | ✓ Yes |
|
| Leah Neukirchen's redo | C | ✓ Yes | ✓ Yes | ✓ Yes |
|
| Jonathan de Boyne Pollard's redo | Bourne Again Shell | ✓ Yes | ✓ Yes | ✕ No |
|
| Gyepi Sam's redux | Go | ✓ Yes | ✓ Yes | ✕ No |
|
| jekor's redo | Haskell | ✕ No | ✓ Yes | ✕ No |
|
| Jonathan de Boyne Pollard's redo | C++ | ✓ Yes | ✓ Yes | ✓ Yes |
|
| Shanti Bouchez-Mongardé's redo | Python | ✓ Yes | ✕ No | ✓ Yes |
|
| Tharre's redo | C | ✕ No | ✓ Yes | ✕ No |
|