This is the third feature release of sbt 1.x, a binary compatible release focusing on new features. sbt 1.x is released under Semantic Versioning, and the plugins are expected to work throughout the 1.x series.
The headline features of sbt 1.3 are out-of-box Coursier library management, ClassLoader layering, IO improvements, and super shell. Combined together we hope these features will improve the user experience of running your builds.
.withAllowInsecureProtocol(true) #4997
CrossVersion.Disabled. Please use CrossVersion.disabled instead sbt/librarymanagement#316
run and test tasks after those tasks complete. This may cause downstream crashes if the task uses ShutdownHooks or if any threads created by the tasks continue running after the task completes. To disable this behavior, either set Compile / run / fork := true or run sbt with -Dsbt.classloader.close=false.
sbt 1.3.0 adopts Coursier for the library management. Coursier is a dependency resolver like Ivy, rewritten in Scala by Alexandre Archambault (@alexarchambault), aiming to be a faster alternative.
Note: Under some situations, Coursier may not resolve the same way as Ivy (for example remote -SNAPSHOTs are cached for 24 hours). If you wish to go back to Apache Ivy for library management, put the following in your build.sbt:
ThisBuild / useCoursier := false
Many people were involved in the effort of bringing Coursier to sbt. Early in 2018 Leonard Ehrenfried (@leonardehrenfried) started the Coursier-backed LM API implementation as lm#190. During the fall, it was further improved by Andrea Peruffo (@andreaTP), and lm-coursier eventually became part of coursier/sbt-coursier repository maintained by Alex. This spring, Eugene (@eed3si9n) revisited this again to make a few more changes so we can swap out the LM engine in #4614 with the help from Alex.
sbt 1.3.0 adds “turbo” mode that enables experimental or advanced features that might require some debugging by the build user when it doesn’t work.
ThisBuild / turbo := true
Initially we are putting the layered ClassLoader (ClassLoaderLayeringStrategy.AllLibraryJars) behind this flag.
sbt has always created two-layer ClassLoaders when evaluating the run and test tasks. The top layer of the ClassLoader contains the scala library jars so that the classes in the scala package may be reused across multiple task evaluations. The second layer loads the rest of the project classpath including the library dependencies and project class files. sbt 1.3.0 introduces experimental classLoaderLayeringStrategy feature that furthers this concept.
Compile / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.Flat
// default
Compile / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.ScalaLibrary
// enabled with turbo
Compile / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.AllLibraryJars
Test / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.Flat
// default
Test / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.ScalaLibrary
// enabled with turbo
Test / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.AllLibraryJars
ClassLoaderLayeringStrategy.Flat includes all classes and JARs except for the Java runtime. The behavior of tasks using this strategy should be similar to forking without the overhead of starting a new jvm.
ClassLoaderLayeringStrategy.ScalaLibrary creates a two-layer ClassLoader where Scala standard library is kept warm, similar to sbt 1.2.x
ClassLoaderLayeringStrategy.AllLibraryJars creates a three-layer ClassLoader where library dependencies, in addition to Scala standard libraries are kept warm
ClassLoaderLayeringStrategy.AllLibraryJars should benefit the response time of run and test tasks. By caching the library jar classloader, the startup latency of the run and test tasks can be reduced significantly when they are run multiple times within the same session. GC pressure is also reduced because libraries jars will not be reloaded every time the task is evaluated.
Note: ClassLoaderLayeringStrategy.AllLibraryJars reuses the singleton object between the tests, which requires libraries to clean after itself.
ClassLoaderLayeringStrategy.Flat on the other hand will benefit certain applications that do not work well with layered ClassLoaders. One such example is Java serialization + serialization proxy pattern used by Scala collections.
ClassLoader layering was contributed by Ethan Atkins (@eatkins) as #4476
In addition to classloader layering, sbt 1.3.0 incorporates numerous performance enhancements including:
As of this writing sbt 1.3.0’s edit-compile-test loop for 5000 source files is faster than that edit-compile-test with three source files using sbt 0.13, Gradle, and other build tools we tested (see build performance for details). These changes were contributed by Ethan Atkins (@eatkins).
sbt 1.3.0 introduces a new type, Glob, that describes a path search query. For example, all of the scala sources in the project directory can be described by Glob(baseDirectory.value, RecursiveGlob / "*.scala") or baseDirectory.value.toGlob / ** / "*.scala", where ** is an alias for RecursiveGlob. Glob expands on PathFinders but they can be composed with no io overhead. Globs can be retrieved using a FileTreeView. For example, one can write:
val scalaSources = baseDirectory.value.toGlob / ** / "*.scala"
val javaSources = baseDirectory.value.toGlob / ** / "*.java"
val allSources = fileTreeView.value.list(Seq(scalaSources, javaSources))
and the FileTreeView will only traverse the base directory once. Globs and FileTreeView were added by Ethan Atkins (@eatkins) in io#178,io#216,io#226
sbt 1.3.0 introduces a new file monitoring implementation. It uses enhanced apis for tracking file change events using os events. It adds a new parser that extracts the specific task(s) for which it will monitor source files and rerun when it detects changes. Only source dependencies of the running tasks are monitored. For example, when running ~compile, changes to test source files will not trigger a new build. Between file events, there are also now options to return to the shell, rerun the previous command(s) or exit sbt. These changes were implemented by Ethan Atkins (@eatkins) in io#178,#216,#226,#4512,#4627.
sbt 1.3.0 automatically watches the build definition sources and displays a warning if you execute a task without reloading. This can be configured to reload automatically as follows:
Global / onChangedBuildSource := ReloadOnSourceChanges
This feature was contributed by Ethan Atkins (@eatkins) in #4664
sbt 1.3.0 provides support to implement custom incremental tasks based on files.
Given a custom task that returns java.nio.file.Path, Seq[java.nio.file.Path], File, or Seq[File],
you can define a few helper tasks to make it more incremental.
import java.nio.file._
import scala.sys.process._
val gccCompile = taskKey[Seq[Path]]("compile C code using gcc")
val gccHeaders = taskKey[Seq[Path]]("header files")
val gccInclude = settingKey[Path]("include directory")
val gccLink = taskKey[Path]("link C code using gcc")
gccCompile / sourceDirectory := sourceDirectory.value
gccCompile / fileInputs += (gccCompile / sourceDirectory).value.toGlob / ** / "*.c"
gccInclude := (gccCompile / sourceDirectory).value.toPath / "include"
gccHeaders / fileInputs += gccInclude.value.toGlob / "*.h"
gccCompile / target := baseDirectory.value / "out"
gccCompile := {
val objectDir = Files.createDirectories((gccCompile / target).value.toPath / "objects")
def objectFile(path: Path): Path =
target.value.toPath / path.getFileName.toString.replaceAll(".c$", ".o")
Files.createDirectories(target.value.toPath)
val headerChanges = gccHeaders.inputFileChanges.hasChanges
val changes = gccCompile.inputFileChanges
changes.deleted.foreach(sf => Files.deleteIfExists(objectFile(sf)))
val sourceFileChanges = changes.created ++ changes.modified
val needRecompile = (sourceFileChanges ++ (if (headerChanges) changes.unmodified else Nil)).toSet
val logger = streams.value.log
gccCompile.inputFiles.map { sf =>
val of = objectFile(sf)
if (!Files.exists(of) || needRecompile(sf)) {
logger.info(s"Compiling $sf")
s"gcc -I${gccInclude.value} -c $sf -o $of".!!
}
of
}
}
Given this setup, gccCompile.inputFiles will return a sequence of all of the input c source files, gccCompile.inputFileChanges returns a data structure containing the created, deleted, modified and unmodified files since the last run of gccCompile while gccHeaders.changedInputFiles returns the headers that have changed since the last run of gccCompile. Taken together, these tasks can be used to incrementally only rebuild the source files that need to be rebuilt given the file system changes since the last time gccCompile completed.
In another task such as gccLink, the result of gccCompile can be tracked as well using gccCompile.outputFileChanges.
gccLink := {
val library = (gccCompile / target).value.toPath / "libmylib.dylib"
val objectFiles = gccCompile.outputFiles
val logger = streams.value.log
if (!Files.exists(library) || gccCompile.outputFileChanges.hasChanges) {
logger.info(s"Rebuilding $library")
s"gcc -dynamiclib -o $library ${objectFiles mkString " "}".!!
}
library
}
The inputs of a task will automatically be monitored by the ~ command which has a new parser that is context aware. A custom clean task is also implemented for any task that generates file outputs. The clean tasks are aggregated across the project and config scopes. For example, Test / clean will clean all of the files generated by tasks in the Test config declared in the Test config but not the files generated in the Compile config.
This feature was contributed by Ethan Atkins (@eatkins) in #4627.
When running in an ANSI-compatible terminal, sbt 1.3.0 will display the currently running tasks. This gives the developer the idea of what tasks are being processed in parallel, and where the build is spending its time. In homage to Gradle’s “Rich Console” and Buck’s “Super Console”, we call ours “Super shell.”
To opt-out put the following in the build:
ThisBuild / useSuperShell := false
or run sbt with --supershell=false (or -Dsbt.supershell=false). This feature was added by Eugene Yokota (@eed3si9n) as #4396/util#196.
To view the task breakdown visually, run sbt with --traces (or -Dsbt.traces=true). This will generate build.traces file, which is viewable using Chrome Tracing chrome://tracing/. This feature was contributed by Jason Zaugg (@retronym).
To output the task timings on screen, run sbt with --timings (or -Dsbt.task.timings=true -Dsbt.task.timings.on.shutdown=true).
sbt 1.3.0 makes it easier to generate SemanticDB. To enable the generation of SemanticDB build-wide:
ThisBuild / semanticdbEnabled := true
ThisBuild / semanticdbVersion := "4.1.9"
ThisBuild / semanticdbIncludeInJar := false
This was added by @eed3si9n as #4410.
sbt 1.3.0 adds a new print command, similar to show but prints directly to standard out.
# sbt -no-colors --error "print akka-cluster/scalaVersion"
2.12.8
This was contributed by David Knapp (@Falmarri) as #4341
Function1 can be appended using +=.
Global / onLoad += { s =>
doSomething()
s
}
This was contributed by Dale Wijnand (@dwijnand) as #4521.
sbt 1.3.0 is first release of sbt that’s been testing on JDK11 extensively. All integration tests on Travis CI are on AdoptOpenJDK’s JDK 11, which were updated by @eed3si9n as #4389/zinc#639/[zinc640].
rt.jar on JDK 11 #4679 by @eatkins
~ in -Dsbt.global.base property to user home. #4367 by @kai-chi
def sequential[A](tasks: Seq[Initialize[Task[A]]]): Initialize[Task[A]]. #4369 by @3tty0n
"sbt/completion" command in sbt to server to complete sbt commands. #4397 by @andreaTP
import sbt.dsl.LinterLevel.Ignore in scope. #4485 by @eatkins
First, I’d like to introduce Ethan Atkins, a core community member of sbt project, and author of Close Watch that uses native code to provide watch service on macOS. Normally I don’t publicize the number of commits, but here’s the top 10 for sbt 1.3.0:
541 Ethan Atkins
369 Eugene Yokota (eed3si9n)
42 Jorge Vicente Cantero (jvican)
35 Łukasz Wawrzyk
34 Dale Wijnand
24 Andrea Peruffo
16 Kenji Yoshida (xuwei-k)
13 Guillaume Martres
7 Arnout Engelen
7 Jason Zaugg
As a community member, Ethan has contributed various IO related improvements to make sbt more responsive in his own time. sbt 1.3.0 reflects many of his ideas.
The last feature release of sbt 1 was sbt 1.2.0 in July, 2018. Since then, we’ve released eight patch releases under sbt 1.2.x for bug fixes, but most of the feature enhancements were merged to develop branch. Over the course of these months, 45 contributors contributors participated in sbt 1.3.0 and Zinc: Ethan Atkins, Eugene Yokota (eed3si9n), Jorge Vicente Cantero (jvican), Łukasz Wawrzyk, Dale Wijnand, Andrea Peruffo, Kenji Yoshida (xuwei-k), Guillaume Martres, Arnout Engelen, Jason Zaugg, Krzysztof Romanowski, Antonio Cunei, Mirco Dotta, OlegYch, Alex Dupre, Nepomuk Seiler, 0lejk4, Alexandre Archambault, Eric Peters, Kazuhiro Sera, Philippus, Som Snytt, Syed Akber Jafri, Thomas Droxler, Veera Venky, bigwheel, Akhtyam Sakaev, Alexey Vakhrenev, Eugene Platonov, Helena Edelson, Ignasi Marimon-Clos, Julien Sirocchi, Justin Kaeser, Kajetan Maliszewski, Leonard Ehrenfried, Mikołaj Jakubowski, Nafer Sanabria, Stefan Wachter, Yasuhiro Tatsuno, Yusuke Izawa, falmarri, ilya, kai-chi, tanishiking, Ólafur Páll Geirsson. Thank you!
