Documentation
¶
Overview ¶
Package benchproc provides tools for filtering, grouping, and sorting benchmark results.
This package supports a pipeline processing model based around domain-specific languages for filtering and projecting benchmark results. These languages are described in "go doc golang.org/x/perf/benchproc/syntax".
The typical steps for processing a stream of benchmark results are:
1. Read a stream of benchfmt.Results from one or more input sources. Command-line tools will often do this using benchfmt.Files.
2. For each benchfmt.Result:
2a. Optionally transform the benchfmt.Result to add or modify keys according to a particular tool's needs. Custom keys often start with "." to distinguish them from file or sub-name keys. For example, benchfmt.Files adds a ".file" key.
2b. Optionally filter the benchfmt.Result according to a user-provided predicate parsed by NewFilter. Filters can keep or discard entire Results, or just particular measurements from a Result.
2c. Project the benchfmt.Result using one or more Projections. Projecting a Result extracts a subset of the information from a Result into a Key. Projections are produced by a ProjectionParser, typically from user-provided projection expressions. An application should have a Projection for each "dimension" of its output. For example, an application that emits a table may have two Projections: one for rows and one for columns. An application that produces a scatterplot could have Projections for X and Y as well as other visual properties like point color and size.
2d. Group the benchfmt.Result according to the projected Keys. Usually this is done by storing the measurements from the Result in a Go map indexed by Key. Because identical Keys compare ==, they can be used as map keys. Applications that use two or more Projections may use a map of maps, or a map keyed by a struct of two Keys, or some combination.
3. At the end of the Results stream, once all Results have been grouped by their Keys, sort the Keys of each dimension using SortKeys and present the data in the resulting order.
Example ¶
Example shows a complete benchmark processing pipeline that uses filtering, projection, accumulation, and sorting.
// Open the example benchmark data.
f, err := os.Open("testdata/suffixarray.bench")
if err != nil {
log.Fatal(err)
}
defer f.Close()
// Create a filter that extracts just "BenchmarkNew" on the value
// "go" of the name key "text". Typically, the filter expression
// would come from a command-line flag.
filter, err := NewFilter(".name:New /text:go")
if err != nil {
log.Fatal(err)
}
// Create a projection. This projection extracts "/bits=" and
// "/size=" from the benchmark name. It sorts bits in the
// default, first-observation order and size numerically.
// Typically, the projection expression would come from a
// command-line flag.
var pp ProjectionParser
projection, err := pp.Parse("/bits,/size@num", filter)
if err != nil {
log.Fatal(err)
}
// Create a projection that captures all configuration not
// captured by the above projection. We'll use this to check
// if there's unexpected variation in other configuration
// fields and report it.
residue := pp.Residue()
// We'll accumulate benchmark results by their projection.
// Projections create Keys, which are == if the projected
// values are ==, so they can be used as map keys.
bySize := make(map[Key][]float64)
var keys []Key
var residues []Key
// Read the benchmark results.
r := benchfmt.NewReader(f, "example")
for r.Scan() {
var res *benchfmt.Result
switch rec := r.Result(); rec := rec.(type) {
case *benchfmt.Result:
res = rec
case *benchfmt.SyntaxError:
// Report a non-fatal parse error.
log.Print(err)
continue
default:
// Unknown record type. Ignore.
continue
}
// Step 1: If necessary, transform the Result, for example to
// add configuration keys that could be used in filters and
// projections. This example doesn't need any transformation.
// Step 2: Filter the result.
if match, err := filter.Apply(res); !match {
// Result was fully excluded by the filter.
if err != nil {
// Print the reason we rejected this result.
log.Print(err)
}
continue
}
// Step 3: Project the result. This produces a Key
// that captures the "size" and "bits" from the result.
key := projection.Project(res)
// Accumulate the results by configuration.
speed, ok := res.Value("sec/op")
if !ok {
continue
}
if _, ok := bySize[key]; !ok {
keys = append(keys, key)
}
bySize[key] = append(bySize[key], speed)
// Collect residue configurations.
resConfig := residue.Project(res)
residues = append(residues, resConfig)
}
// Check for I/O errors.
if err := r.Err(); err != nil {
log.Fatal(err)
}
// Step 4: Sort the collected configurations using the order
// specified by the projection.
SortKeys(keys)
// Print the results.
fmt.Printf("%-24s %s\n", "config", "sec/op")
for _, config := range keys {
fmt.Printf("%-24s %s\n", config, benchunit.Scale(mean(bySize[config]), benchunit.Decimal))
}
// Check if there was variation in any other configuration
// fields that wasn't captured by the projection and warn the
// user that something may be unexpected.
nonsingular := NonSingularFields(residues)
if len(nonsingular) > 0 {
fmt.Printf("warning: results vary in %s\n", nonsingular)
}
Output: config sec/op /bits:32 /size:100K 4.650m /bits:32 /size:500K 26.18m /bits:32 /size:1M 51.39m /bits:32 /size:5M 306.7m /bits:32 /size:10M 753.0m /bits:32 /size:50M 5.814 /bits:64 /size:100K 5.081m /bits:64 /size:500K 26.43m /bits:64 /size:1M 55.60m /bits:64 /size:5M 366.6m /bits:64 /size:10M 821.2m /bits:64 /size:50M 6.390
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Field ¶
type Field struct {
Name string
// IsTuple indicates that this Field is a tuple that does not itself
// have a string value.
IsTuple bool
// Sub is the sequence of sub-Fields for a group field.
Sub []*Field
// contains filtered or unexported fields
}
A Field is a single field of a Projection.
For example, in the projection ".name,/gomaxprocs", ".name" and "/gomaxprocs" are both Fields.
A Field may be a group field with sub-Fields.
func NonSingularFields ¶
NonSingularFields returns the subset of Fields for which at least two of keys have different values.
This is useful for warning the user if aggregating a set of results has resulted in potentially hiding important configuration differences. Typically these keys are "residue" keys produced by ProjectionParser.Residue.
type Filter ¶
type Filter struct {
// contains filtered or unexported fields
}
A Filter filters benchmarks and benchmark observations.
func NewFilter ¶
NewFilter constructs a result filter from a boolean filter expression, such as ".name:Copy /size:4k". See "go doc golang.org/x/perf/benchproc/syntax" for a description of filter syntax.
To create a filter that matches everything, pass "*" for query.
func (*Filter) Apply ¶
Apply rewrites res.Values to keep only the measurements that match the Filter f and reports whether any measurements remain.
Apply returns true if all or part of res.Values is kept by the filter. Otherwise, it sets res.Values to an empty slice and returns false to indicate res was completely filtered out.
If Apply returns false, it may return a non-nil error indicating why the result was filtered out.
type Key ¶
type Key struct {
// contains filtered or unexported fields
}
A Key is an immutable tuple mapping from Fields to strings whose structure is given by a Projection. Two Keys are == if they come from the same Projection and have identical values.
func (Key) Get ¶
Get returns the value of Field f in this Key.
It panics if Field f does not come from the same Projection as the Key or if f is a tuple Field.
func (Key) Less ¶
Less reports whether k comes before o in the sort order implied by their projection. It panics if k and o have different Projections.
func (Key) Projection ¶
func (k Key) Projection() *Projection
Projection returns the Projection describing Key k.
func (Key) String ¶
String returns Key as a space-separated sequence of key:value pairs in field order.
func (Key) StringValues ¶
StringValues returns Key as a space-separated sequences of values in field order.
type KeyHeader ¶
type KeyHeader struct {
// Keys is the sequence of keys at the leaf level of this tree.
// Each level subdivides this sequence.
Keys []Key
// Levels are the labels for each level of the tree. Level i of the
// tree corresponds to the i'th field of all Keys.
Levels []*Field
// Top is the list of tree roots. These nodes are all at level 0.
Top []*KeyHeaderNode
}
A KeyHeader represents a slice of Keys, combined into a forest of trees by common Field values. This is intended to visually present a sequence of Keys in a compact form; primarily, as a header on a table.
All Keys must have the same Projection. The levels of the tree correspond to the Fields of the Projection, and the depth of the tree is exactly the number of Fields. A node at level i represents a subslice of the Keys that all have the same values as each other for fields 0 through i.
For example, given four keys:
K[0] = a:1 b:1 c:1 K[1] = a:1 b:1 c:2 K[2] = a:2 b:2 c:2 K[3] = a:2 b:3 c:3
the KeyHeader is as follows:
Level 0 a:1 a:2
| / \
Level 1 b:1 b:2 b:3
/ \ | |
Level 2 c:1 c:2 c:2 c:3
K[0] K[1] K[2] K[3]
func NewKeyHeader ¶
NewKeyHeader computes the KeyHeader of a slice of Keys by combining common prefixes of fields.
type KeyHeaderNode ¶
type KeyHeaderNode struct {
// Field is the index into KeyHeader.Levels of the Field represented
// by this node. This is also the level of this node in the tree,
// starting with 0 at the top.
Field int
// Value is the value that all Keys have in common for Field.
Value string
// Start is the index of the first Key covered by this node.
Start int
// Len is the number of Keys in the sequence represented by this node.
// This is also the number of leaf nodes in the subtree at this node.
Len int
// Children are the children of this node. These are all at level Field+1.
// Child i+1 differs from child i in the value of field Field+1.
Children []*KeyHeaderNode
}
KeyHeaderNode is a node in a KeyHeader.
type Match ¶
type Match struct {
// contains filtered or unexported fields
}
A Match records the set of result measurements that matched a filter query.
type Projection ¶
type Projection struct {
// contains filtered or unexported fields
}
A Projection extracts some subset of the fields of a benchfmt.Result into a Key.
A Projection also implies a sort order over Keys that is lexicographic over the fields of the Projection. The sort order of each individual field is specified by the projection expression and defaults to the order in which values of that field were first observed.
func (*Projection) Fields ¶
func (p *Projection) Fields() []*Field
Fields returns the fields of p. These correspond exactly to the fields in the Projection's projection expression.
The caller must not modify the returned slice.
func (*Projection) FlattenedFields ¶
func (p *Projection) FlattenedFields() []*Field
FlattenedFields is like Fields, but expands tuple Fields (specifically, ".config") into their sub-Fields. This is also the sequence of Fields used for sorting Keys returned from this Projection.
The caller must not modify the returned slice.
func (*Projection) Project ¶
func (p *Projection) Project(r *benchfmt.Result) Key
Project extracts fields from benchmark Result r according to Projection s and returns them as a Key.
Two Keys produced by Project will be == if and only if their projected fields have the same values. Notably, this means Keys can be used as Go map keys, which is useful for grouping benchmark results.
Calling Project may add new sub-Fields to group Fields in this Projection. For example, if the Projection has a ".config" field and r has a never-before-seen file configuration key, this will add a new sub-Field to the ".config" Field.
If this Projection includes a .units field, it will be left as "" in the resulting Key. The caller should use ProjectValues instead.
func (*Projection) ProjectValues ¶
func (p *Projection) ProjectValues(r *benchfmt.Result) []Key
ProjectValues is like Project, but for each benchmark value of r.Values individually. The returned slice corresponds to the r.Values slice.
If this Projection includes a .unit field, it will differ between these Keys. If not, then all of the Keys will be identical because the benchmark values vary only on .unit.
type ProjectionParser ¶
type ProjectionParser struct {
// contains filtered or unexported fields
}
A ProjectionParser parses one or more related projection expressions.
func (*ProjectionParser) Parse ¶
func (p *ProjectionParser) Parse(projection string, filter *Filter) (*Projection, error)
Parse parses a single projection expression, such as ".name,/size". A projection expression describes how to extract fields of a benchfmt.Result into a Key and how to order the resulting Keys. See "go doc golang.org/x/perf/benchproc/syntax" for a description of projection syntax.
A projection expression may also imply a filter, for example if there's a fixed order like "/size@(1MiB)". Parse will add any filters to "filter".
If an application calls Parse multiple times on the same ProjectionParser, these form a mutually-exclusive group of projections in which specific keys in any projection are excluded from group keys in any other projection. The group keys are ".config" and ".fullname". For example, given two projections ".config" and "commit,date", the specific file configuration keys "commit" and "date" are excluded from the group key ".config". The result is the same regardless of the order these expressions are parsed in.
func (*ProjectionParser) ParseWithUnit ¶
func (p *ProjectionParser) ParseWithUnit(projection string, filter *Filter) (*Projection, *Field, error)
ParseWithUnit is like Parse, but the returned Projection has an additional field called ".unit" that extracts the unit of each individual benchfmt.Value in a benchfmt.Result. It returns the Projection and the ".unit" Field.
Typically, callers need to break out individual benchmark values on some dimension of a set of Projections. Adding a .unit field makes this easy.
Callers should use the ProjectValues method of the returned Projection rather than the Project method to project each value rather than the whole benchfmt.Result.
func (*ProjectionParser) Residue ¶
func (p *ProjectionParser) Residue() *Projection
Residue returns a projection for any field not yet projected by any projection parsed by p. The resulting Projection does not have a meaningful order.
For example, following calls to p.Parse("goos") and p.Parse(".fullname"), Reside would return a Projection with fields for all file configuration fields except goos.
The intended use of this is to report when a user may have over-aggregated results. Specifically, track the residues of all of the benchfmt.Results that are aggregated together (e.g., into a single table cell). If there's more than one distinct residue, report that those results differed in some field. Typically this is used with NonSingularFields to report exactly which fields differ.
Example ¶
ExampleProjectionParser_Residue demonstrates residue projections.
This example groups a set of results by .fullname and goos, but the two results for goos:linux have two different goarch values, indicating the user probably unintentionally grouped uncomparable results together. The example uses ProjectionParser.Residue and NonSingularFields to warn the user about this.
var pp ProjectionParser
p, _ := pp.Parse(".fullname,goos", nil)
residue := pp.Residue()
// Aggregate each result by p and track the residue of each group.
type group struct {
values []float64
residues []Key
}
groups := make(map[Key]*group)
var keys []Key
for _, result := range results(`
goos: linux
goarch: amd64
BenchmarkAlloc 1 128 ns/op
goos: linux
goarch: arm64
BenchmarkAlloc 1 137 ns/op
goos: darwin
goarch: amd64
BenchmarkAlloc 1 130 ns/op`) {
// Map result to a group.
key := p.Project(result)
g, ok := groups[key]
if !ok {
g = new(group)
groups[key] = g
keys = append(keys, key)
}
// Add value to the group.
speed, _ := result.Value("sec/op")
g.values = append(g.values, speed)
// Add residue to the group.
g.residues = append(g.residues, residue.Project(result))
}
// Report aggregated results.
SortKeys(keys)
for _, k := range keys {
g := groups[k]
// Report the result.
fmt.Println(k, mean(g.values), "sec/op")
// Check if the grouped results vary in some unexpected way.
nonsingular := NonSingularFields(g.residues)
if len(nonsingular) > 0 {
// Report a potential issue.
fmt.Printf("warning: results vary in %s and may be uncomparable\n", nonsingular)
}
}
Output: .fullname:Alloc goos:linux 1.325e-07 sec/op warning: results vary in [goarch] and may be uncomparable .fullname:Alloc goos:darwin 1.3e-07 sec/op
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
internal
|
|
|
parse
Package parse implements parsers for golang.org/x/perf/benchproc/syntax.
|
Package parse implements parsers for golang.org/x/perf/benchproc/syntax. |
|
Package syntax documents the syntax used by benchmark filter and projection expressions.
|
Package syntax documents the syntax used by benchmark filter and projection expressions. |