The following metrics characterize the size of the benchmarks. The numbers are based on the SOMns benchmark implementations, but should be similar to the numbers one would measure for the other languages.

Compared to the classic notion of lines of code (LOC), we count only lines of code that have been executed at least once to measure the dynamic size of the program instead of its static size.

The number of classes includes only classes of which at least one method was executed. It is stable across languages with the exception of JavaScript which does not use classes in the version of the language used for this work.

Similar to counting executed LOC, we count methods that have been executed at least once.

In addition to executed methods, we further distinguish methods that are executed for each benchmark iteration. Thus, we separate out code that was only executed once during startup or shutdown. The number of per iteration methods indicates the methods that get likely compiled during benchmarking.

The dynamic metrics characterize the behavior of the benchmarks in more detail.

For method calls, we count at each call site, i.e., the lexical point in the program where a call is made, the number of different receiver types, as well as the number of different target methods that were activated at that point. The number of target methods should always be smaller or equal to the number of different receiver types.

Operators such as + or * are excluded from the method call count.

Observed receiver polymorphism, i.e., different observed receiver types/classes:

Observed target polymorphism, i.e., different activated methods:

Similar to method calls, we measure the number of lexical closures observed at a closure application site. A closure application site is the lexical point where a closure is executed.

As an indication for the recursive behavior, we measure the maximal observed stack height, i.e., the number of method activations on the runtime stack.

We count loops that have been activated at least once. Furthermore, we count the number of times a loop body has been executed.

We count the number of control flow branches that have been taken at least once. This includes if-branches, but also operations that have control flow semantics such as short-cutting or and and operators where the right-hand expression is executed conditionally. Furthermore, we count how often each branch is taken. The reported branch bias ratio is calculated with max(#true, #false) / (#true + #false).

As an overview of the control flow metrics, the following plot aggregates these metrics and puts them into relation to each other.

We track the number of arrays created as well as their overall size. Furthermore, we track the number of objects created and the number of declared fields.

Array allocations:

Object allocations:

We count object field reads and writes that were executed at least once. Furthermore, we report the number of accesses per iteration.

We count the sites of array reads and writes that were executed at least once. Furthermore, we count the number of array reads and writes per iteration.

As an overview of allocation and access metrics, the following plot aggregates and relates them to each other.

We report the number of sites of variable reads and writes that were executed at least once. This includes variables in methods and closures. Furthermore, we report the number of accesses per iteration.

Basic operations are also known as primitives or built-in functions. We include comparisons, arithmetic and bit operations, reading the size of strings or arrays, and string operations. Since the complexity of these operations range from simple integer additions, which can be mapped directly to a processor instruction, up to trigonometric functions, or string comparisons, which require complex algorithms, we categorize them in groups with similar properties.

These groups are defined as follows:

The following plot gives an overview of the usage of these operations by the benchmarks.

The following table gives a numerical overview of the metrics.