Switch lowering in GCC

2024-01-14T00:00:00+00:00

Introduction

We have seen in a previous post that switch statements are more complex than they seem. The compiler is able to implement them as a decision tree (even though maybe it shouldn't). Is that everything it can do?

In this post, I'll focus on gcc. All optimizations, except the bitset lowering, translate to clang. I'll be using the -O3 optimization flag unless indicated otherwise.

I've also slightly edited gcc code and removed certain assertions for brevity.

Examples

Let us start with a simple example. A function that takes two integers x and val. Depending on the value of x, it calls a corresponding function (f0, f1, and so forth) with val as an argument and returns the result. Were we to implement this manually, we'd would use an array of function pointers, where the input serves as the index into the array.

int f0(int val);
int f1(int val);
int f2(int val);
int f3(int val);
int f4(int val);

int compact(int x, int val) {
    switch (x) {
        case 0:
            return f0(val);
        case 1:
            return f1(val);
        case 2:
            return f2(val);
        case 3:
            return f3(val);
        case 4:
            return f4(val);
        default:
            return 0;
    }
    return 0;
}

compact:
	mov	eax, edi
	mov	edi, esi
	cmp	eax, 4
	ja	.L2
	jmp	[QWORD PTR .L4[0+rax*8]]
.L4:
	.quad	.L8
	.quad	.L7
	.quad	.L6
	.quad	.L5
	.quad	.L3
.L5:
	jmp	f3
.L3:
	jmp	f4
.L8:
	jmp	f0
.L7:
	jmp	f1
.L6:
	jmp	f2
.L2:
	xor	eax, eax
	ret

The compiler agrees with this approach, and transforms the switch statement into a jump table.

Moreover, it is able to identify when the values are offset by a constant:

int f0(int val);
int f1(int val);
int f2(int val);
int f3(int val);
int f4(int val);

int compact(int x, int val) {
    switch (x) {
        case 97:
            return f0(val);
        case 98:
            return f1(val);
        case 99:
            return f2(val);
        case 100:
            return f3(val);
        case 101:
            return f4(val);
        default:
            return 0;
    }
    return 0;
}

compact:
	lea	eax, [rdi-97]
	cmp	eax, 4
	ja	.L2
	mov	edi, esi
	jmp	[QWORD PTR .L4[0+rax*8]]
.L4:
	.quad	.L8
	.quad	.L7
	.quad	.L6
	.quad	.L5
	.quad	.L3
.L5:
	jmp	f3
.L3:
	jmp	f4
.L8:
	jmp	f0
.L7:
	jmp	f1
.L6:
	jmp	f2
.L2:
	xor	eax, eax
	ret

Our intuition suggests that the more sparse the values, the less advantageous the jump table conversion is.

Through experimentation, we find that gcc considers the threshold to be around eight jump table entries per case label.

In other words, the following function compiles to a jump table:

int f0(int val);
int f1(int val);
int f2(int val);
int f3(int val);
int f4(int val);

int compact_7(int x, int val) {
    switch (x) {
        case 0:
            return f0(val);
        case 7:
            return f1(val);
        case 14:
            return f2(val);
        case 21:
            return f3(val);
        case 34:
            return f4(val);
        default:
            return 0;
    }
    return 0;
}

compact_7:
	mov	eax, edi
	mov	edi, esi
	cmp	eax, 34
	ja	.L2
	jmp	[QWORD PTR .L4[0+rax*8]]
.L4:
	.quad	.L8
	.quad	.L2
	; ...
	.quad	.L2
	.quad	.L7
	.quad	.L2
	; ...
	.quad	.L2
	.quad	.L6
	.quad	.L2
	; ...
	.quad	.L2
	.quad	.L5
	.quad	.L2
	; ...
	.quad	.L2
	.quad	.L3
.L2:
	xor	eax, eax
	ret
.L8:
	jmp	f0
.L7:
	jmp	f1
.L6:
	jmp	f2
.L5:
	jmp	f3
.L3:
	jmp	f4

But this other function is expanded as a tree of comparisons:

int f0(int val);
int f1(int val);
int f2(int val);
int f3(int val);
int f4(int val);

int sparse_8(int x, int val) {
    switch (x) {
        case 0:
            return f0(val);
        case 8:
            return f1(val);
        case 16:
            return f2(val);
        case 24:
            return f3(val);
        case 40:
            return f4(val);
        default:
            return 0;
    }
    return 0;
}

sparse_8:
	mov	eax, edi
	mov	edi, esi
	cmp	eax, 16
	je	.L2
	jle	.L10
	cmp	eax, 24
	je	.L7
	cmp	eax, 40
	jne	.L1
	jmp	f4
.L10:
	test	eax, eax
	je	.L4
	cmp	eax, 8
	jne	.L1
	jmp	f1
.L1:
	xor	eax, eax
	ret
.L4:
	jmp	f0
.L7:
	jmp	f3
.L2:
	jmp	f2

Here's the comparison tree visualized for clarity:

A couple of observations:

We've only tried equally spaced values so far. Can the compiler identify transformations that are profitable on a subset of all cases?
So far each case led to a different result. What if we had different cases with the same result?
The switch in sparse_8 could be simplified by dividing x by eight before the jump. Interestingly, gcc doesn't find this simplification.

Let's dig into 1. and 2.

Outliers

Altering our first example by including an outlier value shows that the compiler is able to recognize dense subsets of cases.

int f0(int val);
int f1(int val);
int f2(int val);
int f3(int val);
int f4(int val);
int f5(int val);

int outlier(int x, int val) {
    switch (x) {
        case 1:
            return f0(val);
        case 2:
            return f1(val);
        case 3:
            return f2(val);
        case 4:
            return f3(val);
        case 5:
            return f4(val);
        case 100:
            return f5(val);
        default:
            return 0;
    }
    return 0;
}

outlier:
	mov	eax, edi
	mov	edi, esi
	cmp	eax, 5
	jg	.L2
	test	eax, eax
	jle	.L1
	cmp	eax, 5
	ja	.L4
	jmp	[QWORD PTR .L6[0+rax*8]]
.L6:
	.quad	.L4
	.quad	.L4
	.quad	.L9
	.quad	.L8
	.quad	.L7
	.quad	.L5
.L2:
	cmp	eax, 100
	jne	.L1
	jmp	f5
.L7:
	jmp	f3
.L5:
	jmp	f4
.L9:
	jmp	f1
.L8:
	jmp	f2
.L4:
	jmp	f0
.L1:
	xor	eax, eax
	ret

The compiler emits a branch for case 100, but emits a jump table for case 1 through 5.

Contiguous

In our examples so far, each case statement lead to a different code block. Let's test what happens when multiple case labels invoke the same function.

int f1(int val);
int f2(int val);
int f3(int val);
int f4(int val);
int f5(int val);

int blocks_2(int x, int val) {
    switch (x) {
        case 0:  case 1:  case 2:  case 3:
            return f1(val);
        case 21:  ...  case 24:
            return f2(val);
        case 42:  ...  case 45:
            return f3(val);
        case 63:  ...  case 66:
            return f4(val);
        case 84:  ...  case 87:
            return f5(val);
        default:
            return 0;
    }
    return 0;
}

blocks_2:
	mov	eax, edi
	mov	edi, esi
	cmp	eax, 45
	jg	.L2
	cmp	eax, 41
	jg	.L3
	cmp	eax, 3
	jg	.L4
	test	eax, eax
	js	.L1
	jmp	f1
.L2:
	cmp	eax, 66
	jg	.L8
	cmp	eax, 62
	jle	.L1
	jmp	f4
.L4:
	sub	eax, 21
	cmp	eax, 3
	ja	.L1
	jmp	f2
.L8:
	sub	eax, 84
	cmp	eax, 3
	ja	.L1
	jmp	f5
.L3:
	jmp	f3
.L1:
	xor	eax, eax
	ret

We have 20 case labels, with values between 0 and 87. The compiler could lower this switch to a jump table with 88 entries. Our hypothesis for the heuristic is that a jump table conversion is profitable when its size is smaller than 8 * num_case_labels. Since 88 < 20 * 8, the conversion should be profitable in this case, however gcc opted to lower the switch to a sequence of comparisons.

To better understand why, let's examine a specific block of case labels.

int blocks_2(int x, int val) {
    switch (x) {
        // ...
        case 42:  case 43:  case 44:  
        case 45:
            return f3(val);
        // ...
    }
    return 0;
}

blocks_2:
	mov	eax, edi ; Move `val` to $eax
	mov	edi, esi
	cmp	eax, 45
	; If val <= 45, do _not_ jump
	jg	.L2
	cmp	eax, 41
	; If val >= 42, jump to L3
	jg	.L3
	; ...
.L2:
; ...
.L3:
        ; We reach L3 only if 42 <= val <= 45
        jmp	f3        ; Call f3(val)
; ...

The compiler realizes that the cases are contiguous and can be implemented with two comparisons instead of four. If the input of the jump table heuristic is the number of comparisons instead of case labels, things work out:

We have a total of 10 comparisons. Since 10 * 8 < 88, our revised heuristic suggests that a jump table transformation is unprofitable.

Non-contiguous

Let us try a non-contiguous case.

int f1(int val);

int bittest(int x, int val) {
    switch (x) {
        case '0':  case '2':  case '4':  
        case '5':  case '7':  case '9':  
        case 'A':  case 'C':  case 'E':
            return f1(x);
        default:
            return 0;
    }
    return 0;
}

bittest:
	lea	eax, [rdi-48]
	cmp	eax, 21
	ja	.L1
	mov	edx, 2753205  ; ???
	bt	rdx, rax
	jc	.L7
.L1:
	xor	eax, eax
	ret
.L7:
	jmp	f1

The compiler emits non-intuitive machine code. Why is there no cmp nor jump tables? Where does the constant 2753205 come from?

An hint comes from the instruction bt. According to the Intel docs, bt:

Selects the bit in a bit string (specified with the first operand, called the bit base) at the bit-position designated by the bit offset (specified by the second operand) and stores the value of the bit in the CF flag.

If we examine the binary representation of 2753205, we find that it is in fact a bitset. The bits that are set to one correspond to the ASCII values of the case labels, each value being reduced by 48.

Implementation

We've encountered several distinct lowering strategies employed by gcc. For a more in-depth understanding, we can delve into the compiler's source code.

The compiler pass that responsible for lowering switch statements is the (appropriately named) pass_lower_switch, which in turn calls analyze_switch_statement.

Since gcc source can be daunting to follow, I'll provide a step-by-step commented version below. Following this, I will break down each section to explain its functionality.

bool switch_decision_tree::analyze_switch_statement () 
{
    unsigned l = gimple_switch_num_labels (m_switch);
    auto_vec<cluster *> clusters;
    // ...

    compute_cases_per_edge ();

Some preamble code. m_switch points to a gimple object. Gimple is the internal representation used by gcc. Through this, we extract the overall count of cases and the individual counts of cases for each outgoing edge.

We also allocate a vector that will record the lowering strategy chosen for each case label.

    for (unsigned i = 1; i < l; i++)
    {
        tree elt = gimple_switch_label (m_switch, i);
	// ...
        tree low = CASE_LOW (elt);
        tree high = CASE_HIGH (elt);

        profile_probability p
            = case_edge->probability.apply_scale (1, (intptr_t) (case_edge->aux));
        clusters.quick_push (new simple_cluster (low, high, elt, case_edge->dest, p));
        // ...
    }

More setup code. We notice that gimple has a concept of high and low value for each label. That is, contiguous case labels have already joined together at this stage.

For each case label, we allocate a simple_cluster. This cluster serves as a record, capturing the range of values associated with the label, the relevant output edge, and the likelihood of this cluster being chosen. This probability comes from profile-guided optimization. By default it is uniform among case labels.

    // ...
    vec<cluster *> output = bit_test_cluster::find_bit_tests (clusters);

We transform the vector of clusters we have created. The name bit_test_cluster hints to the bitset lowering strategy we saw before.

    /* Find jump table clusters.  */
    vec<cluster *> output2;
    auto_vec<cluster *> tmp;
    // ...
    
    for (unsigned i = 0; i < output.length (); i++) {
        cluster *c = output[i];
        if (c->get_type () != SIMPLE_CASE) {
            if (!tmp.is_empty ()) {
                // ... See below for what's here
                tmp.truncate (0);
            }
            output2.safe_push (c); 
        }
        else
            tmp.safe_push (c);
    }

This section allocates a vector for the transformed cases and a scratch vector. Any case that wasn't altered during the bitset transformation process is gathered in this scratch vector. These cases are then processed and cleared whenever we come across an optimized cluster.

Essentially, in this section we are looking for sequences of case labels that couldn't be converted into bitset clusters.

                vec<cluster *> n = jump_table_cluster::find_jump_tables (tmp);
                output2.safe_splice (n);
                n.release ();

We attempt to convert these consecutive labels into a jump table and add the outcome to the output.

    /* We still can have a temporary vector to test.  */
    if (!tmp.is_empty ()) {
        vec<cluster *> n = jump_table_cluster::find_jump_tables (tmp);
        output2.safe_splice (n);
        n.release ();
    }

This is a coda for the loop above.

    // ...

    bool expanded = try_switch_expansion (output2);
    release_clusters(output2);
    return expanded;
}

Whatever is left goes through another pass, which turns the switch statement into a decision tree. The bitsets and jump table are leaves of the decision tree.

Bitsets

Let's dig further into bit_test_cluster::find_bit_tests. The high level structure of the code is as follow.

Initially, we aim to find the optimal method to bundle all clusters into bitsets. Optimality, in this context, refers to minimizing the number of bitsets used. We do so using dynamic programming.

Subsequently, we iterate over the bitsets and apply a profitability heuristic. If the bitsets meet this heuristic, we emit a bit_test_cluster; otherwise, we keep the original simple_clusters.

vec<cluster *> bit_test_cluster::find_bit_tests (vec<cluster *> &clusters)
{
    // ...

    unsigned l = clusters.length ();
    auto_vec<min_cluster_item> min;
    // ...

    min.quick_push (min_cluster_item (0, 0, 0));
    for (unsigned i = 1; i <= l; i++)
    {
        /* Set minimal # of clusters with i-th item to infinite.  */
        min.quick_push (min_cluster_item (INT_MAX, INT_MAX, INT_MAX));

        for (unsigned j = 0; j < i; j++)
        {
            if (min[j].m_count + 1 < min[i].m_count
                    && can_be_handled (clusters, j, i - 1))
                min[i] = min_cluster_item (min[j].m_count + 1, j, INT_MAX);
        }

        gcc_checking_assert (min[i].m_count != INT_MAX);
    }

For the dynamic programming, we'll use a vector of min_cluster_item structures. Each min_cluster_item has three components:

The first records how many bitsets do we need to represent the first i case labels, where i is the index in the vector.
The second records the start of the last bitset.
The third is unused.

The dynamic programming is as follows. We want to find the minimum number of bitsets to represent the first i case labels.

The base case is straightforward, a single case label can be represented by a single bitset.

Suppose we have already have found the optimal number of bitsets for the first j case label, for all $j < i$ . We define $\mathcal{Opt}_i$ to be the optimal number of splits for the first i elements. This means we already computed $\mathcal{Opt}_j$ , and are now seeking $\mathcal{Opt}_i$ .

If the range $\left(j, i \right]$ of case labels can be represented as a bitset cluster, it must be true that

$%\newcommand\Opt[0]{\textnormal{Opt}} %\newcommand\Opt[0]{\verb!Opt!} \newcommand\Opt[0]{\mathcal{Opt}} \Opt_i <= \Opt_j + 1$

Conversely, assume we have the optimal assignment for the first i labels. If the last bitset starts at label $j^*$ , this assignment is also optimal when restricted to $[0, j^*)$ . Were this not the case, we could improve the assignment on $[0, i]$ as well.

Therefore, it must be that:

$\begin{align*} F &= \left\{ j \mid \verb!can_be_handled! \left( [j, i] \right) \right\} \\[3pt] \Opt_i &= \min_{F}{ \Opt_j } + 1 \end{align*}$

These conditions are exactly what the if statement in the code above checks for.

The can_be_handled function controls whether a range of case labels can be turned into a bitset test. It tests two criteria:

The combined range of all the case labels must be smaller than the width of a register. On contemporary systems, the combined range should be less than 64.
The case labels must have at most 3 unique outgoing edges.

If there is more than one edge, we emit a cascade of bit tests.

    vec<cluster *> output;
    // ...
    
    /* Find and build the clusters.  */
    for (unsigned end = l;;)
    {
        int start = min[end].m_start;

        if (is_beneficial (clusters, start, end - 1))
        {
            bool entire = start == 0 && end == clusters.length ();
            output.safe_push (new bit_test_cluster (clusters, start, end - 1, entire));
        }
        else
            for (int i = end - 1; i >= start; i--)
                output.safe_push (clusters[i]);

        end = start;

        if (start <= 0)
            break;
    }

    output.reverse ();
    return output;
}

Now we convert only bitsets that satisfy the is_beneficial test. In the code count represents the number of case labels handled by the cluster, while uniq denotes the number of distinct outgoing edges.

bool bit_test_cluster::is_beneficial (unsigned count, unsigned uniq)
{
    return (((uniq == 1 && count >= 3)
              || (uniq == 2 && count >= 5)
              || (uniq == 3 && count >= 6)));
}

Jump tables

The logic for jump table conversion mirrors the one used for bit test conversion. It leverages dynamic programming to identify "clusters" that can be profitably converted to a jump table.

Let's delve into the code.

vec<cluster *> jump_table_cluster::find_jump_tables (vec<cluster *> &clusters)
{
    if (!is_enabled ())
        return clusters.copy ();

    unsigned l = clusters.length ();
    auto_vec<min_cluster_item> min;
    min.reserve (l + 1);

We start by preallocating a vector of min_cluster_item. We'll use this vector to keep track of the best jump table assignments for consecutive case clusters.

    min.quick_push (min_cluster_item (0, 0, 0));

We also push the base case for dynamic programming. This corresponds to using a single jump table for the whole switch.

Let's focus now on the main loop.

    unsigned HOST_WIDE_INT max_ratio = /* Used in the main heuristic. See next snippet. */;
                
    for (unsigned i = 1; i <= l; i++)
    {
        // NOTE(xoranth): Base case
        min.quick_push (min_cluster_item (INT_MAX, INT_MAX, INT_MAX));
        
        /* Pre-calculate number of comparisons for the clusters.  */
        HOST_WIDE_INT comparison_count = 0;
        for (unsigned k = 0; k <= i - 1; k++)
        {
            simple_cluster *sc = static_cast<simple_cluster *> (clusters[k]);
            // NOTE(xoranth): get_comparison_count is 2 for contiguous case 
            // labels, and 1 otherwise
            comparison_count += sc->get_comparison_count ();
        }

        for (unsigned j = 0; j < i; j++)
        {
            unsigned HOST_WIDE_INT s = min[j].m_non_jt_cases;
            if (i - j < case_values_threshold ())
                s += i - j;
            
            /* Prefer clusters with smaller number of numbers covered.  */
            if (/* Check if new split is better. See next snippet */) 
            {
                // NOTE(xoranth): Update best score.
                min[i] = min_cluster_item (min[j].m_count + 1, j,
					   s /* `s` is the number of unhandled cases */);
            }

            simple_cluster *sc = static_cast<simple_cluster *> (clusters[j]);
            comparison_count -= sc->get_comparison_count ();
        }
    }

The dynamic programming loop mirrors the one we used for bitsets. We want to find the minimum number of jump tables to represent the first i case labels.

We iterate over splits at the j-th case label, for j < i. During each iteration we track the best scoring split. The special case j == 0 represents a single jump table for the entire switch statement.

Inside this inner loop we also keep track of:

How many distinct jump tables we would need to lower the switch.
The start of the last jump table of the decomposition.
How many cases cannot be handled by a jump table. In the code, this is the accumulator s.

Point 1. and 2. are analogous to the bitset lowering pass, while 3. is new and specific to jump tables.

Namely, there is some overhead to dispatching to a jump table, due to the indirect jump and the need to check bounds. Therefore gcc requires a minimum amount of case labels to generate a jump table.

If a split would generate a jump table that is too small, the size of the jump table is added to an accumulator s, which is used as a tie-breaker between different split candidates.

To compute the scoring, we need the comparison count between j-th and the i-th case label. We compute it summing up the number of comparisons for all labels, and decreasing the count as we progress in the inner loop.

We also need the number of unhandled case labels. We determine it comparing the size of the jump table to the result of the case_values_threshold function. This function is architecture dependent and takes into account the presence of switch-case specific instructions, but on modern x64 and Arm64 machines, it defaults to 5.

unsigned int default_case_values_threshold (void) {
    return (targetm.have_casesi () ? 4 : 5);
}

Let's now focus on the main heuristic, starting with how max_ratio is computed. I've reproduced below the main loop, omitting lines unrelated to the main heuristic.

We've seen in the examples section that gcc uses the ratio between jump table entries and comparisons as a metric for the lowering the heuristic. The code below shows that distinct ratios are used when optimizing for speed (flags -O[123|fast]) or size (flag -Os). Specifically, a ratio of 8 is used when optimizing for speed, while a ratio of 2 is used when optimizing for size.

    unsigned HOST_WIDE_INT max_ratio
        = (optimize_insn_for_size_p ()
                ? param_jump_table_max_growth_ratio_for_size    // This is `2`
                : param_jump_table_max_growth_ratio_for_speed); // This is `8`

    for (unsigned i = 1; i <= l; i++)
    {
        /* Set minimal # of clusters with i-th item to infinite.  */
        min.quick_push (min_cluster_item (INT_MAX, INT_MAX, INT_MAX));

        // NOTE(xoranth): Compute comparison count, see previous snippet

        for (unsigned j = 0; j < i; j++)
        {
            // NOTE(xoranth): Compute unhandled cases, see previous snippet
            
            /* Prefer clusters with smaller number of numbers covered.  */
            if ((min[j].m_count + 1 < min[i].m_count)
               || (min[j].m_count + 1 == min[i].m_count && s < min[i].m_non_jt_cases)
               && can_be_handled (clusters, j, i - 1, max_ratio, comparison_count)) 
            {
                min[i] = min_cluster_item (min[j].m_count + 1, j,
					   s /* `s` is the number of unhandled cases */ );
            }

            // NOTE(xoranth): Update comparison count, see previous snippet
        }
    }

Now onto the main loop. Given two splits, we prefer the one that results in fewer jump tables being emitted. Or, if they would result in the same number of jump tables, we prefer the one that handles the most case labels. We also ignore splits that would result in unprofitable jump tables, where profitability is checked by the can_be_handled function.

The can_be_handled condition, checks for overflow and compares the ratio of case labels to jump table entries:

bool jump_table_cluster::can_be_handled (const vec<cluster *> &clusters,
                                         unsigned start, unsigned end,
                                         unsigned HOST_WIDE_INT max_ratio,
                                         unsigned HOST_WIDE_INT comparison_count)
{
    /* If the switch is relatively small such that the cost of one
       indirect jump on the target are higher than the cost of a
       decision tree, go with the decision tree.

       [...]

       For algorithm correctness, jump table for a single case must return
       true.  We bail out in is_beneficial if it's called just for
       a single case.  */
    if (start == end)
        return true;

    unsigned HOST_WIDE_INT range = get_range (clusters[start]->get_low (), clusters[end]->get_high ());
    /* Check for overflow.  */
    // ...

    return lhs <= max_ratio * comparison_count;
}

The coda of the function also mirrors the one for bitset clusters. It iterates over all jump tables in reverse order, and prunes the ones that fail the is_beneficial check.

    vec<cluster *> output;
    output.create (4);

    /* Find and build the clusters.  */
    for (unsigned int end = l;;)
    {
        int start = min[end].m_start;

        /* Do not allow clusters with small number of cases.  */
        if (is_beneficial (clusters, start, end - 1))
            output.safe_push (new jump_table_cluster (clusters, start, end - 1));
        else
            for (int i = end - 1; i >= start; i--)
                output.safe_push (clusters[i]);
        end = start;

        if (start <= 0) break;
    }

    output.reverse ();
    return output;
}

The is_beneficial check for jump tables aims to prune small jump tables. It compares the number of case labels to case_values_threshold.

bool jump_table_cluster::is_beneficial (const vec<cluster *> &, unsigned start, unsigned end) {
    if (start == end) return false;
    return end - start + 1 >= case_values_threshold ();
}

Binary tree

If gcc cannot emit a bitset cluster or a jump table, it falls back to a binary tree of comparison.

In emitting the tree, it takes into account the probability of each case label cluster. The probability can be obtained via profile-guided optimization, or with the __builtin_expect_with_probability builtin.

The code is pretty straightforward.

void switch_decision_tree::balance_case_nodes (case_tree_node **head, case_tree_node *parent) 
{
    case_tree_node *np = *head;
    if (np) {
        int i = 0;
        case_tree_node **npp, *left;
        profile_probability prob = profile_probability::never ();

Sum probability of all case labels.

        /* Count the number of entries on branch.  Also count the ranges.  */
        while (np) {
            i++;
            prob += np->m_c->m_prob;
            np = np->m_right;
        }

Split the case label list such that both halves have equal probability.

        if (i > 2) {
            npp = head;
            left = *npp;
	    // NOTE(xoranth): this divides prob by 2.
            profile_probability pivot_prob = prob.apply_scale (1, 2);

            while (1) {
                /* Skip nodes while their probability does not reach that amount.  */
                prob -= (*npp)->m_c->m_prob;
                if ((prob.initialized_p () && prob < pivot_prob) || ! (*npp)->m_right)
                    break;
                npp = &(*npp)->m_right;
            }

            np = *npp;
            *npp = 0;
            *head = np;
            np->m_parent = parent;
            np->m_left = left == np ? NULL : left;

Recurse in both halves to build the binary tree.

            balance_case_nodes (&np->m_left, np);
            balance_case_nodes (&np->m_right, np);
            np->m_c->m_subtree_prob = np->m_c->m_prob;
            if (np->m_left)
                np->m_c->m_subtree_prob += np->m_left->m_c->m_subtree_prob;
            if (np->m_right)
                np->m_c->m_subtree_prob += np->m_right->m_c->m_subtree_prob;
        } else {
	    // [...] Base case
        }
    }
}

Conclusion

We've seen the heuristics and lowering strategies gcc employs when compiling switch statements, including:

Compressing multiple case labels into a bitset.
Transforming the switch statement into a jump table.
Transforming the switch statement into a binary decision tree.

For anyone interested in delving deeper, I'd suggest examining the gcc source code, and reading this paper.

Finally, I want to thank B.X.V., C.P., and V.P. for their feedback when writing this post

Optimizing the `pext` perfect hash function

2023-06-11T00:00:00+00:00

This post is a followup to two posts by Wojciech Muła. One on parsing HTTP verbs, and another on using pext for perfect hashing.

The first post analyzes alternate implementations of a function string_to_verb that translates and HTTP verb, represented by a std::string_view, into an enum. It explores three different strategies:

An hardcoded trie, taken from the boost::beast library.
A SWAR strategy that, instead of reading from the input string one character at a time, reads a 32 or 64 bit prefix and switches on that.
A perfect hash function generated by the gperf tool.

The SWAR strategy turns out to be the least performant.

The three strategies are tested on two syntethic datasets:

One with a uniform distribution of all HTTP verbs.
A more realistic one, which only contains the three most frequent verbs (GET/PUT/POST).

The second post introduces a new strategy. This strategy first branches on the length of the input. Then it uses the pext Intel-specific instruction to extract a few discriminating bits from the input. Finally interprets those bits as a index into a per-length lookup table. We'll call this new strategy pext_by_len.

This second implementation allows each codepath to know at compile-time how much data it needs to load from memory. We will see that this is critical for performance.

Wojciech Muła doesn't test this new strategy on the original problem, but it finds it to perform very well on a range of similar tasks.

In this post I will:

Reproduce the results of the original post on my machine. I will add a further annotation on the number of cache misses.
Backport the new strategy to the original problem and quickly discuss its performance.
Analyze the SWAR strategy from the original post to see why it performs so badly. In particular, we will see that GCC implements the SWAR strategy as a trie, which leads to a substantial amount of cache misses.
Modify the SWAR strategy to use pext as well. This will use a global table instead of per-length ones as in pext_by_len. We'll see how some characteristics of pext synergize with the use of SWAR techniques and a global table.
Further optimize things by replacing memcmp with a more specialized implementation.

Benchmarking setup

All benchmarks were run on my machine, a ThinkPad laptop with an Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz.

Here’s what I did to set up the benchmark program:

I excluded the second core of the machine (logical cores 1 and 5) from the scheduler using the isolcpus kernel boot parameter.
I disabled turboboost and set the CPU frequency to 45% of the maximum value before running the benchmark.
I pinned the benchmark program to logical core 1 using taskset.

To ensure accurate results and compensate for cache warming effects, I ran each benchmark approximately 256'000 times. In the tables below, I report both the average timing in nanoseconds and cycles and the median absolute percent error across all runs. Additionally, I report the cache misses rate obtained through hardware perfomance counters, as it will be useful in the analysis.

Reproducing previous results

As a start, I have reproduced the benchmarks on my machine, timing both in nanoseconds and cycles, and added an annotation for cache misses.

Of the strategies in the first post, we see that SWAR implementations lag behind the boost::beast implementation, while using perfect hashing yields the best performance by far. The ranking of the results matches the results of the original post.

The backported pext_by_len strategy shines on the three verbs benchmark, and is still the second best on the full benchmark.

We see that pext_by_len has a lot more cache misses and takes more time on the full benchmark. This is the first hint that the input size branch is difficult to predict, and that removing it might yield improvements.

ns/op	err%	cyc/op	miss%	all verbs
14.91	1.6%	29.22	51.9%	`noop`
63.68	0.1%	126.02	21.7%	`reference_impl`
72.16	0.1%	142.79	10.6%	`swar`
68.98	0.2%	136.47	10.8%	`swar32`
67.39	0.2%	133.39	10.6%	`swar32 v2`
29.22	0.4%	57.66	0.6%	`perfect_hash`
44.21	0.2%	87.50	14.9%	`pext_by_len`

ns/op	err%	cyc/op	miss%	GET/PUT/POST
14.77	1.2%	29.10	46.5%	`noop`
45.76	0.7%	90.62	9.8%	`reference_impl`
50.24	0.1%	99.48	4.3%	`swar`
49.45	0.1%	97.92	3.6%	`swar32`
47.80	0.1%	94.65	4.9%	`swar32 v2`
38.07	0.9%	75.15	2.7%	`perfect_hash`
31.65	0.2%	62.65	7.2%	`pext_by_len`

Profiling the SWAR implementation

Let's analyze the SWAR 32 strategy. The code is roughly as follows:

verb string_to_verb(std::string_view v)
{
    if (v.size() < 3 or v.size() > 13)
        return verb::unknown;

    // Type punning with `memcpy`
    union {
    	char buf[13];
    	uint32_t value;
    }
    value = 0;
    memcpy(buf, v.data(), v.size());

    switch (value) {
	// ...
        case STRING_CONST('M', 'K', 'A', 'C'):      // The STRING_CONST macro turns "MKAC" into a 32 bit integer.
            if (v.substr(4) == /*MKAC*/"TIVITY"_sv)
                return verb::mkactivity;
            break;
	// ...
    }

    return verb::unknown;
}

The performance table shows a very high level of cache misses. Given that the switch statement has 33 cases, we suspect it might be the culprit.

To confirm this, we can use Linux's perf utility:

$ env NANOBENCH_ENDLESS=swar32 timeout 10 perf record -e branches,branch-misses taskset -c 1 ./benchmark_nb
[ perf record: Woken up 15 times to write data ]
[ perf record: Captured and wrote 3.583 MB perf.data (78219 samples) ]

An initial look at perf's output doesn't show any particular hotspot. To see why, let us look at how GCC compiles the switch statement:

Samples        lea  rax,[rdi-0x3]
               cmp  rax,0xa
             ↓ ja   510
               push rbp
               mov  rdx,rdi
               mov  rbp,rsi
               push rbx
     1         mov  rbx,rdi
               sub  rsp,0x18                         ; reserve space for `buf`
               mov  rdi,rsp
               mov  QWORD PTR [rsp],0x0              ; zero out the first 64 bits of `buf`
             → call memcpy@plt                       ; copy the `v` to `buf`
     6         mov  eax,DWORD PTR [rsp]              ; load the first 4 characters of `buf` into $eax
               ; This is the body of the switch statement. We compare the first 4 characters of `buf`
               ; lexicographically, to create a n-ary tree.
               ; Due to endianess, verbs are ordered by the last to the first letter.
    31         cmp  eax,0x49424e55                   ; buf[:4] == "UNBI"
             ↓ je   4f0                              ; Verb must be UNBIND or not a verb. Check the rest of buf
             ↓ ja   d0
     1         cmp  eax,0x44414548                   ; buf[:4] == "HEAD"
             ↓ je   4e0
   117       ↓ ja   90
    97         cmp  eax,0x43414b4d                   ; buf[:4] == "MKAC"
             ↓ je   488
   108       ↓ jbe  150
    28         cmp  eax,0x43454843                   ; buf[:4] == "CHEC"
             ↓ je   3a0
               cmp  eax,0x43544150                   ; buf[:4] == "PATC"
             ↓ jne  2a0
     9         cmp  BYTE PTR [rbp+0x4],0x48
    47         mov  edx,0x1c
             ↓ jne  180
             ↓ jmp  109
               nop
               ; ...

The code listing above shows the output of perf. On the left, under the Samples column, we see how many branch misses there were for each instruction. On the right, is the assembly code of the string_to_verb function.

We see that the switch statement has been expanded to a series of comparisons. Interestingly, we see not only jumps on equality (je), but also for greater (ja) or smaller-or-equal (jbe). This suggests that GCC has turned the switch statement into a decision tree. With some post-processing, we can extract the structure of the decision tree and visualize it:

Since the switch statement has been decoded into multiple comparisons, we'll need to postprocess the data and sum the count of branch misses across multiple instructions.

We also see that not all misses are recorded on a jump instruction. E.g.

;Samples       Instructions          ; Comment
; ...
    97         cmp  eax,0x43414b4d   ; buf[:4] == "MKAC"
             ↓ je   488
   108       ↓ jbe  150
; ...

This is a phenomenon known as skew. Namely, since there are multiple instructions in flight in an out-of-order processor, the CPU cannot precisely attribute the branch miss to a single instruction and can be off by an instruction or two.

To account for this we can use the following heuristic to attribute branch misses to the switch statement:

Assign each branch miss to the previous cmp instruction seen in the assembly.
Aggretate across the argument of the cmp instructions. Instructions of the kind cmp eax,0x[...] correspond to our switch statement.

When we do so, we see that more than 40% of the branch misses come from our switch statement:

kind	miss cnt	miss %
`cmp eax`	1830	47.7 %
`mov`	718	18.7 %
`cmp DWORD PTR [rsp+0x4]`	642	16.7 %
`cmp DWORD PTR [rbp+0x4]`	193	5.0 %
`jmp`	149	3.9 %
`cmp BYTE PTR [rbp+0x4]`	128	3.3 %
`cmp rbx`	54	1.4 %
`lea`	49	1.3 %
`add`	23	0.6 %
`cmp WORD PTR [rbp+0x8]`	23	0.6 %
`xor`	19	0.5 %
`cmp rax`	3	0.1 %
`cmp BYTE PTR [rbp+0x8]`	3	0.1 %

Branchless SWAR using `pext`

We want to eliminate the switch statement and replace it with a branchless solution, such as a jump table. However, the compiler didn't automatically do this for us because the values we're comparing against are sparse, which would result in a large jump table. A naive jump table of pointers would be 4 megabytes.

But what if we ignored some of the bits in our comparison value? Most of our verbs consist of uppercase ASCII letters. By examining the binary representation of uppercase ASCII letters, we can see that the upper 3 bits of each letter are always the same.

C	Binary	C	Binary
A	0b01000001	N	0b01001110
B	0b01000010	O	0b01001111
C	0b01000011	P	0b01010000
D	0b01000100	Q	0b01010001
E	0b01000101	R	0b01010010
F	0b01000110	S	0b01010011
G	0b01000111	T	0b01010100
H	0b01001000	U	0b01010101
I	0b01001001	V	0b01010110
J	0b01001010	W	0b01010111
K	0b01001011	X	0b01011000
L	0b01001100	Y	0b01011001
M	0b01001101	Z	0b01011010

Even discarding those 3 bits would reduce our jump table size to 1 kilobyte!

Ideally we would want to discard even more bits, to maximally compress our jump table. To do this, we need:

A fast way to extract the bits.
A fast to search for a subset of bits that uniquely identify a verb.

The first requirement is straightforward. pext accomplishes this in a single cycle.

The second requirement is a bit more complex. Instead of performing a naive search over all possible 32-bit unsigned integers, we can use a technique called "snoob" that allows us to enumerate only integers with a specific number of set bits. For example, a brute force search for a bitmask that extracts 7 bits needs up to four billion iterations. One that uses snoob needs only three million.

However, even with this optimization, we still face a challenge with the verbs PROPFIND and PROPPATCH, as they share a common 4-character prefix. While we could add a branch to handle this particular case, our preference is to maintain a completely branchless solution. To distinguish them we exploit the fact that PROPFIND and PROPPATCH have a different length.

A quick python oneliner confirms that the first four characters of our verb together with its length uniquely identify the verb.

Python 3.11.3 (main, May 24 2023, 00:00:00) [GCC 12.3.1 20230508 (Red Hat 12.3.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> verbs = [
    "ACL", "BIND", "CHECKOUT", "CONNECT", "COPY", "DELETE",
    "GET", "HEAD", "LINK", "LOCK", "M-SEARCH", "MERGE",
    "MKACTIVITY", "MKCALENDAR", "MKCOL", "MOVE", "NOTIFY", "OPTIONS",
    "PATCH", "POST", "PROPFIND", "PROPPATCH", "PURGE", "PUT",
    "REBIND", "REPORT", "SEARCH", "SUBSCRIBE", "TRACE", "UNBIND",
    "UNLINK", "UNLOCK", "UNSUBSCRIBE",
]

>>> prefix_and_len = {(len(v), v[:4]) for v in verbs}
>>> len(verbs) == len(prefix_and_len)
True

The length has another nice property. Our function takes an std::string_view as a parameter. A string_view is implemented as a struct of pointer to data and length. Due to the x64 calling convention, the length will already be passed in a register (rdi in this case). We can confirm this by examining the assembly code once more:

; Here we implement the check
; if (v.size() < 3 or v.size() > 13)
;       return verb::unknown;
; $rdi is the length of `v`
lea  rax,[rdi-0x3] 			; $rax = v.size() - 3ul
; Notice that the previous subtraction *underflows* if `v.size() < 3`
cmp  rax,0xa				; check if `$rax > 10`. If `v.size() < 3`, `$rax` is a very big number and this check still succeeds
ja   510
[...]

We can simply "mix in" the string length by xor-ing with the output of pext:

uint32_t find_mask() {
    uint32_t best_mask = 0u;
    for (uint32_t mask = 0b1'111'111u; snoob(mask) > mask; mask = snoob(mask)) {
        // Our lookup function. Reads up to the first 4 bytes, does `pext` with mask and then XORs with the size
        auto lam = [=](string_view sv) { 
            uint32_t val = 0u;
            memcpy(&val, sv.data(), min(sv.size(), sizeof(val)));
            return _pext_u32(val, mask) ^ sv.size();
        };

        // VERBS_MAP is an hash map of verb to number
        // `is_uniq` checks that all keys in VERBS_MAP hash to different indices
        if (is_uniq<128ul>(VERBS_MAP, lam)) {
            best_mask = mask;
            break;
        }
    }

    return best_mask; // Will be zero if no good mask is found
}

This runs in ~50ms and quickly finds an appropriate bitmask:

$ ./find_mask 
Found mask: 33687137

The implementation is very straightforward. For simplicity, I generate the lookup table at startup. An alternative would be to use code generation.

static const uint32_t MASK = 33687137u;
uint32_t lut_idx (std::string_view sv) { 
    uint32_t val = 0u;
    memcpy(&val, sv.data(), std::min(sv.size(), sizeof(val)));
    return _pext_u32(val, MASK) ^ sv.size();
};


using lut_elem_t = uint8_t; //verb;
static const std::array<lut_elem_t, 128> LOOKUP_TABLE = []() {
    using namespace boost::beast::http;
    std::array<lut_elem_t, 128> lookup_table {(lut_elem_t) verb::unknown};

    for (unsigned int as_int = 0; as_int < verb_count; ++as_int) {
        std::string_view sv = as_string((verb) as_int);
        lookup_table[lut_idx(sv)] = (lut_elem_t) as_int;
    }

    return lookup_table;
}();

verb
string_to_verb(std::string_view v)
{
    if (v.size() < 3 or v.size() > 13) {
        return verb::unknown;
    }

    verb res = (verb) LOOKUP_TABLE[lut_idx(v)];
    if (v == as_string(res)) {
        return res;
    }
    return verb::unknown;
}

With this change, we see greatly improved results. All branch misses are gone. On the full benchmark, this implementation (dubbed pext) is the second fastest, beaten only the hash table generated by gperf.

ns/op	err%	cyc/op	miss%	all verbs
63.68	0.1%	126.02	21.7%	`reference_impl`
68.98	0.2%	136.47	10.8%	`swar32`
29.22	0.4%	57.66	0.6%	`perfect_hash`
44.21	0.2%	87.50	14.9%	`pext_by_len`
39.74	0.2%	78.70	0.4%	`pext`

On the common verbs benchmark results are less good, but still improved.

ns/op	err%	cyc/op	miss%	GET/PUT/POST
45.76	0.7%	90.62	9.8%	`reference_impl`
49.45	0.1%	97.92	3.6%	`swar32`
38.07	0.9%	75.15	2.7%	`perfect_hash`
31.65	0.2%	62.65	7.2%	`pext_by_len`
43.15	0.2%	85.44	1.8%	`pext`

Here our solution is beaten both by gperf and by the per-length pext solution.

Memcpy woes

Despite the improvements in the previous section, our solution is still dominated by gperf. Can we do better?

Let us profile again with perf, this time focusing on cycles spent on each instruction.

; ...
  0.05 │      mov    eax,0x4
  1.51 │      cmp    rdi,rax
  0.01 │      cmovbe rax,rdi                      ; Compute min(sv.size(), 4);
  0.04 │      mov    rbp,rsi
  1.81 │      mov    DWORD PTR [rsp+0xc],0x0      ; Zero out buf
  0.01 │      mov    rbx,rdi
  0.01 │      mov    esi,eax			  ; Put min(sv.size(), 4) into $esi
  0.03 │      test   eax,eax
       │    ↓ je     45
  2.63 │      xor    eax,eax
  	      ; GCC has inlined std::memcpy. This is a for loop that copies the bytes
  	      ; from the input string one by one into `buf`
  	      ; $eax is the counter, $esi is min(sv.size(), 4), $rbp is a pointer to the input string
  	      ; and $rsp+0xc is a pointer to buf
  2.35 │34:   mov    edx,eax
  2.80 │      movzx  ecx,BYTE PTR [rbp+rdx*1+0x0]
  2.72 │      inc    eax
  2.62 │      mov    BYTE PTR [rsp+rdx*1+0xc],cl
  2.42 │      cmp    eax,esi
  0.05 │    ↑ jb     34
  	      ; After we have finished copying the input string into buf, copy buf back into a register
 38.93 │45:   mov    eax,DWORD PTR [rsp+0xc]
  0.08 │      mov    edx,0x2020661
  8.38 │      pext   eax,eax,edx
  2.96 │      xor    eax,ebx
 14.18 │      movzx  r12d,BYTE PTR [rax+0x424620]
; ...

We see that gcc is loading data from memory in a non-optimal way. It has inlined memcpy into a for loop. This is likely because we are passing a variable length to memcpy.

We can get gcc to generate better assembly by giving it an hint. We can use a union to load the first three characters, and load the fourth character conditionally on input length. This is undefined behaviour in C++, but is explicitely supported by gcc.

uint32_t load_sv(std::string_view sv) {
    union {
        char as_char[4];
        uint32_t res;
    };
    as_char[0] = sv[0];
    as_char[1] = sv[1];
    as_char[2] = sv[2];
    as_char[3] = (sv.size() > 3ul) ? sv[3] : '\0';
    return res;
}

With this change, our implementation is competitive with gperf on both benchmarks.

ns/op	err%	cyc/op	miss%	all verbs
29.22	0.4%	57.66	0.6%	`perfect_hash`
44.21	0.2%	87.50	14.9%	`pext_by_len`
39.74	0.2%	78.70	0.4%	`pext`
29.75	1.4%	58.71	0.5%	`pext v2`

It is still behind the per-length pext solution on the common verbs benchmark though:

ns/op	err%	cyc/op	miss%	GET/PUT/POST
38.07	0.9%	75.15	2.7%	`perfect_hash`
31.65	0.2%	62.65	7.2%	`pext_by_len`
43.15	0.2%	85.44	1.8%	`pext`
38.67	0.4%	76.47	2.5%	`pext v2`

Unsafe memcpy optimizations

A look at the assembly shows that we are still using 3 different movs to load the beginning of our string into memory.

; Samples
  8.11 │      movzx edx,BYTE PTR [rsi+0x2]  ; Load the first two bytes
  0.38 │      movzx eax,WORD PTR [rsi]      ; Load the third byte
  1.02 │      shl   edx,0x10
  1.28 │      or    eax,edx                 ; Combine the first three bytes
  5.70 │      mov   rbx,rdi
       │      mov   rbp,rsi
       │      xor   edx,edx
  2.11 │      cmp   rdi,0x3                 ; Check if v.size() is bigger than 3
  0.10 │    ↓ je    2c                      ; If v.size() is bigger than 3, load the fourth byte. Otherwise $edx will be zero
  6.14 │      movzx edx,BYTE PTR [rsi+0x3]
  6.49 │2c:   shl   edx,0x18
  1.05 │      or    eax,edx                 ; Combine with the fourth byte

Ideally, we would like to load the four bytes in a single mov $edx, DWORD PTR [rsi], but we can't. The issue is the following: a string_view is not guaranteed to be null-terminated. Therefore, for inputs of length 3 a DWORD read might touch uninitialized memory.

That said, we might be able to do better by borrowing some tricks from glibc.

We've seen that the memcmp implementation reads beyond the end of the array when the input pointers are far from a page boundary. Since we are loading only 4 bytes from one pointer, the page boundary check simplifies to:

$\newcommand\PageSize[0]{\textnormal{PageSize}} \newcommand\BitAnd[0]{\mathbin{\&}} \newcommand\Ptr[1]{{p}_{#1}} \Ptr{1} \BitAnd \left( \PageSize \right) < \left( \PageSize - 4\right)$

But we can simplify even further. Notice that, by the point we are loading our input string into a register, we have already checked that the length of the string is bigger than three. Therefore the only case where we could cause a pagefault is when the input pointer is exactly 3 bytes before a page boundary.

If the input pointer is at least 4 bytes from a page boundary, a DWORD read doesn't cross the page boundary.
If the input pointer is 2 or less bytes from a page boundary, a read does cross a page boundary, but we know that the next page is already allocated, since we know that at least 3 bytes after the input pointer are allocated.

This further refines the page boundary check to:

$\Ptr{1} \BitAnd \left( \PageSize \right) \neq \left( \PageSize - 3\right)$

At a first glance, inputs of length three still need extra handling. We are loading a byte beyond the end of the string that we need to zero. We could do so branchlessly using the bzhi instruction:

Initially, it would appear that inputs of length three require additional handling. For inputs of length three we load a byte beyond the end of the string, which needs to be zeroed out. One way to do so is the bzhi instruction:

uint32_t data = _bzhi_u32(load(v.data()), 8ul * v.size());

However, an even better approach is to do nothing. In other words, not masking the extra byte.

To understand why this works, observe that pext discards input bits, but does not mix them. As a result, we can distinguish the output bits of pext into two groups:

Bits extracted from the first three bytes of the input string (the prefix).
Bits extracted from the fourth byte of the input string (the suffix).

We can also split the inputs into two kinds:

Inputs of length three.
All other inputs.

Bits in the prefix are sufficient to distinguish strings of lenghth three by construction. However the fourth byte only affects bits in the suffix. Therefore it is enough to ensure that bits extracted from the prefix distinguish strings of length three from keys of length four or more.

That would guarantee that altering the bits from the suffix cannot cause a collision. We can simply add multiple entries in our lookup table for strings of length three, one for each combination of bits in the suffix.

This is true for our mask, and can be enforced by exhaustive checking in our find_mask function.

With this improvement, we beat gperf on the full benchmark!

ns/op	err%	cyc/op	miss%	all verbs
29.22	0.4%	57.66	0.6%	`perfect_hash`
44.21	0.2%	87.50	14.9%	`pext_by_len`
39.74	0.2%	78.70	0.4%	`pext`
29.75	1.4%	58.71	0.5%	`pext v2`
26.25	0.7%	51.71	0.0%	`pext v3`

We also beat pext_by_len on the restricted benchmark.

ns/op	err%	cyc/op	miss%	GET/PUT/POST
38.07	0.9%	75.15	2.7%	`perfect_hash`
31.65	0.2%	62.65	7.2%	`pext_by_len`
43.15	0.2%	85.44	1.8%	`pext`
38.67	0.4%	76.47	2.5%	`pext v2`
26.06	0.2%	51.51	0.0%	`pext v3`

Optimizing the final check

Can we do even better? Running the latest version of our benchmark under perf shows that the bottleneck is now memcmp:

$ perf report -M intel --input=pext-v3-cycles-gcc.data
Samples: 118K of event 'cycles:pppu', Event count (approx.): 51634360904
Overhead  Command       Shared Object         Symbol
  29.87%  benchmark_nb  benchmark_nb          [.] ankerl::nanobench::Bench::run<run_benchmark<boost::beast::http:
  29.50%  benchmark_nb  libc.so.6             [.] __memcmp_avx2_movbe
  23.83%  benchmark_nb  benchmark_nb          [.] pext::string_to_verb_v3
  12.06%  benchmark_nb  benchmark_nb          [.] boost::beast::http::as_string
   2.02%  benchmark_nb  benchmark_nb          [.] memcmp@plt

That is, the bottleneck is not hashing our input string view, but checking that our hash has not given us a false positive:

    verb
    string_to_verb_v3(std::string_view v)
    {
        if (v.size() < 3 or v.size() > 13) {
            return verb::unknown;
        }

        verb res = (verb) LOOKUP_TABLE[lut_idx_v3(v)];

	// string_view::operator!= calls `memcmp` under the hood
        if (v != as_string(res)) { // BOTTLENECK!
            return verb::unknown;
        } else {
            return res;
        }
    }

We notice several inefficiencies in our false positive check:

Perf shows that memcmp is not getting inlined. Our page alignment check logic is inspired by memcmp's implementation, so it is likely that inlining memcmp the compiler would be able to elide redundant computations.
memcmp has to branch on the size of the input arrays. But we know that our input string size is between 3 and 13 characters. Therefore we can use 16 bytes loads and omit all branching.

memcmp needs to check that it is safe to read beyond the end of the array for both inputs. But we control one of the input arrays, and we can just pad the input with nulls.
memcmp has to find the first different byte, we don't. So we can use vptest instead of vmovmskb and save one instruction.
Our lookup table implementation has an inefficient layout. We are using a lookup table of string_views, but a string_view is itself a pair of (pointer, length). This leads to a double indirection on lookup. A better implementation would keep the length and the data side by side, in the same cacheline.

After implementing the changes above, we see a substantial speedup on both benchmarks:

ns/op	err%	cyc/op	miss%	all verbs
29.22	0.4%	57.66	0.6%	`perfect_hash`
44.21	0.2%	87.50	14.9%	`pext_by_len`
39.74	0.2%	78.70	0.4%	`pext`
29.75	1.4%	58.71	0.5%	`pext v2`
26.25	0.7%	51.71	0.0%	`pext v3`
22.04	1.1%	43.40	0.0%	`pext v4`

ns/op	err%	cyc/op	miss%	GET/PUT/POST
38.07	0.9%	75.15	2.7%	`perfect_hash`
31.65	0.2%	62.65	7.2%	`pext_by_len`
43.15	0.2%	85.44	1.8%	`pext`
38.67	0.4%	76.47	2.5%	`pext v2`
26.06	0.2%	51.51	0.0%	`pext v3`
21.98	1.0%	43.28	0.0%	`pext v4`

Interpreting the results

When we began optimizing, we first examined the full benchmark and concentrated our efforts on reducing branch misses. This ultimately improved performance on the full benchmark.

However, when it came to the restricted benchmark, branch misses were not as significant of an issue. We'd expect the restricted benchmark to be more friendly to a branchy implementation, as fewer cases means easier to predict branches.

Then why did our branchless implementation improve the restricted benchmark, even surpassing pext_by_len?

One way to interpret the results is the following. When using a well-performing perfect hash, hashing is no longer the limiting factor. Rather memory loads and memcmp become bottlenecks.

To address this, the pext_by_len implementation uses branching based on input size, and optimizes memcpy based on size.

Our implementation takes a different approach. We specializes memcpy to always compare 16 bytes, and branch to a slow path only when the input is aligned to page size.

But a branch on pointer alignment is more predictable than one on input size. Assuming the input pointer is a random number, the slow path will be hit approximately 0.37% of the time:

$\frac{15}{4096} = 0.003662109375 \approx 0.37\%$

Therefore, the trade-off between the two options is a net win.

Acknowledgments

I want to thank A.Y., N.S., V.P. and in particular B.X.V. for their help in writing and editing this post. I also want to thank Wojciech Muła for his analysis of the problem.

All my code is available here.

A look inside `memcmp` on Intel AVX2 hardware.

2023-04-27T00:00:00+00:00

memcmp is a C standard library function that compares two arrays lexicographically. It will be familiar to C programmers, and it is often used as a building block for string operations in other languages. For instance, C++ std::string_view comparison uses memcmp under the hood in some implementations.

While it can be implemented in C using a simple for loop, Glibc provides an optimized implementation written in assembly. By digging into this implementation, we can learn a lot about low level optimization.

This post will procede in four steps. First I'll provide a quick summary of what memcmp does. Then we will delve into the assembly specialization for x86-64 AVX2, dividing it into three logical sections:

Handling of small arrays, where "small" denotes arrays smaller than 32 bytes.
Handling of arrays ranging from 32 to 256 bytes.
Handling of large arrays.

The section dedicated to small arrays is the longest, as the logic for small arrays is the least straightforward. 32 bytes is smaller than the size of an AVX2 SIMD register, and usually handling small arrays requires either:

Writing scalar code.
Some masking support from the hardware, that AVX2 doesn't provide.

We will see how the implementation avoids the need for either.

Summary of `memcmp`

According to cppreference, memcmp:

Reinterprets the objects pointed to by lhs and rhs as arrays of unsigned char and compares the first count bytes of these arrays. The comparison is done lexicographically.

The sign of the result is the sign of the difference between the values of the first pair of bytes (both interpreted as unsigned char) that differ in the objects being compared.

A pure C implementation of such a function is very straightforward, we just loop over the array till we find a difference. If we do find one, we return the signed difference of the two bytes:

int simple_memcmp(const void* p1, const void* p2, size_t count) {
    const unsigned char* lhs = p1;
    const unsigned char* rhs = p2;
    
    for (size_t i = 0; i < count; ++i)
        if (lhs[i] != rhs[i]) 
            return (int) lhs[i] - (int) rhs[i];

    return 0;
}

In the x64 calling convention, the first arguments are passed in registers. So p1 and p2 will be passed respectively in $rsi and $rdi, while the count will be passed in $rdx.

Let's now dig into the assembly.

Efficient handling of short arrays

Immediately after the entry point, we branch to a specialized routine that handles arrays smaller than 32 bytes.

    cmp        rdx,0x20          ; 0x20 is 32 in hexadecimal
    jb         2e0

Following the jump, we see some pretty unusual code.

     ; If count is smaller or equal to 1, go to a specialized routine.
2e0: cmp        edx,0x1
     jbe        360
     mov        eax,edi
     or         eax,esi          ; $rax, $rsi are p1, p2
     and        eax,0xfff        ; 0xfff is 4095 in hexadecimal
     cmp        eax,0xfe0        ; 0xfe0 is 4096 - 32 in hexadecimal
     jg         320

To unravel this code, let's consider a simple case. To compare two arrays which are 32 bytes long, we can load each to a YMM register using AVX2 loadu instruction. We can then compare the two YMM registers in a single instruction.

However, if count is less than 32 bytes, we still load 32 bytes of data. We'll then have to "discard" the comparison results for the excess bytes. Since we are reading beyond the end of the arrays, we need to be careful not to touch and unmapped page and cause a pagefault.

To avoid this, we must check if the next page is allocated. To be safe, we should check that bytes $[p_i, p_i+32)$ don't cross a page boundary. Pages are always aligned the page size, so we just need to check that the pointer modulo the page size isn't too close to the page boundary. In math terms:

$\newcommand\PageSize[0]{\textnormal{PageSize}} \newcommand\BitAnd[0]{\mathbin{\&}} \newcommand\BitOr[0]{\mathbin{\|}} \newcommand\BitMod[0]{\mathbin{\%}} \newcommand\Ptr[1]{{p}_{#1}} \begin{align*} \Ptr{i}' &< \PageSize - 32 \end{align*}$

for $\Ptr{i}' = \Ptr{i} \pmod \PageSize$ and $i \in \Set{1, 2}$ .

We can express both conditions as a single expression taking the maximum of $\Ptr{1}'$ , $\Ptr{2}'$ :

$\begin{align*} \PageSize - 32 > max(\Ptr{1}', \Ptr{2}') \end{align*}$

This has the advantage that $max(\Ptr{1}', \Ptr{2}')$ can be bound from above.

$\def\arraystretch{1.4em} \newcommand\htmlRef[1]{\href{###1}{\text{\footnotesize #1}}} \begin{array}{rcl|l} \max(\Ptr{1}', \Ptr{2}') &\lt& \Ptr{1}' \BitOr \Ptr{2}' & \text{\htmlRef{binorineq}} \\ &=& (\Ptr{1} \BitAnd (\PageSize - 1)) \BitOr ( \Ptr{2} \BitAnd (\PageSize - 1)) & \text{\htmlRef{pow2eq}} \\ &=& (\Ptr{1} \BitOr \Ptr{2}) \BitAnd (\PageSize - 1) & \text{\htmlRef{disteq}} \end{array}$

On x86-64 hardware, only page sizes of 4Kb, 2Mb and 1Gb are allowed. To be safe, we'll assume that the page size is 4096 bytes. With that in mind, we can establish the following condition, which is enough to prevent a page fault:

$(\Ptr{1} \BitOr \Ptr{2}) \BitAnd 4095 < 4064$

This condition corresponds to the assembly above.

Fast comparison using AVX2 and BMI

After loading the two arrays into ymm registers, we need to:

Compare them for equality.
Mask or discard the comparisons of characters beyond the end of the array.

AVX2 doesn't have byte-level masked loads, and loading a mask likely requires a lookup table. Therefore using AVX2 instructions to mask excessive characters is inefficient.

A better approach is to use vpcmpeqb for comparison, followed by vpmovmskb to move the comparison result to a general purpose register and finally clear the excess bits using bit manipulation tricks.

vpcmpeqb compares each byte in its two inputs, and returns -1 if they are equal and 0 if they differ. vpmovmskb sets the n-th bit of its destination register to the sign bit of the n-th byte in its input ymm register.

Combining vpcmpeqb and vpmovmskb will return an integer where the n-th bit is one if the n-th bytes of the input arrays are equal, and zero otherwise.

However, since memcmp returns 0 if the inputs are equal, we need to bitwise negate the output of vpmovmskb.

Finally we need to mask the excess bits. The most straightforward way is to use a shift to create a bitmask, like in this C example:

uint32_t zero_high_bits(uint32_t input, uint32_t count) {
    uint32_t mask = ~(1 << count);      // E.g. 3 becomes 0b00011111...1
    return input & mask;                // Keep only the count lowest bits
}

Fortunately most Intel and AMD provide an instruction that does all the above. The instruction is bzhi from the BMI2 extension. It has a throughput of 1 ins/cycle on both Skylake and Zen, which is extremely good. Also, while BMI2 and AVX2 are separate extensions, every CPU produced that supports AVX2 supports BMI.

This leads to the following assembly

     vmovdqu    ymm2,YMMWORD PTR [rsi]
     vpcmpeqb   ymm2,ymm2,YMMWORD PTR [rdi]
     vpmovmskb  eax,ymm2
     inc        eax
     bzhi       edx,eax,edx
     ; `bzhi` sets the flags if the input is not zero. 
     jne        e0
     xor        eax,eax
     vzeroupper
     ret

This shows the entire process for two example strings:

Handling the unequal case

If after clearing out the extra bits, edx is not zero, we found a difference. In particular, if the first $m$ bytes are identical, edx will contain $m$ zeros followed by a 1.

Fortunately, BMI contains another useful instruction that helps us. tzcnt counts how many trailing zeros a register has.

$\newcommand\tzcnt[0]{\textnormal{tzcnt}} \tzcnt ( 0\textnormal{b}\underbrace{00000}_{m} 1 \textnormal{xxxx} ) = m$

Now that we have the index of the first differing byte, we take their signed difference to obtain the result.

 e0:  tzcnt      eax,eax
      movzx      ecx,BYTE PTR [rsi+rax*1]
      movzx      eax,BYTE PTR [rdi+rax*1]
      sub        eax,ecx
      vzeroupper
      ret

Avoiding the loop epilogue

Now that we know that the input arrays are bigger than 32 characters, there's no need to worry about pagefaults or clearing "extra" bits.

We can load, compare and vpmovmskb in increments of 32 bytes till we have less than 32 bytes left to compare. The first four steps of the logic are inlined.

     cmp        rdx,0x40
     jbe        264                         ; Go to EPILOGUE 1
     vmovdqu    ymm2,YMMWORD PTR [rsi+0x20]
     vpcmpeqb   ymm2,ymm2,YMMWORD PTR [rdi+0x20]
     vpmovmskb  eax,ymm2
     inc        eax
     jne        100
     cmp        rdx,0x80
     jbe        250                         ; Go to EPILOGUE 2
     vmovdqu    ymm3,YMMWORD PTR [rsi+0x40]
     vpcmpeqb   ymm3,ymm3,YMMWORD PTR [rdi+0x40]
     vpmovmskb  eax,ymm3
     inc        eax
     jne        120                         ; Go to EPILOGUE 3
     vmovdqu    ymm4,YMMWORD PTR [rsi+0x60]
     vpcmpeqb   ymm4,ymm4,YMMWORD PTR [rdi+0x60]
     vpmovmskb  ecx,ymm4
     inc        ecx
     jne        15b                         ; Go to EPILOGUE 4
     cmp        rdx,0x100
     ja         170                         ; Go to main loop PROLOGUE

The assembly above can only handle arrays whose length is a multiple of 32. What can we do when that's not the case?

Two options come to mind. First, we can write a scalar loop to handle the remaining bytes. Alternatively, we can reuse the logic that is used for small arrays.

But there's an even better solution: read and test the last 32 bytes of the array. By doing this, we eliminate the need to read beyond the end of the array. Then, unlike the small arrays logic, we read only memory we know is allocated. In particular, we don't need a page alignment check.

It's important to note that although there is overlap between this final iteration and the previous ones, it doesn't affect the end result. Our objective is to locate the position of the first difference in the entire array (if any). But we know that the overlapping bytes contain no difference, and therefore don't affect the outcome.

The only place where we need to be careful is if a difference is found. In that case, tzcnt will tell us the offset from end_of_array - 32, instead than begin_of_array + num_iter * 32. We can adjust the final stanza to account for that:

     ; EPILOGUE 1
264: vmovdqu    ymm1,YMMWORD PTR [rsi+rdx*1-0x20]
     vpcmpeqb   ymm1,ymm1,YMMWORD PTR [rdi+rdx*1-0x20]
     vpmovmskb  eax,ymm1
     inc        eax
     jne        2c0                         ; If a difference found, go to tzcnt stanza
     vzeroupper
     ret                                    ; No difference, return 0
     ; ...

2c0: tzcnt      eax,eax
     ; $eax now contains the offset of the first difference from `end_of_array - 32`
     add        eax,edx                     ; $edx is the size of the array.
     ; $rsi is the beginning of the first array
     ; $rax now contains `size_of_array + offset`
     ; 0x20 is 32
     ; So $rsi + $rax is `begin_of_array + size_of_array + offset`
     ; But `begin_of_array + size_of_array = end_of_array` so
     ; So $rsi + $rax - 0x20 = end_of_array - 32 + offset
     ; which is the position of the first different byte
     movzx      ecx,BYTE PTR [rsi+rax*1-0x20]
     ; Ditto for the second array
     movzx      eax,BYTE PTR [rdi+rax*1-0x20]
     sub        eax,ecx
     vzeroupper
     ret                                    ; Return difference of first unequal bytes

Handling big arrays

Finally we reach the main logic. This logic is called only for arrays bigger than 256 elements. It is optimized with the idea that there will be many iterations before we find the difference. In particular, it has a prologue and a main loop.

     ; PROLOGUE
170: lea        rdx,[rdi+rdx*1-0x80]    ; Turn count into ptr to end of array
     sub        rsi,rdi                 ; Difference of two input ptrs
     and        rdi,0xffffffffffffffe0  ; Align to 32 bytes
     sub        rdi,0xffffffffffffff80
     ; LOOP
180: vmovdqu    ymm1,YMMWORD PTR [rsi+rdi*1]
     vpcmpeqb   ymm1,ymm1,YMMWORD PTR [rdi]
     vmovdqu    ymm2,YMMWORD PTR [rsi+rdi*1+0x20]
     vpcmpeqb   ymm2,ymm2,YMMWORD PTR [rdi+0x20]
     vmovdqu    ymm3,YMMWORD PTR [rsi+rdi*1+0x40]
     vpcmpeqb   ymm3,ymm3,YMMWORD PTR [rdi+0x40]
     vmovdqu    ymm4,YMMWORD PTR [rsi+rdi*1+0x60]
     vpcmpeqb   ymm4,ymm4,YMMWORD PTR [rdi+0x60]
     vpand      ymm5,ymm2,ymm1
     vpand      ymm6,ymm4,ymm3
     vpand      ymm7,ymm6,ymm5          ; Tree reduction
     vpmovmskb  ecx,ymm7
     inc        ecx
     jne        140                     ; Found differece
     sub        rdi,0xffffffffffffff80  ; Add 32 * 4 to first array ptr
     cmp        rdi,rdx
     jb         180                     ; Go to LOOP

The loop optimizations are pretty standard. The loop is unrolled, having four comparisons at a time in order to offset the overhead from pointer manipulation and to better exploit instruction level parallelism. Also, instead of checking the result of each comparison individually, we bitwise and them together first and check only the aggregated result. This exploits the fact that vpand can run on three CPU ports, while vpmovmskb can only run on one.

More interesting is the prologue. There's two optimizations of note here:

The pointer to the second array is replaced by the difference of second to first. Loads from the second array can still be expressed using x86-64 base+offset memory indexing. E.g.
```
    vmovdqu    ymm1,YMMWORD PTR [p1 + (p2 - p1) * 1]
```
This saves us from having to increment the second pointer inside the loop.
The pointer to the first array is rounded down to 32 bytes. This ensures that at least half of the memory loads will be aligned, resulting in a higher throughput.

Wrapup

We have investigated an optimized implementation of memcmp which uses SIMD instructions extensively. We have several techniques to:

Quickly find differences in an array using SIMD and BMI2 instructions.
Handle data smaller than the SIMD register size.
Eliminating the need for a separate scalar epilogue to handle arrays of uneven sizes.
Extract instruction level parallelism from loops.

For people that want to dig even deeper, I'd recommend looking at the assembly of various glibc routines. For a reference on SIMD instructions on Intel and their latencies and throughput, I would reccomend the excellent uops.info.

Finally, I want to thank A.Y., B.X.V., C.P., N.S. and V.P. for their invaluable feedback in writing this post

We used the inequality $\max(a, b) < a \BitOr b$ .

We used the following equality $a \pmod {2^k} = a \BitAnd (2^k - 1)$ and the fact that the page size is a power of two.

We used the distributivity property $(a \BitAnd c) \BitOr (b \BitAnd c) = (a \BitOr b) \BitAnd c$ .