Previously, in the “Escaping the Giant Switch” episode, we learned how to break one enormous interpreter compilation unit into many independently compiled bytecode handlers.

This left us with an appealing hypothesis: if we can find the locally optimal assembly for each handler, putting those handlers together should give us a globally optimal interpreter.

We put that hypothesis to the test with the Crema interpreter, with one constraint: its Java source code had to stay maintainable.

As compiler engineers, we occasionally entertain a slightly arrogant belief: programmers should write straightforward Java, and our extremely smart compiler should handle the rest.

But when the emitted assembly falls short, who should change?

Should the compiler learn a more general optimization, or should the interpreter expose a fact that only it knows?

Act I: The Expandables

Let us start with iconst_0, one of the simplest Java bytecodes. It pushes the integer zero onto the operand stack.

For this example, we need only a compact model of the interpreter frame and stack state:

class Frame {
    /* Raw primitive values in local-variable and operand-stack slots. */
    final long[]   prims;
    /* Object references in local-variable and operand-stack slots. */
    final Object[] refs;
    /* Bytecode of the method being interpreted. */
    final byte[]   code;
}

class Stack {
    /* Next free operand-stack slot in Frame.prims/refs. */
    int top;
}

static int iconst0Handler(int bci, Frame frame, Stack stack) {
    int slot = stack.top;
    frame.prims[slot] = 0;
    stack.top = slot + 1;
    return bci + 1;
}

Here the bci parameter is the bytecode index, the byte offset of the current instruction within the method’s bytecode array.

When frame and stack cross the handler boundary as ordinary arguments, the relevant fast path assembly is:

    ; regBCI:   incoming bci
    ; regFrame: incoming frame
    ; regStack: incoming stack
    mov     regPrims, QWORD PTR [regFrame + OFFSET_prims]              ; load frame.prims
▶   mov     regTop32, DWORD PTR [regStack + OFFSET_top]                ; load stack.top
    test    regPrims, regPrims                                         ; is prims null?
    je      null_prims
    cmp     regTop32, DWORD PTR [regPrims + OFFSET_length]             ; is top out of bounds?
    jae     out_of_bounds
    mov     QWORD PTR [regPrims + regTop * 8 + OFFSET_arrayStart], 0   ; prims[slot] = 0
    inc     regTop32                                                   ; advance the stack top
▶   mov     DWORD PTR [regStack + OFFSET_top], regTop32                ; write back stack.top
    inc     regBCI32                                                   ; advance to the next bytecode
    ret
    ; Here and below, reg*32 denotes the lower 32 bits of the corresponding register.

For a bytecode that merely pushes zero, this is quite an entourage. Updating stack.top, for example, requires loading the field into a register, incrementing it, and storing it back.

In the previous article, we taught objects to cross compilation boundaries in pieces, while keeping the handler’s Java signature intact. We call this argument expansion, and distinguish two forms according to whether the original object also crosses the boundary.

With virtual expansion, the fields travel independently, without the original object reference. The generated stub reconstructs the object around those values and calls the original handler; after inlining, escape analysis can eliminate the reconstructed object.

With materialized expansion, the original object reference travels alongside selected fields. The handler receives those fields directly while retaining access to the backing object and its other fields.

For Frame, we choose materialized expansion and recruit prims , refs, and code as the expandables. For Stack, we choose virtual expansion. It sends top to join the expandables but stays behind itself.
Together, these choices change the handler’s boundary tuple from <bci, frame, stack> to <bci, frame, frame.prims, frame.refs, frame.code, stack.top>.

The tuple grew. The assembly shrank.

With the specialized calling convention, these values arrive in the registers where the handler needs them:

    ; regBCI:   incoming bci
    ; regFrame: incoming frame
    ; regPrims: incoming frame.prims
    ; regRefs:  incoming frame.refs
    ; regCode:  incoming frame.code
    ; regTop:   incoming stack.top
▶   test    regPrims, regPrims                                         ; is prims null?
▶   je      null_prims
    cmp     regTop32, DWORD PTR [regPrims + OFFSET_length]             ; is top out of bounds?
    jae     out_of_bounds
    mov     regSlot32, regTop32                                        ; zero-extend top into regSlot
    mov     QWORD PTR [regPrims + regSlot * 8 + OFFSET_arrayStart], 0  ; prims[slot] = 0
    inc     regTop32                                                   ; advance the stack top
    inc     regBCI32                                                   ; advance to the next bytecode
    ; Threaded dispatch replaces the return with a jump to the next handler
    jmp     tail_call_target(regCode, regBCI)

stack.top now arrives in the lower 32 bits of regTop, and its updated value stays there for the next handler.

The field traffic is now gone, but the null check and bounds check are still there.

Native Image’s global data-flow analysis can prove some references non-null. However, when that proof is unavailable, the compiler must retain the null check.

Fortunately, argument expansion lets us specify non-nullness as part of the handler interface. This contract applies both to direct reference arguments such as frame and to expanded reference fields such as frame.prims.
In Graal terminology, the parameter receives a non-null stamp, which describes the values that parameter can hold. The handler compilation can use that stamp to eliminate the internal null check.

The check still has to happen. Just not in every handler. The compiler inserts null checks before handler calls from the dispatch loop and before threaded tail calls. For a field such as Frame.prims, which is final, the outgoing expanded argument is still the same non-null reference. The compiler can therefore fold away the null check before the outgoing threaded tail call too.

With argument expansion and the non-null contract, the relevant threaded-handler assembly is:

    ; regBCI:   incoming bci
    ; regFrame: incoming frame       (non-null)
    ; regPrims: incoming frame.prims (non-null)
    ; regRefs:  incoming frame.refs  (non-null)
    ; regCode:  incoming frame.code  (non-null)
    ; regTop:   incoming stack.top
▶   cmp     regTop32, DWORD PTR [regPrims + OFFSET_length]             ; is top out of bounds?
▶   jae     out_of_bounds
    mov     regSlot32, regTop32                                        ; zero-extend top into regSlot
    mov     QWORD PTR [regPrims + regSlot * 8 + OFFSET_arrayStart], 0  ; prims[slot] = 0
    inc     regTop32                                                   ; advance the stack top
    inc     regBCI32                                                   ; advance to the next bytecode
    jmp     tail_call_target(regCode, regBCI)

For these unchanged references, argument expansion and the non-null contract effectively hoist the checks out of the threaded handler chain.

Less assembly. Same Java.

So far, the division of labor works.

But what about the bounds check?

Act II: Handler, Compiler, Verifier, Witness

The comparison between slot and prims.length is there for a reason. Java requires lastore to throw an ArrayIndexOutOfBoundsException when the index is outside the array bounds.

That requirement comes from the host Java code implementing the interpreter; the guest instruction iconst_0 merely pushes zero and prescribes no such exception. An out-of-bounds access here would mean an interpreter invariant had been broken, but unless the compiler can prove that slot is valid, the check stays.

Keeping that check costs more than the comparison alone. For instance, the cold exception path may keep values alive on the fast path, increasing register pressure and potentially causing spills.

So how can the compiler prove that slot is a valid index and avoid this cost?

If slot is constant and the array length is known, the proof can be immediate; if, in a counted loop, range analysis proves that every value of slot stays within the array, the check can be eliminated; if an earlier guard established 0 <= slot && slot < prims.length, the compiler can reuse it for every access it dominates.

All of these optimizations require the compiler to derive the proof from information in its graph.

Our tiny handler, however, is compiled as a standalone unit. Its graph receives stack.top and frame.prims as independent inputs, with no proof that the resulting slot is a valid index into the array.

Fortunately, the Java bytecode verifier can serve as a witness outside that graph.

Before the JVM executes a method’s bytecode, the bytecode verifier establishes a valid frame shape at every reachable bytecode index. Among other properties, it proves that every operand-stack operation stays within the max_stack recorded in the method’s class-file Code attribute. As long as an interpreter allocates operand-stack storage for that same max_stack and implements the verified operations faithfully, every operand-stack access is within the allocated storage.

In that sense, verification plays the role of a guard that dominates the entire interpretation. But it lives outside the handler’s compilation unit, so the compiler cannot reuse it as a dominating guard in the handler graph.

Could we use the same stamp-based contract that previously removed the null checks?

Just as a stamp can express non-nullness, it can also express a constant integer range such as [0, 63]. However, the required range is [0, prims.length), whose upper bound is a runtime value rather than a compiler constant. The stamp system cannot express this relationship between two runtime values.

For an ordinary Java array access, without a proof in the graph, the compiler must retain precisely the comparison and branch we want to remove.

Alternatively, we can make the verifier-established invariant an implicit assumption in the interpreter source, replacing the checked array write with an Unsafe-backed access:

static int iconst0Handler(int bci, Frame frame, Stack stack) {
    int slot = stack.top;
    UNSAFE.putLong(frame.prims, slot * 8L + OFFSET_arrayStart, 0);
    stack.top = slot + 1;
    return bci + 1;
}

The bounds check no longer enters the handler’s graph, and the relevant assembly becomes:

    ; regBCI:   incoming bci
    ; regFrame: incoming frame       (non-null)
    ; regPrims: incoming frame.prims (non-null)
    ; regRefs:  incoming frame.refs  (non-null)
    ; regCode:  incoming frame.code  (non-null)
    ; regTop:   incoming stack.top
▶   movsxd  regSlot, regTop32                                          ; sign-extend top into regSlot
    mov     QWORD PTR [regPrims + regSlot * 8 + OFFSET_arrayStart], 0  ; prims[slot] = 0
    inc     regTop32                                                   ; advance the stack top
    inc     regBCI32                                                   ; advance to the next bytecode
    jmp     tail_call_target(regCode, regBCI)

The array-length load, conditional branch, and cold exception edge have all disappeared. Correctness now rests on verification, frame allocation, and the interpreter preserving the verified stack discipline.

One Move Too Many

The bounds check is gone. But the zero-extension has quietly become a sign-extension.

Why?

Consider an input stack.top = -1, giving us slot = -1. Sign-extension preserves that value as 0xFFFFFFFF_FFFFFFFF, while zero-extension produces 0x00000000_FFFFFFFF. Using the latter in the Unsafe access would compute the wrong address, potentially causing a segmentation fault or memory corruption.

Of course, this input cannot occur in a correctly implemented interpreter executing verified bytecode. But that guarantee remains implicit in the source, so the compiler cannot use it to rule out negative inputs and must preserve Java’s signed conversion.

Before the switch to Unsafe, the compiler had to insert a bounds check.
Once that check passed, it knew that slot was non-negative at the array access. For a non-negative int, sign-extension and zero-extension produce the same 64-bit value, so it could replace the former with the latter. On processors that support move elimination, the resulting register move can be handled during register renaming, avoiding an execution step that sign-extension would require.

Zero-extension also gives the compiler an opportunity to eliminate the move entirely — a small prize, but one that can be significant in benchmarks such as SPECjvm. On x64, writing a 32-bit register also clears its upper 32 bits. If the compiler knows that the instruction that produced slot already did this, it may use that register directly as the address index.

Then why couldn’t the compiler eliminate the zero-extending move in our expanded handler before the switch to Unsafe?

Our stack.top arrives as an expanded parameter, so its producing instruction is outside this compilation unit. The calling convention promises an int in the lower 32 bits of regTop; it does not promise that the upper 32 bits are zero. Consider another input, stack.top = 1. The incoming register could contain 0x00000001_00000001, with the correct int value in its lower half. Using the entire register as an address index would include those non-zero upper bits, potentially causing a segmentation fault or memory corruption.

Proving that stack.top already arrives in a register containing its correctly extended, full-width value would require reasoning across compilation boundaries. We can avoid that proof by making the full-width value part of the handler interface. Changing the index’s representation to long preserves the invariants established by verification. Plus, with Unsafe, we are no longer constrained by Java’s int array indices, since the access takes a long byte offset.

We therefore changed both stack.top and bci to long, so each arrives as a 64-bit index ready for addressing. Here too, the verifier testifies that branch targets stay within the bytecode array. Neither index needs widening in each handler.

Our running example now becomes:

class Stack {
    long top;
}

static long iconst0Handler(long bci, Frame frame, Stack stack) {
    long slot = stack.top;
    UNSAFE.putLong(frame.prims, slot * 8L + OFFSET_arrayStart, 0);
    stack.top = slot + 1;
    return bci + 1;
}

The incoming stack top can serve directly as the address index:

    ; regBCI:   incoming bci         (long)
    ; regFrame: incoming frame       (non-null)
    ; regPrims: incoming frame.prims (non-null)
    ; regRefs:  incoming frame.refs  (non-null)
    ; regCode:  incoming frame.code  (non-null)
    ; regTop:   incoming stack.top   (long)
    mov     QWORD PTR [regPrims + regTop * 8 + OFFSET_arrayStart], 0
    inc     regTop
    inc     regBCI
    jmp     tail_call_target(regCode, regBCI)  ; long bci also avoids widening in the unshown dispatch code

Store zero. Advance the indices. Dispatch.

That is more like it.

Supplementary Testimony

The verifier establishes more than stack capacity.
For an interpreter, those guarantees fall into two useful categories:

• Access bounds: operand-stack and local-variable accesses stay within the storage allocated from max_stack and max_locals; bytecode operands form complete instructions; branch and exception-handler targets name valid instruction boundaries; and constant-pool indices refer to valid entries.
• Type safety: every instruction consumes and produces compatible verification types; stack-manipulation forms are legal; field accesses, invocations, and returns agree with their descriptors; and uninitialized references obey the constructor rules.

The type-safety proofs are normally implicit in the handler implementation. For example, an iadd handler expects two integers on the operand stack without checking their runtime types.

Other type guarantees belong to a particular bytecode site rather than to every execution of an instruction. How can we make those facts useful to a shared handler?

Consider baload, which operates on either a byte[] or a boolean[].
Its shared handler contains a type check to distinguish the two array kinds at run time. At a given BCI, however, verification fixes the non-null array operand’s type as either byte[] or boolean[]; it cannot admit both at the same site. Different baload sites may have different array types, so this knowledge cannot be baked into a single shared handler. The guest-array null and bounds checks are still required.

We could use the verifier’s result to rewrite baload at that BCI to a byte[]- or boolean[]-specific opcode before execution begins.

Alternatively, we can record this site-specific information through runtime quickening.

On the first execution with a non-null array, the generic handler observes its type and rewrites that BCI to select one of two shared specialized handlers. A single runtime observation would not normally justify permanent specialization; the verifier’s BCI-specific guarantee makes it sufficient here. Subsequent executions use the specialized handler without repeating the type check. The specialized opcode carries the site’s type information into handler selection.

The same pattern applies when the initial execution establishes information that does not come from verification. For a field access, the original handler resolves the field, performs the opcode-specific access checks, and then quickens the opcode. The quickened handler retrieves the cached resolved entry without repeating resolution or those access checks.

To summarize, the compiler must generate correct code for every input its interface permits, while the interpreter’s external guarantees rule out some of those inputs. When the compiler cannot see those guarantees, we can still use them to choose a different implementation.

We have done so through unchecked accesses, wider parameters, and specialized opcodes, putting external knowledge to work without teaching the compiler its original proofs.

But how far should we push this?

Act III: How to Train Your Compiler

Let us return to the iconst_0 example. Can we make its assembly even simpler?

Suppose we carried the address of the next primitive element instead of its logical slot. In C-like code, initializing the pointer and pushing zero would look something like this:

int64_t *stack = &frame.prims[top];  // Initialize from the logical top outside of the handlers
...
*stack = 0;                          // Push zero
stack++;                             // Advance by eight bytes

With the pointer carried in regStack across threaded tail calls, pushing zero and advancing it could compile to:

    mov     QWORD PTR [regStack], 0
    add     regStack, 8

The simpler addressing mode does not necessarily improve performance.
On Intel Skylake, both the original indexed store and this direct-address store execute as two micro-operations. The direct-address form can use an additional store-address port, which may help under contention, but does not guarantee a cycle saving.

The interpreter’s Java implementation, meanwhile, becomes more cumbersome. The stack top no longer names a logical slot. To address both primitive and reference arrays directly, we now have to maintain two pointers, advancing each according to its array’s element size. We must also keep these pointers into the managed heap valid across garbage collection and safepoints.

Making the source so much harder to read is a poor trade when we cannot even establish a performance gain. The choice becomes less clear when keeping the source tidy leaves avoidable work in the generated code.

The Pursuit of Constants

Consider putfield, whose handler branches according to the kind of field it stores into. An int occupies one stack slot, while a long occupies two, so their values start at stack.top — 1 and stack.top-2, respectively.
In either case, the receiver sits in the preceding slot.

Using our Frame and Stack abstraction, a natural implementation is to share the slot calculations and receiver lookup before branching on the field kind:

static long putfieldHandler(long bci, Frame frame, Stack stack) {
    JavaField field           = resolveField(frame, bci);
    JavaKind  kind            = field.kind();
    long      valueSlot       = stack.top - kind.slotCount();
    long      receiverSlot    = valueSlot - 1;
    Object    receiver        = UNSAFE.getReference(frame.refs, receiverSlot * OFFSET_referenceScale + OFFSET_arrayStart);
    Object    nonNullReceiver = nullCheck(receiver);
    long      offset          = field.offset();
    boolean   isVolatile      = field.isVolatile();

    switch (kind) {
        case Int -> {
            // OFFSET_endian is 4 on big-endian targets, otherwise 0.
            int value = UNSAFE.getInt(frame.prims, valueSlot * 8L + OFFSET_arrayStart + OFFSET_endian);
            if (isVolatile) {
                UNSAFE.putIntVolatile(nonNullReceiver, offset, value);
            } else {
                UNSAFE.putInt(nonNullReceiver, offset, value);
            }
        }
        case Long -> {
            long value = UNSAFE.getLong(frame.prims, valueSlot * 8L + OFFSET_arrayStart);
            if (isVolatile) {
                UNSAFE.putLongVolatile(nonNullReceiver, offset, value);
            } else {
                UNSAFE.putLong(nonNullReceiver, offset, value);
            }
        }
        // Other valid field kinds omitted.
        default -> throw shouldNotReachHere();
    }
    UNSAFE.putReference(frame.refs, receiverSlot * OFFSET_referenceScale + OFFSET_arrayStart, null);
    stack.top = receiverSlot;
    return bci + 3;
}

The relevant x64 assembly snippet leading into the switch is the following.

    ; regBCI:   incoming bci         (long)
    ; regFrame: incoming frame       (non-null)
    ; regPrims: incoming frame.prims (non-null)
    ; regRefs:  incoming frame.refs  (non-null)
    ; regCode:  incoming frame.code  (non-null)
    ; regTop:   incoming stack.top   (long)
    ; regKindObject holds the decompressed JavaKind reference; setup and null checks omitted.
    ; ... receiver decompression omitted ...
▶   mov     QWORD PTR [rsp + 0x28], regFieldObject  ; spill decompressed field reference
    movsxd  regSlots,      DWORD PTR [regKindObject + OFFSET_slotCount]
    mov     regValueSlot, regTop
    sub     regValueSlot, regSlots          ; valueSlot = top - kind.slotCount()
    mov     regReceiver32, DWORD PTR [regRefs + regValueSlot * 4 + OFFSET_arrayStart - 4]
    mov     regReceiverSlot, regValueSlot
    dec     regReceiverSlot                 ; receiverSlot = valueSlot - 1
    test    regReceiver32, regReceiver32    ; is receiver null?
    je      null_receiver

▶   mov     QWORD PTR [rsp + 0x60], regTop          ; spill stack top
    ; ... field offset/modifier loads omitted ...
    mov     regKind32,     DWORD PTR [regKindObject + OFFSET_ordinal]
    ; ... ordinal bounds check omitted ...
▶   mov     QWORD PTR [rsp + 0x10], regKindObject   ; spill decompressed JavaKind reference
    ; ... ordinal jump table, typed stores, and threaded dispatch omitted ...

At first glance, that is exactly what we asked for. It loads the slot count from JavaKind, then computes valueSlot and receiverSlot before the switch. The receiver load folds the subtraction into its address, but receiverSlot is still computed separately for the later update to stack.top.

Yet once we enter a switch arm, the slot count is fixed, giving us an opportunity for constant folding. In the int case, for example, replacing the slot count with one would simplify valueSlot to stack.top-1 and receiverSlot to stack.top-2. The memory accesses could then absorb those constant offsets into their address displacements. For the receiver, that gives DWORD PTR [regRefs + regTop * 4 + OFFSET_arrayStart-8]. The stack update could subtract two directly from top at the end.

That suggests a manual rewrite. Move the shared setup into the branches and specialize it for each kind:

static long putfieldHandler(long bci, Frame frame, Stack stack) {
    JavaField field = resolveField(frame, bci);
    JavaKind kind = field.kind();
    // Verification and linking guarantee a valid field kind.
    switch (kind) {
        case Int -> {
            long receiverSlot = stack.top - 2;
            long valueSlot = stack.top - 1;
            Object receiver = UNSAFE.getReference(frame.refs, receiverSlot * OFFSET_referenceScale + OFFSET_arrayStart);
            Object nonNullReceiver = nullCheck(receiver);
            int value = UNSAFE.getInt(frame.prims, valueSlot * 8L + OFFSET_arrayStart + OFFSET_endian);
            if (field.isVolatile()) {
                UNSAFE.putIntVolatile(nonNullReceiver, field.offset(), value);
            } else {
                UNSAFE.putInt(nonNullReceiver, field.offset(), value);
            }
            UNSAFE.putReference(frame.refs, receiverSlot * OFFSET_referenceScale + OFFSET_arrayStart, null);
            stack.top = receiverSlot;
        }
        case Long -> {
            long receiverSlot = stack.top - 3;
            long valueSlot = stack.top - 2;
            Object receiver = UNSAFE.getReference(frame.refs, receiverSlot * OFFSET_referenceScale + OFFSET_arrayStart);
            Object nonNullReceiver = nullCheck(receiver);
            long value = UNSAFE.getLong(frame.prims, valueSlot * 8L + OFFSET_arrayStart);
            if (field.isVolatile()) {
                UNSAFE.putLongVolatile(nonNullReceiver, field.offset(), value);
            } else {
                UNSAFE.putLong(nonNullReceiver, field.offset(), value);
            }
            UNSAFE.putReference(frame.refs, receiverSlot * OFFSET_referenceScale + OFFSET_arrayStart, null);
            stack.top = receiverSlot;
        }
        // Other valid field kinds omitted.
        default -> throw shouldNotReachHere();
    }
    return bci + 3;
}

For the int case, the resulting addresses are:

    ; Int arm after duplicating the setup; other instructions omitted.
    ; Same incoming registers; regTop still holds the incoming stack.top.
    ; OFFSET_arrayStart = 8 and reference scale = 4 in this build.
    mov     regReceiver32, DWORD PTR [regRefs + regTop * 4]  ; disp = 8 - 4 * 2
    ; ... receiver decompression omitted ...
    test    regReceiver32, regReceiver32                   ; is receiver null?
    je      null_receiver_int
    mov     regValue32,    DWORD PTR [regPrims + regTop * 8] ; disp = 8 - 8 * 1
    ; ... stack update, volatility test, and field store omitted ...

The other arms fold their constant slot offsets into the addresses in the same way.

More Java. Less assembly.

But why can’t the compiler do this for us?

Graal can fold a final field read when the object containing the field is a known constant. But javac lowers an enum switch to an integer switch derived from ordinal(), typically through a synthetic mapping array. Entering the int case constrains this integer; it does not turn kind into the constant JavaKind.Int in the compiler graph. The slotCount read used to compute valueSlot therefore still has a dynamic receiver.

Teaching the compiler to map an ordinal back to its JavaKind instance could enable branch-local constant folding. That would require trusting the enum’s canonical-instance invariant, much as final-field folding trusts that a field’s value will not change.

HotSpot’s C2 compiler does not trust arbitrary instance final fields for constant folding by default, though it makes exceptions for certain classes. Its experimental -XX:+TrustFinalNonStaticFields option broadens that trust. Graal uses a different policy.

Even if the compiler could establish the enum identity in each branch, that alone would solve only half of our problem. It could specialize valueSlot within each branch, but the shared receiver access would still use the dynamic slot count before the switch. Eliminating that load would require duplicating the receiver access and null check into the branches as well.
The default branch would need to preserve the original calculation and null check before throwing its own exception. This is possible, but it requires teaching the compiler when that duplication is worth the extra code.

Let us leave that as a feature request for the compiler.

A Little Too Helpful

While we admire the result of constant folding, the compiler may be undoing some of our careful duplication. For instance, each branch calls field.offset(), which reads the same field offset from the resolved metadata object. Writing that call inside each arm does not necessarily keep the load there.

Graal can turn an eligible field load into a floating read: a load whose position is determined by its dependencies rather than fixed to a point in the control flow. Equivalent reads with matching dependencies can then become one shared node through global value numbering.

As long as the value remains shared, the scheduler must place it where it is available to all its uses. When those uses lie in different switch arms, this means placing it in a common dominator before the switch, provided its dependencies allow it. Sharing the load saves instruction bytes, but its result now has to stay live across the dispatch. In the duplicated handler, the field offset and modifier reads have moved before the switch:

    ; Before kind dispatch; regField holds the compressed field reference.
    ; save values before reusing their registers.
▶   mov     QWORD PTR [rsp + 0x20], regKindObject   ; spill decompressed JavaKind reference
▶   mov     DWORD PTR [rsp + 0x1c], regKindRef32    ; spill compressed JavaKind reference
▶   mov     QWORD PTR [rsp + 0x28], regFieldObject  ; spill decompressed field reference
▶   mov     QWORD PTR [rsp + 0x30], regFrame        ; spill frame reference
    movsxd  regOffset,  DWORD PTR [regHeapBase + regField * 8 + OFFSET_fieldOffset]
    mov     regFlags32, 0x50df
    and     regFlags16, WORD PTR [regHeapBase + regField * 8 + OFFSET_modifiers]
    movzx   regFlags32, regFlags16
    mov     regValueSlot, regTop
    dec     regValueSlot                   ; top - 1
    lea     regReceiverSlot, [regTop - 2]  ; shared by the one-slot arms
    lea     regNextBCI, [regBCI + 3]
    ; ... ordinal range check and jump-table dispatch ...

The same can happen to arithmetic. Across the one-slot cases — boolean, byte, char, and others — the same stack.top-2 calculation appears for the outgoing stack top. If these calculations become one shared node, it must execute before the switch, even though not every arm needs the result.

Graal’s partial-redundancy-aware scheduler can undo this sharing by duplicating a computation into the branches that need it. Its general heuristic favors splitting when doing so reduces execution frequency. The stack.top-3 calculation shared by the long and double arms illustrates this: the scheduler splits it back into those arms, avoiding the calculation on the more frequent one-slot paths. But the shared stack.top-2 calculation serves most of the execution, so splitting it offers too little frequency saving — even though it would shorten live ranges.

Keeping the computation shared is a reasonable choice until registers become scarce. The jump-table dispatch generated from the switch needs its selector and a register for the target address, alongside the interpreter state carried to the next handler. Hoisted loads and slot calculations add to that live set even though only one arm will execute. If their overlapping lifetimes exceed the available registers, the allocator must spill something, which is unfortunately the case in many handlers with complicated branching logic. The compiler’s good intentions in sharing computations then turn into extra stores and reloads.

How do we train the compiler to reduce spilling?

We can start with compiler directives. One of my favorites is GraalDirectives.opaque, which returns the same runtime value but hides it behind a distinct compiler node. Wrapping field and stack.top separately in each switch arm prevents the compiler from simply treating the resulting expressions as shared work. With each copy used only in its own arm, the scheduler can keep the dependent reads and arithmetic there.

Our int case now looks like this:

case Int -> {
    field = GraalDirectives.opaque(field);
    long top = GraalDirectives.opaque(stack.top);
    long receiverSlot = top - 2;
    long valueSlot = top - 1;
    // ... receiver check, field store, and stack update using field and these slots ...
}

It works. It’s just ugly.

Every arm now describes both the bytecode’s behavior and how we want it compiled. Worse, the desired placement depends on the compiler’s treatment of these directives, so a compiler change may send us back to rearrange the source.

We can instead teach the compiler the tradeoff directly: accept a little more generated code in exchange for shorter live ranges.

We restrict read hoisting and sharing, and teach the scheduler to favor separate copies of arithmetic in the consuming branches. The field metadata loads and slot calculations now stay inside the arms, without the source directives:

    ; No spills before the switch; dispatch omitted.

int_case:
▶   mov     QWORD PTR [rsp + 0x18], regKindObject   ; spill decompressed JavaKind reference
▶   mov     QWORD PTR [rsp + 0x20], regFieldObject  ; spill decompressed field reference
    mov     regReceiver32, DWORD PTR [regRefs + regTop * 4]
    lea     regReceiverSlot, [regTop - 2]
    test    regReceiver32, regReceiver32
    je      null_receiver_int
    ; ... value load and modifier handling omitted ...
    mov     regOffset32, DWORD PTR [regHeapBase + regField * 8 + OFFSET_fieldOffset]
    ; ... volatility check, receiver decompression, offset widening, and field store ...

The long arm computes top-3 instead. Neither arm carries the other’s stack update through the switch.

These choices deliberately work against optimizations we built for general-purpose compilation. Sharing work is usually sensible, but our tiny handlers have tightly constrained register interfaces. We therefore special-case handler stubs, putting this policy in the compiler instead of repeating it through directives in every handler.

Das Leben der Anderen

Even then, spills can remain.

The two stores inside the int arm preserve values for the null-check exception path and a later safepoint slow path, omitted from the assembly excerpt.

Neither slow path uses these values for its actual work. Why keep them alive?

At a call site — including the runtime call on a safepoint poll’s slow path — the compiler may need to describe the caller’s Java execution state for deoptimization, debugging, or runtime stack inspection. Graal records this execution state in a frame state. That can include local variables the call itself never uses, potentially keeping them alive beyond their last executable use. It can even force a reference to be decompressed solely for bookkeeping, although subsequent field accesses need only its compressed form.

In our putfield example, both slow paths retain the decompressed field and JavaKind references alongside other parameters and locals.

Keeping a value alive does not require spilling it immediately. There are enough registers for the live values just before this null check. But these references remain live alongside the temporaries needed for the field store, increasing register pressure inside the arm.

On handler fast paths, Graal’s register allocator favors spilling values with no remaining register-requiring uses when it needs to free a register. Neither decompressed reference has another executable use, making both attractive spill candidates. Here, the resulting interval assignments place the stores before the null check, so the fast path pays too.

One source-level response is to stop giving intermediate results their own locals. For putfield, we could remove the kind local and write the switch directly:

switch (field.kind()) {
    // ... same arms ...
}

The same approach becomes less pleasant in invokestatic:

static long invokeStaticHandler(long bci, Frame frame, Stack stack) {
    CallInfo callInfo = resolveCall(bci, frame);
    Object retObj = VM.dispatchCall(
            popArguments(
                    callInfo.argumentCount == 0
                            ? EMPTY_ARRAY
                            : new Object[callInfo.argumentCount],
                    frame,
                    stack,
                    callInfo),
            bci,
            frame,
            callInfo);
    // ... return handling and next BCI omitted ...
}

To avoid an intermediate local, we have crammed argument allocation, preparation, and invocation into one expression — an anti-pattern adopted to satisfy the compiler’s bookkeeping.

But our handler stubs cannot deoptimize. Unless debugging or runtime inspection requires their local values, we can discard that bookkeeping.

We therefore teach the compiler which frame-state values these stubs actually need, and prune the rest before register allocation. We retain the BCI used to report the guest’s execution position during stack walking, along with lock information. Live references needed by the GC remain tracked.

The two spill stores disappear from the int arm:

int_case:
    mov     regReceiver32, DWORD PTR [regRefs + regTop * 4]
    test    regReceiver32, regReceiver32
    je      null_receiver_int
    ; ... value load, field store, stack update, and continuation ...

Now a local variable can go back to being a name that helps us read the code.

One Spill Flew Over the Sibling’s Nest

Return to the version before frame-state pruning. The remaining mystery is why the int arm spills both references as soon as it begins.

At the null check, 11 live intervals overlap, against 12 allocatable general-purpose registers in our x64 build. Pressure rises later in the field-store sequence, where 13 intervals overlap.

Why pay for two spills on entry?

The answer starts in another arm.

Graal’s linear-scan allocator lays out basic blocks and instructions along a one-dimensional axis, then represents each value’s lifetime as ranges on that axis. Those intervals can contain gaps and be split into children with different register or stack assignments. Allocation follows this linear order, while liveness analysis and reconciliation between blocks still use the control-flow graph.

In the putfield compilation, the allocator visits the Object arm before the int arm. There, 14 live intervals compete for 12 registers. The decompressed kind and field references become spill victims, freeing registers for the remaining work. The stores appear in the blocks where that pressure rises.

But their stack assignments extend into the later int arm too. The switch still supplies both references in registers. The Object arm’s stores never execute on the int path, so data-flow resolution must insert new stores to satisfy the int arm’s inherited stack assignments. It places them at the arm’s entrance, before the null check.

The int arm has register pressure of its own, but these two stack assignments originate in its sibling. It inherits both spills — and pays for them on entry, even though it may not need to free both registers.

We therefore teach the compiler to give a sibling another chance at a register.

When a spilled interval extends into another block, the allocator checks the locations supplied by its actual predecessors. If the stack assignment does not arrive on every incoming path, it splits the interval at that block’s entrance and allocates the new child separately.

The new child prefers the register already supplied by an incoming path. This lets the arms make different choices for the same value.

That independence costs compilation time. Each split requires predecessor checks and creates another interval to allocate, which may itself be split again. Across a large method, this extra work can erode the speed that makes linear scan attractive.

Yet again, we give handler stubs special treatment. After so many amendments to the compiler’s usual rules, an uncomfortable question presents itself.

Why not write assembly?

Act IV: Everything Everywhere All at Once

Well, that is not for me to decide.

When my boss Doug Simon and the team started the Maxine VM project, the idea was Java on Java.

I never convinced them to add __asm__ to Java, so I need an excuse to convince myself.

Luckily, there is one.

Our iconst_0 and putfield examples already use a Stack object, but the handlers still access the backing arrays directly.

Let us encapsulate those accesses in the stack abstraction. Instead of:

class Stack {
    long top;
}

static long iconst0Handler(long bci, Frame frame, Stack stack) {
    long slot = stack.top;
    UNSAFE.putLong(frame.prims, slot * 8L + OFFSET_arrayStart, 0);
    stack.top = slot + 1;
    return bci + 1;
}

We can write:

class Stack {
    long top;

    void pushInt(Frame frame, int value) {
        UNSAFE.putLong(frame.prims, top * 8L + OFFSET_arrayStart, value);
        top++;
    }

    int popInt(Frame frame) {
        top--;
        return (int) UNSAFE.getLong(frame.prims, top * 8L + OFFSET_arrayStart);
    }
}

static long iconst0Handler(long bci, Frame frame, Stack stack) {
    stack.pushInt(frame, 0);
    return bci + 1;
}

Stack now provides an overlay on the frame, while virtual argument expansion keeps that overlay in the source without carrying an object into the emitted assembly.

The handler no longer needs to know where zero goes.

We can now subclass Stack and override pushInt, caching values in instance fields instead of writing them immediately to the backing array.
Again, argument expansion carries these fields between handlers in registers.

That gives us top-of-stack caching.

One Field, Many Roles

Let us return to the simplest cache from the previous article, with two primitive fields, tos1 and tos2, and a template variable tosDepth counting the cached JVM slots. We can view these fields as an extension of Frame.prims: they hold stack slots just like the array elements, except they live in registers. top marks the end of the materialized stack; the cached slots extend it. Object references remain in the backing frame, so before pushing one, we write back the cached values and reset tosDepth to zero.

The following table shows a selection of handlers, how they interpret the cached slots, and what they leave for the next handler.

The types represented by tos1 and tos2 change across handlers, but we need no runtime type tests. The bytecode verifier establishes the operand types, and the cache protocol preserves their representation. The remaining question is how to use those bits without paying for a conversion.

We declare tos1 and tos2 as long, wide enough to hold the bits of any primitive value. Reading an int through an ordinary Java cast introduces a narrowing operation into the compiler graph. Depending on how it is lowered and allocated, that operation can introduce a register move and a separate register for the narrowed value.

For an integer operand, however, our protocol already places the value in the register’s lower 32 bits. We use a compiler directive to read those bits as an int, without introducing a separate narrowing operation. The upper bits need not be zero; a 32-bit use simply does not read them. Writing an integer back into a cache field needs the matching treatment: preserve its low 32 bits without requesting Java’s signed widening to long. unpackInt ignores the upper bits; packInt leaves them unspecified. On x64, the 32-bit arithmetic already supplies the bits we need.

That leaves two more mysterious entries in the table: Preserved and Unconstrained. A preserved value must reach the next handler unchanged, but passing it through requires no knowledge of its type.
For nop(tosDepth=2), both fields have Schrödinger’s type: their bits have a meaning, but we have no reason to look. Matching incoming and outgoing ABI registers lets the allocator carry these values through without moves, provided it keeps those registers intact.

Passing the bits through is enough until we need to write them back to the frame. Then we must know how much stack space they occupy. Our current state machine avoids that ambiguity by counting slots rather than values: a fully cached long uses both cache fields, including its padding.
Caching two longs would require a state that tells us more. We will come back to this later.

An unconstrained field need not preserve even its bits. For nop(tosDepth=1), we can pass whatever bits happen to occupy the register for tos2. We express this freedom through compiler directives, allowing the allocator to reuse that register inside the handler without preserving or restoring its contents. We need not even put zero there before the next handler.

We have already complained about directives cluttering the source.
Fortunately, we can sweep this machinery under the Stack overlay’s carpet. The handler compilation knows the concrete stack implementation, so it can inline the overridden operations.

// Directive names in this illustrative example describe their intended contracts.
class CachedStack extends Stack {
    static final int NO_SLOTS_CACHED = 0;
    static final int ONE_SLOT_CACHED = 1;
    static final int TWO_SLOTS_CACHED = 2;

    long tos1;
    long tos2;
    int tosDepth = NO_SLOTS_CACHED;

    @Override
    void pushInt(Frame frame, int value) {
        switch (tosDepth) {
            case ONE_SLOT_CACHED -> {
                tos2 = GraalDirectives.packInt(value);
                tosDepth = TWO_SLOTS_CACHED;
            }
            case TWO_SLOTS_CACHED -> {
                UNSAFE.putLong(frame.prims, top * 8L + OFFSET_arrayStart, tos1);
                top++;
                tos1 = tos2;
                tos2 = GraalDirectives.packInt(value);
                // The cache stays full: 2 → 2.
            }
            default -> {
                assert tosDepth == NO_SLOTS_CACHED;
                tos1 = GraalDirectives.packInt(value);
                tosDepth = ONE_SLOT_CACHED;
            }
        }
    }

    @Override
    int popInt(Frame frame) {
        return switch (tosDepth) {
            case ONE_SLOT_CACHED -> {
                tosDepth = NO_SLOTS_CACHED;
                long result = tos1;
                tos1 = GraalDirectives.unconstrainedValue();
                yield GraalDirectives.unpackInt(result);
            }
            case TWO_SLOTS_CACHED -> {
                tosDepth = ONE_SLOT_CACHED;
                long result = tos2;
                tos2 = GraalDirectives.unconstrainedValue();
                yield GraalDirectives.unpackInt(result);
            }
            default -> {
                assert tosDepth == NO_SLOTS_CACHED;
                top--;
                yield (int) UNSAFE.getLong(frame.prims, top * 8L + OFFSET_arrayStart);
            }
        };
    }
}

The handler sees only stack operations:

static long iadd(long bci, Frame frame, Stack stack) {
    int right = stack.popInt(frame);
    int left = stack.popInt(frame);
    stack.pushInt(frame, left + right);
    return bci + 1;
}

One Handler, Many Variants

Let us manually inline the two pops and the push in iadd, keeping only the operations that affect tosDepth:

static long iadd(long bci, Frame frame, CachedStack stack) {
    // First popInt.
    switch (stack.tosDepth) {
        case ONE_SLOT_CACHED -> stack.tosDepth = NO_SLOTS_CACHED;
        case TWO_SLOTS_CACHED -> stack.tosDepth = ONE_SLOT_CACHED;
        default -> { }
    }

    // Second popInt.
    switch (stack.tosDepth) {
        case ONE_SLOT_CACHED -> stack.tosDepth = NO_SLOTS_CACHED;
        case TWO_SLOTS_CACHED -> stack.tosDepth = ONE_SLOT_CACHED;
        default -> { }
    }

    // pushInt.
    switch (stack.tosDepth) {
        case ONE_SLOT_CACHED -> stack.tosDepth = TWO_SLOTS_CACHED;
        case TWO_SLOTS_CACHED -> { }
        default -> stack.tosDepth = ONE_SLOT_CACHED;
    }
    return bci + 1;
}

The previous article introduced handler specialization: give the incoming template variable a constant value and let constant folding do its job. For iadd(tosDepth=2), each state update determines which arm of the next switch survives. The markers below trace that sequence:

static long iadd_2(long bci, Frame frame, CachedStack stack) {
    // Represents the incoming template constant, not a runtime assignment.
    stack.tosDepth = TWO_SLOTS_CACHED;

    // First popInt.
    switch (stack.tosDepth) {
        case ONE_SLOT_CACHED -> stack.tosDepth = NO_SLOTS_CACHED;
▶       case TWO_SLOTS_CACHED -> stack.tosDepth = ONE_SLOT_CACHED;
        default -> { }
    }

    // Second popInt.
    switch (stack.tosDepth) {
▶       case ONE_SLOT_CACHED -> stack.tosDepth = NO_SLOTS_CACHED;
        case TWO_SLOTS_CACHED -> stack.tosDepth = ONE_SLOT_CACHED;
        default -> { }
    }

    // pushInt.
    switch (stack.tosDepth) {
        case ONE_SLOT_CACHED -> stack.tosDepth = TWO_SLOTS_CACHED;
        case TWO_SLOTS_CACHED -> { }
▶       default -> stack.tosDepth = ONE_SLOT_CACHED;
    }
    return bci + 1;
}

Restoring the operand operations, constant folding and dead-code elimination leave:

static long iadd_2(long bci, Frame frame, CachedStack stack) {
    int right = GraalDirectives.unpackInt(stack.tos2);
    int left = GraalDirectives.unpackInt(stack.tos1);
    stack.tos1 = GraalDirectives.packInt(left + right);
    stack.tos2 = GraalDirectives.unconstrainedValue();
    stack.tosDepth = ONE_SLOT_CACHED;
    return bci + 1;
}

The corresponding assembly is compact:

    ; regBCI:   incoming bci         (long)
    ; regFrame: incoming frame       (non-null)
    ; regPrims: incoming frame.prims (non-null)
    ; regRefs:  incoming frame.refs  (non-null)
    ; regCode:  incoming frame.code  (non-null)
    ; regTop:   incoming stack.top   (long)
    ; regTos1:  incoming stack.tos1  (long)
    ; regTos2:  incoming stack.tos2  (long)
    add     regTos1_32, regTos2_32              ; Java int addition
    inc     regBCI
    jmp     tail_call_target(regCode, regBCI, ONE_SLOT_CACHED)

The same process derives iadd(tosDepth=0) and iadd(tosDepth=1), which fetch two operands or one operand from the backing frame, respectively.

The handlers describe bytecode behavior; the stack implementation supplies caching behavior wherever those operations occur.
Inlining brings them together, and constant folding derives each specialized variant. This is an aspect-oriented way to implement the optimization.

We still design the transitions in the stack abstraction, but need not calculate the outgoing tosDepth separately for every handler. The compiler fills in that column for us. That state selects the next already-specialized handler at the threaded tail call.

That only works while the compiler can determine the outgoing state on each path. Different branches may lead to different constant states, but invocation bytecodes are less cooperative: they may pop arguments in a loop whose iteration count depends on the target method. Our implementations flush the cache before that loop and finish with a known empty-cache state. Code that operates directly on the backing frame likewise needs the cached values written back first.

Flushing is less alarming than it sounds. These are the stack stores the uncached interpreter would have performed earlier; caching merely postpones them, and may eliminate some before a flush is needed.

Writing cached values back does not require allocating the overlay object. It can remain virtual, provided we do not let it escape through a call.

One Interpreter, Many Stack Implementations

Now we can change the stack implementation without rewriting the handlers. A different subclass can encode a different state machine.

Our first model spends two cache fields on one long, including its padding, because the state counts slots rather than values.
But the state could tell us more.

Let <CAT1> mean an int or float, occupying one JVM slot, and <CAT2> mean a long or double, occupying two. We then have six populated states: <CAT1>, <CAT2>, <CAT1,CAT1>, <CAT1,CAT2>, <CAT2,CAT1>, and <CAT2,CAT2>, with pairs ordered from bottom to top. Seven, including the empty cache.

Each cache field now holds a complete payload. The state tells materialization how much space to reserve for it in the backing frame.
For <CAT1,CAT1>, we write two integers or floats and advance top by two slots. For <CAT2,CAT2>, we write two longs or doubles and advance top by four. The fields still extend the backing array, but the number of slots they represent depends on the state.

Two registers. Now room for two longs.

We can go further and distinguish the actual primitive types: <INT>, <FLOAT>, <LONG>, <DOUBLE>, and ordered pairs such as <INT,LONG> or <DOUBLE,FLOAT>. That lets the stack implementation choose a register representation as well as a slot width. We keep integer payloads in long fields and add double fields for floating-point payloads, which can travel in XMM registers on x64. A float retains its raw bits rather than being numerically widened to a double. Consecutive floating-point handlers can then operate on those payloads without shuttling them between general-purpose and floating-point registers.

The state determines which fields contain valid payloads and must be written back during materialization; unused representations must not be treated as stack values. The expansion configuration exposes that validity to the compiler, so those unused fields need not occupy registers either.

There is a bill for all this information. Four single-value states and sixteen ordered pairs, plus the empty state, give us twenty-one states. Each incoming state creates another handler variant to compile, and each viable variant contributes generated code.

Some handler/state combinations are impossible, and we can say so. The typed popInt rejects a cached operand whose kind is not INT using an unreachable-path directive. For iadd entering in <DOUBLE,INT>, the first pop succeeds, but the second encounters the cached double. Once specialization exposes that contradiction, the compiler propagates the unreachable marker back to the entry, discarding the preceding work and reducing the variant to a tiny trap. This reduces both the remaining compilation effort and the emitted code size.

To reduce the number of variants further, we can also choose not to cache some perfectly legal combinations.

For example, we could keep only the empty state, the four single-type states, and the four same-type pairs: nine states. A push that would create a mixed-type pair would instead write the existing cache and the incoming value to the frame, then return to the empty-cache state. <INT,DOUBLE> is legal for DASTORE, but we can still omit it and flush instead.

Whether we add states or drop them, the changes stay in the stack subclass and its expansion configuration.

The handlers still push, pop, and compute.

The Fast and the Curious

With those pieces in place, let us step back and see how far we have come.

We threaded the handlers, then used the verifier’s knowledge to remove checks and widen their indices. We taught the compiler to make different tradeoffs for these tiny handlers. Finally, we cached the top of the stack, keeping more interpreter state in registers.

To see how these changes translate into performance, we selected Sieve and Mandelbrot from the Are We Fast Yet suite. Both are arithmetic-heavy workloads that exercise the handler and operand-stack operations discussed above. All measurements are interpreter-only, with JIT compilation disabled.

Our starting point is Crema before threaded tail calls. For TOS, we compare the basic slot cache, the category cache, and the typed cache.

Threading gives both workloads a substantial boost, but neither catches HotSpot until we teach the compiler the handler-specific tradeoffs.

With the basic cache, Sieve runs about 48% faster than HotSpot; Mandelbrot, about 34%. Sieve heavily exercises baload and bastore.
Two differences likely help here. We quicken these bytecodes for the array type; in particular, bastore no longer needs HotSpot’s byte-versus-boolean check. Our two-entry primitive cache can also keep both the index and value in registers, whereas HotSpot caches only the top operand.

The category cache brings Mandelbrot to 41% faster. Mandelbrot is heavy on double arithmetic: the basic slot cache holds only one double, while the category cache can keep both operands in registers.

The typed cache is the interesting twist. Although llvm-mca estimates 7.5% fewer cycles than the category-cache version for the interpreter instruction sequence corresponding to Mandelbrot’s hot loop, the typed cache runs slower in the benchmark. One possible culprit is code size: more handler variants mean a larger instruction footprint, potentially increasing pressure on the instruction cache.

Two arithmetic-heavy workloads do not settle the performance of an entire interpreter. They do show that readable Java and competitive interpreter code can coexist — even when getting there takes quite a bit of teaching.

The End Credits

We started by asking how close ordinary Java could get to the assembly we wanted. The answer depended on how we divided the work between the interpreter’s source code and the compiler. The source describes the execution model and supplies knowledge the compiler cannot discover on its own. The compiler turns that description into machine code, sometimes with policies tailored to the interpreter.

The harder question is where the design should live. When the design lives only in comments or in the programmer’s head, the implementation can drift away from it. Expressed through the right abstraction, it becomes something we can maintain and extend — and something the compiler can turn into efficient code.

Escape analysis deserves a credit of its own. Together with scalar replacement, it lets an object remain a useful source abstraction without requiring an object in the generated code. Its fields become independent compiler values that can live in registers. Argument expansion carries that idea across handler boundaries. The design stays in the source. Its objects need not survive into the assembly.

We still care about every load, move, and spill. We would just rather not write each one ourselves.

There is still call overhead to address.
That is where we go next.

Chasing the Perfect Assembly was originally published in graalvm on Medium, where people are continuing the conversation by highlighting and responding to this story.

Last year, Thomas, bearer of the One Graal, came to us with a prompt:

「 /goal make the GraalPy interpreter faster than CPython 」

To understand what Thomas was asking, it helps to know what the two interpreters are made of. The GraalPy interpreter is written in Java. CPython’s is written in C. So the real assignment was to make a Java interpreter outrun a C one, without giving up the high-level Java programming model.

The good news was that we controlled the compiler.

The bad news was that we controlled the compiler.

There are very few hard limits when you own the entire stack. There are also very few excuses.

If the branch predictor was unhappy, that was our problem.

If the register allocator was unhappy, that was our problem.

So we started looking for something dangerous to optimize.

The Giant Switch

The most obvious suspect was the interpreter dispatch loop. To keep the discussion concrete, let us start with a tiny program:

static int plus1(int x) {
    return x + 1;
}

On a stack-based virtual machine, this could be compiled into a bytecode sequence like:

iload_0
iconst_1
iadd
ireturn

Even if you have never looked at Java bytecode before, the instructions are fairly self-explanatory: load the first local variable (which happens to be the method parameter in this example), push the constant 1, add the two values, and return the result.

One natural way to implement such a bytecode interpreter is a large dispatch loop:

while (true) {
    int opcode = bytecodes[pc];
    switch (opcode) {
        case ILOAD_0:
            stack[sp++] = locals[0];
            pc++;
            break;
        case ICONST_1:
            stack[sp++] = 1;
            pc++;
            break;
       case IADD:
            int b = stack[--sp];
            int a = stack[--sp];
            stack[sp++] = a + b;
            pc++;
            break;
       case IRETURN:
            return stack[--sp];
       // imagine a few hundred more friends here
    }
}

Before continuing, it is worth acknowledging that this is not the only way to implement a bytecode interpreter. HotSpot’s template interpreter, for example, is famously efficient. Rather than relying on a giant switch loop, it generates platform-specific native code for executing bytecodes.

That option is powerful, but it also moves the burden onto handwritten or explicitly generated low-level machinery. Our challenge was different: keep the interpreter expressed through Java objects and methods, while still giving the compiler enough information to recover some of the same efficiencies automatically.

The real dispatch loop was, of course, much larger than this toy example, with hundreds of bytecode handlers.

Still, the execution pattern remains the same: every bytecode handler eventually returns to the same dispatch loop, which fetches the next opcode and jumps to the corresponding handler.

This architecture has many virtues. It is well understood, easy to maintain, and surprisingly effective. It also has a tendency to accumulate state. The program counter lives there. The operand stack lives there. The local variables live there. Profiling information lives there. Before long, every piece of interpreter state is flowing through the same compilation unit.

That last detail would eventually matter a great deal.

For now, all we saw was a giant switch sitting in the middle of one of our hottest execution paths.

Naturally, we blamed it for the performance gap.

Take I: Thread the Switch

The first theory was branch prediction.

At the end of each non-terminating bytecode handler, control returns to the switch. The interpreter reads the next opcode and uses the same dispatch branch to select the next handler. So every transition in our plus1 example passes through the same machine-code branch.

Modern processors do not wait politely for every branch to resolve. They speculate. The processor predicts where a branch will go and starts executing along that path before the branch has actually completed.

When the prediction is correct, much of that work is effectively free. When the prediction is wrong, the speculative work is discarded and execution restarts from the correct target. Ideally, the processor would learn that after iload_0 comes iconst_1, after iconst_1 comes iadd, and after iadd comes ireturn.

Unfortunately, the predictor does not see bytecodes. It sees machine instructions.

From the processor’s perspective, all of these transitions originate from the same dispatch branch.

To build some intuition, imagine that each branch instruction is a signpost. Whenever the branch resolves, the signpost turns to point toward the destination it just took. The next time execution reaches that signpost, the processor speculates by following its current direction. Under this simple model, the shared dispatch signpost will mispredict every transition in our example.

Real branch predictors use considerably richer history, but each time the branch executes, the predictor must still choose one specific target and begin speculating along that path.

This is the classic motivation behind threaded dispatch. The trick is to duplicate the signposts.

The bytecode sequence stays the same, but each handler gets its own copy of the dispatch branch at a different machine-code location.

The obvious question is how to generate such a layout. Fortunately, the replay itself is surprisingly straightforward.

We expose this through a feature of the Truffle Bytecode DSL. The dispatch sequence is marked for special treatment:

while (true) {
    int opcode = bytecodes[pc];
    switch (markThreadedSwitch(opcode)) {
        // ...
    }
}

Because we control the compiler, that marker can trigger a late code-generation step. During code generation, we record the machine code for the dispatch path, from the loop header through the table jump generated for the switch. At the very end of compilation, the compiler replaces each interpreter backedge with a copy of those exact machine-code bytes.

No re-optimization is required. No new register allocation is required.

As a result, all registers, memory operands, and addressing modes remain exactly as they were in the original dispatch sequence. The important difference is that the branch instruction is no longer shared by every bytecode transition.

Some patching is still required. Branches leaving the copied region, metadata references, and other bookkeeping details need to be adjusted. These details matter, but they do not change the basic idea: copy the generated dispatch path to each backedge.

One unpredictable dispatch branch becomes many more predictable ones.

That was the theory, at least.

The Rabbit Hole

As expected, threaded dispatch helped, but the magnitude never felt right.

Honestly, I never really bought branch prediction as the whole story.

So we opened the intermediate representation our compiler emits.

As compiler engineers, this is usually the point where we hope to find a small, local mistake instead of a new research project.

Fortunately, there was one.

The shared dispatch loop header was carrying around a surprising amount of register pressure.

To understand why this matters, imagine an interpreter that keeps enough state live to exceed the number of available registers. On x86–64, that is not as difficult as it sounds. Despite decades of progress elsewhere in computing, most of us are still developing on machines with 16 general-purpose registers. Intel APX promises relief, but it has yet to appear on the machines that generate our management update charts.

Eventually, some values must be spilled to memory. In our case, the scheduling heuristics had a tendency to pull interpreter states into the shared dispatch loop header.

The unfortunate part is that many of those states were not needed by every bytecode handler. An interpreter state that was only relevant to a handful of bytecodes could still compete for registers in the shared dispatch path executed by all of them. The more registers a handler needed locally, the fewer good choices remained for values live around the shared loop header.

A straightforward mitigation was to tweak the heuristics so fewer interpreter states were scheduled into the loop header.

This did not make register pressure disappear. It changed where the cost was paid. An interpreter state needed by one handler should ideally compete for registers in that handler, not in the shared dispatch path executed by every bytecode. That is a much better trade-off.

Ordinarily, increasing code size would be a concern.

At this point we were already duplicating the dispatch loop onto every interpreter backedge. Worrying about a few more instructions felt somewhat academic.

The optimization worked, surprisingly well.

Unfortunately, this was also where we fell into the rabbit hole.

We had reduced the cost of the shared dispatch path, but the compiler was still fundamentally reasoning about the same enormous program.

A giant switch had become a giant compilation unit. A giant compilation unit had become a giant live-range problem.

Values from unrelated bytecode handlers were still competing for the same limited register budget. For an interpreter, not all values are equal. Spilling a rarely used handler-local temporary is annoying. Spilling the program counter, the stack pointer, the bytecode array, or frequently accessed frame state can add memory traffic to almost every bytecode transition.

While a register allocator can often find excellent solutions, it cannot exhaustively explore the optimization space of a large compilation unit in a reasonable amount of time.

It must therefore make more heuristic decisions, and heuristic decisions inevitably leave some performance on the table.

Register allocation was only where the problem first became obvious. The same shape affected every optimization that had to reason about the interpreter as one enormous body.

We could keep tuning heuristics and teaching the compiler new tricks.

Or we could ask a more uncomfortable question.

Why was the entire interpreter being optimized as one giant compilation unit in the first place?

Take II: Escape the Giant Switch

Once we framed the problem as a giant compilation unit, the obvious solution seemed equally obvious.

Split it.

After all, compilers already know how to compile multiple methods independently. Why force the entire interpreter through one enormous optimization problem?

In fact, there is a remarkably straightforward way to do exactly that. Every bytecode handler can simply become its own method:

@DontInline
static void iload0(...) { ... }

@DontInline
static void iconst1(...) { ... }

@DontInline
static void iadd(...) { ... }

Now every handler is compiled independently. The giant compilation unit disappears. The giant live-range problem disappears. The register allocator can focus on one bytecode handler at a time.

The problem is gone. Unfortunately, so is most of the performance.

The issue is that compilation boundaries are not free.

For an interpreter, there is quite a lot of state to carry around. The bytecode stream. The program counter. The operand stack. The stack pointer. Profiling information. Exception state. Various pieces of bookkeeping accumulated over years of evolution.

Inside a single compilation unit, many of these values can remain virtual. The compiler is free to keep them in registers and move them around as it sees fit.

Across a method boundary, that illusion breaks down: values must cross a calling convention, and virtual state may need to be materialized.

In a language like Java, primitive interpreter state is particularly awkward. One option is to store everything in an object and pass that object from handler to handler. The register allocator will certainly appreciate the smaller compilation units.

The memory subsystem, less so.

We would have traded a register allocation problem for a memory traffic problem. What we really wanted was something contradictory.

We wanted each bytecode handler to be compiled independently, as if it were its own compilation unit.

At the same time, we wanted interpreter state to flow between handlers as if no compilation boundary existed at all.

In other words, we wanted optimization across compilation units.

Normally, those words do not belong in the same sentence.

The Valhalla-Shaped Trick

The obvious question was how to move interpreter state across compilation boundaries without sacrificing the optimization opportunities we had just fought so hard to recover.

The idea itself was not entirely new.

Project Valhalla had already spent years exploring a similar observation: programmers like objects, but compilers often prefer their contents.

Consider our running example:

class State {
    int pc;
    int sp;
}

static void iadd(State state, int[] stack) {
    int b = stack[state.sp--];
    int a = stack[state.sp--];
    stack[++state.sp] = a + b;
    state.pc++;
}

From the interpreter author’s perspective, this is exactly what it should be: a small mutable object carrying interpreter state from one bytecode handler to the next. From the compiler’s perspective, however, the interesting part is rarely the object itself. The interesting part is the values it contains.

Conceptually, instead of reasoning about State state, the compiler would much rather reason about <pc, sp>.

So we took the idea and developed it further for interpreter state. Rather than passing interpreter objects across compilation boundaries, we expand selected fields into a tuple of values. Together with the existing handler arguments, that gives the compiler the boundary tuple it actually wants to optimize. To preserve the original programming model, the compiler leaves the original handler signature unchanged and synthesizes a small stub:

static void __stub_iadd(int pc, int sp, int[] stack) {
    State state = new State(pc, sp);  // To be virtualized
    iadd(state, stack);               // To be inlined
}

After inlining, escape analysis can often remove the reconstructed state object from the optimized representation and replace its fields with compiler values. Even when it cannot do so globally, Graal’s control-flow-aware Partial Escape Analysis can apply the same optimization on the fast path where object materialization is unnecessary.

Expanding interpreter state into a tuple solves only half of the problem. The tuple still needs to come back, but Java methods are much happier accepting many inputs than returning many outputs. So the generated stub grows a tuple return value:

// Pseudo-Java
static <int, int, int[]> __stub_iadd(int pc, int sp, int[] stack) {
    State state = new State(pc, sp);  // To be virtualized
    iadd(state, stack);               // To be inlined
    return <state.pc, state.sp, stack>
}

At first glance, this may look like constructing and returning a tuple object. The important detail is where the updated values land. The generated calling convention returns the tuple through the same locations that carried it in. If pc enters the stub in one register, the updated pc' comes back in that same register. The caller-side write-back can then be inserted mechanically:

// Pseudo-Java
<pc', sp', stack> =
    __stub_iadd(state.pc, state.sp, stack);
state.pc = pc';
state.sp = sp';

Interpreter state now follows a round trip:

The state objects preserve the Java programming model on both sides of the boundary. Interpreter authors keep working with objects, fields, and ordinary method signatures, while the compiler uses the tuple to carry selected state between compilations as register values.

Escape analysis also runs independently in the caller, which may keep the original state object virtual.

The write-back itself does not necessarily become memory traffic. If both state objects remain virtual, the updated values may continue flowing as compiler values throughout the entire round trip. In practice, this gives us much of what we wanted from escape analysis across compilation boundaries, without requiring the two compilations to become one.

The register allocator offers another perspective. Previously, the dispatch loop and all bytecode handlers formed one enormous register-allocation problem. Now the caller and each handler are allocated independently, with fixed argument locations connecting them into one continuous execution.

The Hidden Boundary Tax

There are still some limits to what this interface can express. In principle, we could carry every interpreter state across the call boundary. In practice, we must reserve part of the register budget for each handler’s local work. Even a simple bytecode such as iadd needs scratch registers for its local calculation, so the boundary tuple cannot occupy every useful register.

In the old giant compilation unit, an unselected value could still be visible to the inlined handler body and remain register-resident through scalar replacement. After outlining, a value that is not part of the boundary tuple is no longer carried into the handler as a compiler value. If the handler needs it, it must recover the value from materialized interpreter state through an ordinary memory access.

Usually this is still acceptable. We can select the most important states, and leave statistically negligible state to be recovered from memory when a handler actually needs it.

The more unavoidable issue is the call itself.

Besides ordinary call mechanics such as return-address bookkeeping and frame setup, a managed runtime like Native Image has to emit safepoint metadata for the call, describing where live references are. Unlike non-call operations, calls force the compiler to treat the callee as a black box. Live references residing in registers may be clobbered during the call, even if our special calling convention restores them once the call returns. That means the references need a GC-visible home, which often means a stack slot.

The outlined call boundary therefore turns part of the original register-pressure problem into a backup-and-restore problem around the stub call.

Yet the call also gives us the clue for escaping that cost. When the handler returns, the updated tuple already occupies the locations where the next handler expects its inputs. At that point, the obvious question is no longer just how to make the call cheaper. Why are we returning to the caller at all?

Thread the Calls

The answer, of course, is that we do not.

Instead, each handler stub performs the dispatch itself. This optimization is often called tail call threading. After executing the bytecode implementation, it fetches the next opcode, looks up the corresponding handler stub, and jumps directly to it. The bytecode stream therefore becomes a first-class part of the handler interface, extending the tuple to <pc, sp, stack, bytecodes>.

The generated code now looks roughly like this:

// Pseudo-Java
static <int, int, int[], short[]>
__stub_iadd(int pc, int sp, int[] stack, short[] bytecodes) {
    State state = new State(pc, sp);  // To be virtualized
    iadd(state, stack);               // To be inlined
    int nextOpcode = bytecodes[state.pc];
    jump handlerTable[nextOpcode]
        with <state.pc, state.sp, stack, bytecodes>
}

The specialized calling convention makes this natural: the jump requires no reshuffling and transfers the current interpreter state directly.

Viewed from a distance, the handlers resemble LEGO bricks: the shared tuple interface forms the studs that let them snap directly together.

The dispatch loop still exists. It simply stops getting invited to every bytecode.

The Benchmark Everybody Came For

At this point, we should probably return to the benchmark that drove this whole adventure.

To recap, the first suspect was branch prediction in the dispatch loop, and we threaded the switch to attack it. The spill came later; we tuned the scheduling heuristic and pushed the spill cost into individual handlers. Eventually, the compilation unit itself became suspicious. We outlined the handlers, then threaded the tail-call between them.

The final, threaded-tail-call configuration is GraalVM 25.1 as released. The other configurations use that same release with the relevant optimizations selectively disabled, allowing us to reconstruct each stage of the journey. All measurements are interpreter-only, with GraalPy’s JIT compilation disabled.

We selected a few arithmetic-heavy benchmarks that are representative of the interpreter optimizations discussed above. Other workloads are dominated by different costs, such as guest-language calls or attribute access, and therefore depend more heavily on the implementation of the Python runtime itself.

bytecode-benchmark fits the story most clearly. The baseline starts below CPython. Threading the switch brings it to roughly parity, and tuning the scheduling pushes it ahead.

Then we got greedy. To thread execution directly from one handler to the next, we first outlined the handlers into separate compilation units, and hit the boundary tax. The handlers were smaller, but every call was surrounded by bookkeeping memory accesses.

Tail-call threading is what makes that structure pay. Once the updated tuple can flow directly from one handler to the next through the specialized calling convention, bytecode-benchmark moves comfortably ahead of CPython.

arith-binop and sieve start from stronger baselines but follow the same broad progression.

Overall, the benchmark result is not simply that threading wins. It is more specific: threading makes the handler boundary cheap enough that outlining stops being structurally fatal.

That leaves a more interesting question. If the compilation boundary between caller and handler no longer has to be an optimization boundary, what about the boundary between one handler and the next?

“One More Thing”

The most frequently accessed values in a stack interpreter are, unsurprisingly, the values at the top of the stack. A typical arithmetic bytecode may load values from the operand stack, perform an operation, and immediately push the result back. Even after escaping the giant switch, these stack accesses remained firmly on the hot path.

This is not a new observation. Top-of-stack caching is a well-established interpreter optimization. The question was how to make it expressible in ordinary Java code while still giving the compiler a register-level representation to optimize. The answer, again, was the tuple.

By this point, interpreter state was already flowing between handlers as a tuple of register-resident values. Nothing limited that tuple to the existing interpreter state, such as the program counter or stack pointer: it could also carry recent operand-stack values, keeping them in registers across handler boundaries.

Conceptually, the handler tuple grows from <pc, sp, stack, bytecodes> to <pc, sp, stack, bytecodes, tos1, tos2, tosDepth>.

Here tosDepth says how many cached values are currently valid. When tosDepth is 1, tos1 is the current top of stack. When tosDepth is 2, tos2 is the current top of stack, while tos1 is the value beneath it.

At this point, some readers may be counting registers and becoming increasingly uncomfortable. Under a conventional calling convention, they would be right. Sooner or later additional arguments would spill onto the stack, undoing much of the benefit we were trying to achieve.

Fortunately, we control the compiler.

Nothing requires us to stop at the six integer argument registers provided by the platform ABI. Handler stubs never cross a language boundary, so we are free to employ a considerably more adventurous calling convention and use essentially the entire allocatable register file for interpreter state. The tuple may be growing, but it is still very much intended to stay in registers.

In our running example, iload_0 pushes the local value into tos1. Then iconst_1 pushes the constant into tos2. Finally, iadd consumes tos2 and tos1, then leaves the result in tos1.

The important part is not merely that tos1 and tos2 avoid memory accesses inside one handler. The value loaded by iload_0 remains in a register across iconst_1 and into iadd, without ever touching the operand stack.

No stack loads.

No stack stores.

At least not immediately.

Of course, reality is slightly less convenient. Two registers can only hold two stack values. Sooner or later a bytecode sequence will push a third value, then a fourth, and eventually the cache must spill into the backing operand stack. Likewise, some handlers may need to materialize cached values before interacting with code that expects the operand stack to reside in memory. The interpreter therefore carries tosDepth as part of its state and lets individual handlers decide when values should remain in registers and when they must be synchronized with the backing stack array.

Without further help, every push and pop now has to account for all possible cache states. A push is no longer simply stack[++sp] = value;. It starts to look more like:

switch (tosDepth) {
    case 0:
        tos1 = value;
        tosDepth = 1;
        break;
    case 1:
        tos2 = value;
        tosDepth = 2;
        break;
    case 2:
        stack[++sp] = tos1;
        tos1 = tos2;
        tos2 = value;
        break;
}

A pop has the same problem in reverse.

After spending several sections moving interpreter state into registers, we had accidentally introduced a new runtime state machine.

The (Compile-) Time (State) Machine

If tosDepth merely describes the shape of the cached stack, perhaps it should not travel in the tuple at all. Instead, we could turn the runtime state machine into a compile-time one, sending that decision backward in time from every bytecode execution to handler generation.

Consider iadd written in the most boring possible way:

static void iadd(...) {
    int b = stack.pop();
    int a = stack.pop();
    stack.push(a + b);
    state.pc++;
}

Without specialization, each pop and push must inspect tosDepth at runtime, as in the push example above.

Instead of generating a single handler that must reason about every possible cache state, we generate multiple handler variants. Each variant gets assigned a constant tosDepth, which is enough for ordinary compiler optimizations to fold away the irrelevant arms of push and pop. After those branches disappear, the updated tosDepth often becomes deterministic as well. Dispatch can then use that value to choose the next variant.

Returning to our LEGO example, imagine that each bytecode brick now bundles several handler variants, one for each possible incoming tosDepth. We stack these composite bricks vertically, and the tosDepth produced by one handler selects the path into the next.

The state machine still exists, but it no longer lives as control flow inside every handler. It lives in the transitions between handler variants.

Now consider the iadd<2> specialization. The compiler knows that both operands already reside in registers. The first pop becomes int b = tos2;. The second pop becomes int a = tos1;. And the push becomes tos1 = a + b;.

After constant folding and dead-code elimination, the handler is essentially reduced to tos1 = tos1 + tos2;.

Consequently, tosDepth no longer needs to occupy a runtime slot in the tuple.

The columns are not runtime branches; they are separately generated handler variants. The highlighted path follows iload_0<0> to iconst_1<1>, then iadd<2> and ireturn<1>. Each handler’s output cache shape determines the next variant, so neither a runtime tosDepth value nor a test of it appears along the path.

We traded runtime state transit for instruction bytes: multiple handler variants eliminate a runtime value and its control flow, freeing a register for actual interpreter state. The previous sections taught the interpreter how to carry state efficiently across compilation boundaries. This section taught it that some state never needed to travel in the first place.

The remaining concern is register pressure. For the mechanism to work well, the handler interface needs enough registers to carry two or more cached top-of-stack values in addition to the existing interpreter state. That is already a tight budget. GraalPy’s tail-call-threaded handlers pass a fairly heavy argument tuple today, and adding more top-of-stack values can exhaust registers and push other state back to memory.

The situation is even trickier because the Python stack can hold both primitive values and object references. Those may need special treatment, or even separate cached top-of-stack values, which would make the interface wider again. At that point, the optimization starts fighting the same register-pressure problem it was meant to avoid.

We are still exploring ways to mitigate this register cost, or at least warn interpreter authors when the handler interface becomes too wide.

Take III: ?

This article started with a giant switch.

The giant switch led to a spill.

The spill led to a giant compilation unit.

And the giant compilation unit turned out to be hiding far more than a few misplaced registers.

We first threaded the switch, giving each bytecode handler its own dispatch branch. We then tuned scheduling so handler-local state spilled only where necessary, rather than across the entire dispatch loop. Finally, we outlined the handlers into separate compilation units and connected them through a specialized tail-call convention. Interpreter state crosses those compilation boundaries as a tuple of values in fixed register locations, allowing the handlers to remain separate compilation units without turning their boundaries into optimization boundaries.

The threaded-handler implementation shipped in GraalVM 25.1.3 and is enabled by default in GraalPy. On the interpreter-only, arithmetic-heavy workloads shown above, it takes bytecode-benchmark from below CPython to comfortably above it, while arith-binop reaches roughly three times CPython.

This does not make every Python workload three times faster. Calls, attribute access, and other runtime operations are dominated by different parts of the GraalPy implementation. Even within the handler interface, registers remain a finite resource: carrying more interpreter state eventually risks pushing existing values back to memory.

That is also where the next experiment begins. If compilation boundaries no longer prevent values from remaining visible to the optimizer, the same interface can carry cached operand-stack values directly from one handler to the next. The mechanism works; finding the right register budget is still ongoing work.

For now, the giant switch is gone.

Until the next rabbit hole.

Escaping the Giant Switch was originally published in graalvm on Medium, where people are continuing the conversation by highlighting and responding to this story.