Chasing the Perfect Assembly
Matching the Speed of Hand-Written Assembly with an Interpreter Written in Java
Previously, in the “Escaping the Giant Switch” episode, we learned how to break one enormous interpreter compilation unit into many independently compiled bytecode handlers.
This left us with an appealing hypothesis: if we can find the locally optimal assembly for each handler, putting those handlers together should give us a globally optimal interpreter.
We put that hypothesis to the test with the Crema interpreter, with one constraint: its Java source code had to stay maintainable.
As compiler engineers, we occasionally entertain a slightly arrogant belief: programmers should write straightforward Java, and our extremely smart compiler should handle the rest.
But when the emitted assembly falls short, who should change?
Should the compiler learn a more general optimization, or should the interpreter expose a fact that only it knows?
Act I: The Expandables
Let us start with iconst_0, one of the simplest Java bytecodes. It pushes the integer zero onto the operand stack.
For this example, we need only a compact model of the interpreter frame and stack state:
class Frame {
/* Raw primitive values in local-variable and operand-stack slots. */
final long[] prims;
/* Object references in local-variable and operand-stack slots. */
final Object[] refs;
/* Bytecode of the method being interpreted. */
final byte[] code;
}
class Stack {
/* Next free operand-stack slot in Frame.prims/refs. */
int top;
}
static int iconst0Handler(int bci, Frame frame, Stack stack) {
int slot = stack.top;
frame.prims[slot] = 0;
stack.top = slot + 1;
return bci + 1;
}Here the bci parameter is the bytecode index, the byte offset of the current instruction within the method’s bytecode array.
When frame and stack cross the handler boundary as ordinary arguments, the relevant fast path assembly is:
; regBCI: incoming bci
; regFrame: incoming frame
; regStack: incoming stack
mov regPrims, QWORD PTR [regFrame + OFFSET_prims] ; load frame.prims
▶ mov regTop32, DWORD PTR [regStack + OFFSET_top] ; load stack.top
test regPrims, regPrims ; is prims null?
je null_prims
cmp regTop32, DWORD PTR [regPrims + OFFSET_length] ; is top out of bounds?
jae out_of_bounds
mov QWORD PTR [regPrims + regTop * 8 + OFFSET_arrayStart], 0 ; prims[slot] = 0
inc regTop32 ; advance the stack top
▶ mov DWORD PTR [regStack + OFFSET_top], regTop32 ; write back stack.top
inc regBCI32 ; advance to the next bytecode
ret
; Here and below, reg*32 denotes the lower 32 bits of the corresponding register.For a bytecode that merely pushes zero, this is quite an entourage. Updating stack.top, for example, requires loading the field into a register, incrementing it, and storing it back.
In the previous article, we taught objects to cross compilation boundaries in pieces, while keeping the handler’s Java signature intact. We call this argument expansion, and distinguish two forms according to whether the original object also crosses the boundary.
With virtual expansion, the fields travel independently, without the original object reference. The generated stub reconstructs the object around those values and calls the original handler; after inlining, escape analysis can eliminate the reconstructed object.
With materialized expansion, the original object reference travels alongside selected fields. The handler receives those fields directly while retaining access to the backing object and its other fields.
For Frame, we choose materialized expansion and recruit prims , refs, and code as the expandables. For Stack, we choose virtual expansion. It sends top to join the expandables but stays behind itself.
Together, these choices change the handler’s boundary tuple from <bci, frame, stack> to <bci, frame, frame.prims, frame.refs, frame.code, stack.top>.
The tuple grew. The assembly shrank.
With the specialized calling convention, these values arrive in the registers where the handler needs them:
; regBCI: incoming bci
; regFrame: incoming frame
; regPrims: incoming frame.prims
; regRefs: incoming frame.refs
; regCode: incoming frame.code
; regTop: incoming stack.top
▶ test regPrims, regPrims ; is prims null?
▶ je null_prims
cmp regTop32, DWORD PTR [regPrims + OFFSET_length] ; is top out of bounds?
jae out_of_bounds
mov regSlot32, regTop32 ; zero-extend top into regSlot
mov QWORD PTR [regPrims + regSlot * 8 + OFFSET_arrayStart], 0 ; prims[slot] = 0
inc regTop32 ; advance the stack top
inc regBCI32 ; advance to the next bytecode
; Threaded dispatch replaces the return with a jump to the next handler
jmp tail_call_target(regCode, regBCI)stack.top now arrives in the lower 32 bits of regTop, and its updated value stays there for the next handler.
The field traffic is now gone, but the null check and bounds check are still there.
Native Image’s global data-flow analysis can prove some references non-null. However, when that proof is unavailable, the compiler must retain the null check.
Fortunately, argument expansion lets us specify non-nullness as part of the handler interface. This contract applies both to direct reference arguments such as frame and to expanded reference fields such as frame.prims.
In Graal terminology, the parameter receives a non-null stamp, which describes the values that parameter can hold. The handler compilation can use that stamp to eliminate the internal null check.
The check still has to happen. Just not in every handler. The compiler inserts null checks before handler calls from the dispatch loop and before threaded tail calls. For a field such as Frame.prims, which is final, the outgoing expanded argument is still the same non-null reference. The compiler can therefore fold away the null check before the outgoing threaded tail call too.
With argument expansion and the non-null contract, the relevant threaded-handler assembly is:
; regBCI: incoming bci
; regFrame: incoming frame (non-null)
; regPrims: incoming frame.prims (non-null)
; regRefs: incoming frame.refs (non-null)
; regCode: incoming frame.code (non-null)
; regTop: incoming stack.top
▶ cmp regTop32, DWORD PTR [regPrims + OFFSET_length] ; is top out of bounds?
▶ jae out_of_bounds
mov regSlot32, regTop32 ; zero-extend top into regSlot
mov QWORD PTR [regPrims + regSlot * 8 + OFFSET_arrayStart], 0 ; prims[slot] = 0
inc regTop32 ; advance the stack top
inc regBCI32 ; advance to the next bytecode
jmp tail_call_target(regCode, regBCI)For these unchanged references, argument expansion and the non-null contract effectively hoist the checks out of the threaded handler chain.
Less assembly. Same Java.
So far, the division of labor works.
But what about the bounds check?
Act II: Handler, Compiler, Verifier, Witness
The comparison between slot and prims.length is there for a reason. Java requires lastore to throw an ArrayIndexOutOfBoundsException when the index is outside the array bounds.
That requirement comes from the host Java code implementing the interpreter; the guest instruction iconst_0 merely pushes zero and prescribes no such exception. An out-of-bounds access here would mean an interpreter invariant had been broken, but unless the compiler can prove that slot is valid, the check stays.
Keeping that check costs more than the comparison alone. For instance, the cold exception path may keep values alive on the fast path, increasing register pressure and potentially causing spills.
So how can the compiler prove that slot is a valid index and avoid this cost?
If slot is constant and the array length is known, the proof can be immediate; if, in a counted loop, range analysis proves that every value of slot stays within the array, the check can be eliminated; if an earlier guard established 0 <= slot && slot < prims.length, the compiler can reuse it for every access it dominates.
All of these optimizations require the compiler to derive the proof from information in its graph.
Our tiny handler, however, is compiled as a standalone unit. Its graph receives stack.top and frame.prims as independent inputs, with no proof that the resulting slot is a valid index into the array.
Fortunately, the Java bytecode verifier can serve as a witness outside that graph.
Before the JVM executes a method’s bytecode, the bytecode verifier establishes a valid frame shape at every reachable bytecode index. Among other properties, it proves that every operand-stack operation stays within the max_stack recorded in the method’s class-file Code attribute. As long as an interpreter allocates operand-stack storage for that same max_stack and implements the verified operations faithfully, every operand-stack access is within the allocated storage.
In that sense, verification plays the role of a guard that dominates the entire interpretation. But it lives outside the handler’s compilation unit, so the compiler cannot reuse it as a dominating guard in the handler graph.
Could we use the same stamp-based contract that previously removed the null checks?
Just as a stamp can express non-nullness, it can also express a constant integer range such as [0, 63]. However, the required range is [0, prims.length), whose upper bound is a runtime value rather than a compiler constant. The stamp system cannot express this relationship between two runtime values.
For an ordinary Java array access, without a proof in the graph, the compiler must retain precisely the comparison and branch we want to remove.
Alternatively, we can make the verifier-established invariant an implicit assumption in the interpreter source, replacing the checked array write with an Unsafe-backed access:
static int iconst0Handler(int bci, Frame frame, Stack stack) {
int slot = stack.top;
UNSAFE.putLong(frame.prims, slot * 8L + OFFSET_arrayStart, 0);
stack.top = slot + 1;
return bci + 1;
}The bounds check no longer enters the handler’s graph, and the relevant assembly becomes:
; regBCI: incoming bci
; regFrame: incoming frame (non-null)
; regPrims: incoming frame.prims (non-null)
; regRefs: incoming frame.refs (non-null)
; regCode: incoming frame.code (non-null)
; regTop: incoming stack.top
▶ movsxd regSlot, regTop32 ; sign-extend top into regSlot
mov QWORD PTR [regPrims + regSlot * 8 + OFFSET_arrayStart], 0 ; prims[slot] = 0
inc regTop32 ; advance the stack top
inc regBCI32 ; advance to the next bytecode
jmp tail_call_target(regCode, regBCI)The array-length load, conditional branch, and cold exception edge have all disappeared. Correctness now rests on verification, frame allocation, and the interpreter preserving the verified stack discipline.
One Move Too Many
The bounds check is gone. But the zero-extension has quietly become a sign-extension.
Why?
Consider an input stack.top = -1, giving us slot = -1. Sign-extension preserves that value as 0xFFFFFFFF_FFFFFFFF, while zero-extension produces 0x00000000_FFFFFFFF. Using the latter in the Unsafe access would compute the wrong address, potentially causing a segmentation fault or memory corruption.
Of course, this input cannot occur in a correctly implemented interpreter executing verified bytecode. But that guarantee remains implicit in the source, so the compiler cannot use it to rule out negative inputs and must preserve Java’s signed conversion.
Before the switch to Unsafe, the compiler had to insert a bounds check.
Once that check passed, it knew that slot was non-negative at the array access. For a non-negative int, sign-extension and zero-extension produce the same 64-bit value, so it could replace the former with the latter. On processors that support move elimination, the resulting register move can be handled during register renaming, avoiding an execution step that sign-extension would require.
Zero-extension also gives the compiler an opportunity to eliminate the move entirely — a small prize, but one that can be significant in benchmarks such as SPECjvm. On x64, writing a 32-bit register also clears its upper 32 bits. If the compiler knows that the instruction that produced slot already did this, it may use that register directly as the address index.
Then why couldn’t the compiler eliminate the zero-extending move in our expanded handler before the switch to Unsafe?
Our stack.top arrives as an expanded parameter, so its producing instruction is outside this compilation unit. The calling convention promises an int in the lower 32 bits of regTop; it does not promise that the upper 32 bits are zero. Consider another input, stack.top = 1. The incoming register could contain 0x00000001_00000001, with the correct int value in its lower half. Using the entire register as an address index would include those non-zero upper bits, potentially causing a segmentation fault or memory corruption.
Proving that stack.top already arrives in a register containing its correctly extended, full-width value would require reasoning across compilation boundaries. We can avoid that proof by making the full-width value part of the handler interface. Changing the index’s representation to long preserves the invariants established by verification. Plus, with Unsafe, we are no longer constrained by Java’s int array indices, since the access takes a long byte offset.
We therefore changed both stack.top and bci to long, so each arrives as a 64-bit index ready for addressing. Here too, the verifier testifies that branch targets stay within the bytecode array. Neither index needs widening in each handler.
Our running example now becomes:
class Stack {
long top;
}
static long iconst0Handler(long bci, Frame frame, Stack stack) {
long slot = stack.top;
UNSAFE.putLong(frame.prims, slot * 8L + OFFSET_arrayStart, 0);
stack.top = slot + 1;
return bci + 1;
}The incoming stack top can serve directly as the address index:
; regBCI: incoming bci (long)
; regFrame: incoming frame (non-null)
; regPrims: incoming frame.prims (non-null)
; regRefs: incoming frame.refs (non-null)
; regCode: incoming frame.code (non-null)
; regTop: incoming stack.top (long)
mov QWORD PTR [regPrims + regTop * 8 + OFFSET_arrayStart], 0
inc regTop
inc regBCI
jmp tail_call_target(regCode, regBCI) ; long bci also avoids widening in the unshown dispatch codeStore zero. Advance the indices. Dispatch.
That is more like it.
Supplementary Testimony
The verifier establishes more than stack capacity.
For an interpreter, those guarantees fall into two useful categories:
- Access bounds: operand-stack and local-variable accesses stay within the storage allocated from
max_stackandmax_locals; bytecode operands form complete instructions; branch and exception-handler targets name valid instruction boundaries; and constant-pool indices refer to valid entries. - Type safety: every instruction consumes and produces compatible verification types; stack-manipulation forms are legal; field accesses, invocations, and returns agree with their descriptors; and uninitialized references obey the constructor rules.
The type-safety proofs are normally implicit in the handler implementation. For example, an iadd handler expects two integers on the operand stack without checking their runtime types.
Other type guarantees belong to a particular bytecode site rather than to every execution of an instruction. How can we make those facts useful to a shared handler?
Consider baload, which operates on either a byte[] or a boolean[].
Its shared handler contains a type check to distinguish the two array kinds at run time. At a given BCI, however, verification fixes the non-null array operand’s type as either byte[] or boolean[]; it cannot admit both at the same site. Different baload sites may have different array types, so this knowledge cannot be baked into a single shared handler. The guest-array null and bounds checks are still required.
We could use the verifier’s result to rewrite baload at that BCI to a byte[]- or boolean[]-specific opcode before execution begins.
Alternatively, we can record this site-specific information through runtime quickening.
On the first execution with a non-null array, the generic handler observes its type and rewrites that BCI to select one of two shared specialized handlers. A single runtime observation would not normally justify permanent specialization; the verifier’s BCI-specific guarantee makes it sufficient here. Subsequent executions use the specialized handler without repeating the type check. The specialized opcode carries the site’s type information into handler selection.
The same pattern applies when the initial execution establishes information that does not come from verification. For a field access, the original handler resolves the field, performs the opcode-specific access checks, and then quickens the opcode. The quickened handler retrieves the cached resolved entry without repeating resolution or those access checks.
To summarize, the compiler must generate correct code for every input its interface permits, while the interpreter’s external guarantees rule out some of those inputs. When the compiler cannot see those guarantees, we can still use them to choose a different implementation.
We have done so through unchecked accesses, wider parameters, and specialized opcodes, putting external knowledge to work without teaching the compiler its original proofs.
But how far should we push this?
Act III: How to Train Your Compiler
Let us return to the iconst_0 example. Can we make its assembly even simpler?
Suppose we carried the address of the next primitive element instead of its logical slot. In C-like code, initializing the pointer and pushing zero would look something like this:
int64_t *stack = &frame.prims[top]; // Initialize from the logical top outside of the handlers
...
*stack = 0; // Push zero
stack++; // Advance by eight bytesWith the pointer carried in regStack across threaded tail calls, pushing zero and advancing it could compile to:
mov QWORD PTR [regStack], 0
add regStack, 8The simpler addressing mode does not necessarily improve performance.
On Intel Skylake, both the original indexed store and this direct-address store execute as two micro-operations. The direct-address form can use an additional store-address port, which may help under contention, but does not guarantee a cycle saving.
The interpreter’s Java implementation, meanwhile, becomes more cumbersome. The stack top no longer names a logical slot. To address both primitive and reference arrays directly, we now have to maintain two pointers, advancing each according to its array’s element size. We must also keep these pointers into the managed heap valid across garbage collection and safepoints.
Making the source so much harder to read is a poor trade when we cannot even establish a performance gain. The choice becomes less clear when keeping the source tidy leaves avoidable work in the generated code.
The Pursuit of Constants
Consider putfield, whose handler branches according to the kind of field it stores into. An int occupies one stack slot, while a long occupies two, so their values start at stack.top — 1 and stack.top-2, respectively.
In either case, the receiver sits in the preceding slot.
Using our Frame and Stack abstraction, a natural implementation is to share the slot calculations and receiver lookup before branching on the field kind:
static long putfieldHandler(long bci, Frame frame, Stack stack) {
JavaField field = resolveField(frame, bci);
JavaKind kind = field.kind();
long valueSlot = stack.top - kind.slotCount();
long receiverSlot = valueSlot - 1;
Object receiver = UNSAFE.getReference(frame.refs, receiverSlot * OFFSET_referenceScale + OFFSET_arrayStart);
Object nonNullReceiver = nullCheck(receiver);
long offset = field.offset();
boolean isVolatile = field.isVolatile();
switch (kind) {
case Int -> {
// OFFSET_endian is 4 on big-endian targets, otherwise 0.
int value = UNSAFE.getInt(frame.prims, valueSlot * 8L + OFFSET_arrayStart + OFFSET_endian);
if (isVolatile) {
UNSAFE.putIntVolatile(nonNullReceiver, offset, value);
} else {
UNSAFE.putInt(nonNullReceiver, offset, value);
}
}
case Long -> {
long value = UNSAFE.getLong(frame.prims, valueSlot * 8L + OFFSET_arrayStart);
if (isVolatile) {
UNSAFE.putLongVolatile(nonNullReceiver, offset, value);
} else {
UNSAFE.putLong(nonNullReceiver, offset, value);
}
}
// Other valid field kinds omitted.
default -> throw shouldNotReachHere();
}
UNSAFE.putReference(frame.refs, receiverSlot * OFFSET_referenceScale + OFFSET_arrayStart, null);
stack.top = receiverSlot;
return bci + 3;
}The relevant x64 assembly snippet leading into the switch is the following.
; regBCI: incoming bci (long)
; regFrame: incoming frame (non-null)
; regPrims: incoming frame.prims (non-null)
; regRefs: incoming frame.refs (non-null)
; regCode: incoming frame.code (non-null)
; regTop: incoming stack.top (long)
; regKindObject holds the decompressed JavaKind reference; setup and null checks omitted.
; ... receiver decompression omitted ...
▶ mov QWORD PTR [rsp + 0x28], regFieldObject ; spill decompressed field reference
movsxd regSlots, DWORD PTR [regKindObject + OFFSET_slotCount]
mov regValueSlot, regTop
sub regValueSlot, regSlots ; valueSlot = top - kind.slotCount()
mov regReceiver32, DWORD PTR [regRefs + regValueSlot * 4 + OFFSET_arrayStart - 4]
mov regReceiverSlot, regValueSlot
dec regReceiverSlot ; receiverSlot = valueSlot - 1
test regReceiver32, regReceiver32 ; is receiver null?
je null_receiver
▶ mov QWORD PTR [rsp + 0x60], regTop ; spill stack top
; ... field offset/modifier loads omitted ...
mov regKind32, DWORD PTR [regKindObject + OFFSET_ordinal]
; ... ordinal bounds check omitted ...
▶ mov QWORD PTR [rsp + 0x10], regKindObject ; spill decompressed JavaKind reference
; ... ordinal jump table, typed stores, and threaded dispatch omitted ...At first glance, that is exactly what we asked for. It loads the slot count from JavaKind, then computes valueSlot and receiverSlot before the switch. The receiver load folds the subtraction into its address, but receiverSlot is still computed separately for the later update to stack.top.
Yet once we enter a switch arm, the slot count is fixed, giving us an opportunity for constant folding. In the int case, for example, replacing the slot count with one would simplify valueSlot to stack.top-1 and receiverSlot to stack.top-2. The memory accesses could then absorb those constant offsets into their address displacements. For the receiver, that gives DWORD PTR [regRefs + regTop * 4 + OFFSET_arrayStart-8]. The stack update could subtract two directly from top at the end.
That suggests a manual rewrite. Move the shared setup into the branches and specialize it for each kind:
static long putfieldHandler(long bci, Frame frame, Stack stack) {
JavaField field = resolveField(frame, bci);
JavaKind kind = field.kind();
// Verification and linking guarantee a valid field kind.
switch (kind) {
case Int -> {
long receiverSlot = stack.top - 2;
long valueSlot = stack.top - 1;
Object receiver = UNSAFE.getReference(frame.refs, receiverSlot * OFFSET_referenceScale + OFFSET_arrayStart);
Object nonNullReceiver = nullCheck(receiver);
int value = UNSAFE.getInt(frame.prims, valueSlot * 8L + OFFSET_arrayStart + OFFSET_endian);
if (field.isVolatile()) {
UNSAFE.putIntVolatile(nonNullReceiver, field.offset(), value);
} else {
UNSAFE.putInt(nonNullReceiver, field.offset(), value);
}
UNSAFE.putReference(frame.refs, receiverSlot * OFFSET_referenceScale + OFFSET_arrayStart, null);
stack.top = receiverSlot;
}
case Long -> {
long receiverSlot = stack.top - 3;
long valueSlot = stack.top - 2;
Object receiver = UNSAFE.getReference(frame.refs, receiverSlot * OFFSET_referenceScale + OFFSET_arrayStart);
Object nonNullReceiver = nullCheck(receiver);
long value = UNSAFE.getLong(frame.prims, valueSlot * 8L + OFFSET_arrayStart);
if (field.isVolatile()) {
UNSAFE.putLongVolatile(nonNullReceiver, field.offset(), value);
} else {
UNSAFE.putLong(nonNullReceiver, field.offset(), value);
}
UNSAFE.putReference(frame.refs, receiverSlot * OFFSET_referenceScale + OFFSET_arrayStart, null);
stack.top = receiverSlot;
}
// Other valid field kinds omitted.
default -> throw shouldNotReachHere();
}
return bci + 3;
}For the int case, the resulting addresses are:
; Int arm after duplicating the setup; other instructions omitted.
; Same incoming registers; regTop still holds the incoming stack.top.
; OFFSET_arrayStart = 8 and reference scale = 4 in this build.
mov regReceiver32, DWORD PTR [regRefs + regTop * 4] ; disp = 8 - 4 * 2
; ... receiver decompression omitted ...
test regReceiver32, regReceiver32 ; is receiver null?
je null_receiver_int
mov regValue32, DWORD PTR [regPrims + regTop * 8] ; disp = 8 - 8 * 1
; ... stack update, volatility test, and field store omitted ...The other arms fold their constant slot offsets into the addresses in the same way.
More Java. Less assembly.
But why can’t the compiler do this for us?
Graal can fold a final field read when the object containing the field is a known constant. But javac lowers an enum switch to an integer switch derived from ordinal(), typically through a synthetic mapping array. Entering the int case constrains this integer; it does not turn kind into the constant JavaKind.Int in the compiler graph. The slotCount read used to compute valueSlot therefore still has a dynamic receiver.
Teaching the compiler to map an ordinal back to its JavaKind instance could enable branch-local constant folding. That would require trusting the enum’s canonical-instance invariant, much as final-field folding trusts that a field’s value will not change.
HotSpot’s C2 compiler does not trust arbitrary instance final fields for constant folding by default, though it makes exceptions for certain classes. Its experimental -XX:+TrustFinalNonStaticFields option broadens that trust. Graal uses a different policy.
Even if the compiler could establish the enum identity in each branch, that alone would solve only half of our problem. It could specialize valueSlot within each branch, but the shared receiver access would still use the dynamic slot count before the switch. Eliminating that load would require duplicating the receiver access and null check into the branches as well.
The default branch would need to preserve the original calculation and null check before throwing its own exception. This is possible, but it requires teaching the compiler when that duplication is worth the extra code.
Let us leave that as a feature request for the compiler.
A Little Too Helpful
While we admire the result of constant folding, the compiler may be undoing some of our careful duplication. For instance, each branch calls field.offset(), which reads the same field offset from the resolved metadata object. Writing that call inside each arm does not necessarily keep the load there.
Graal can turn an eligible field load into a floating read: a load whose position is determined by its dependencies rather than fixed to a point in the control flow. Equivalent reads with matching dependencies can then become one shared node through global value numbering.
As long as the value remains shared, the scheduler must place it where it is available to all its uses. When those uses lie in different switch arms, this means placing it in a common dominator before the switch, provided its dependencies allow it. Sharing the load saves instruction bytes, but its result now has to stay live across the dispatch. In the duplicated handler, the field offset and modifier reads have moved before the switch:
; Before kind dispatch; regField holds the compressed field reference.
; save values before reusing their registers.
▶ mov QWORD PTR [rsp + 0x20], regKindObject ; spill decompressed JavaKind reference
▶ mov DWORD PTR [rsp + 0x1c], regKindRef32 ; spill compressed JavaKind reference
▶ mov QWORD PTR [rsp + 0x28], regFieldObject ; spill decompressed field reference
▶ mov QWORD PTR [rsp + 0x30], regFrame ; spill frame reference
movsxd regOffset, DWORD PTR [regHeapBase + regField * 8 + OFFSET_fieldOffset]
mov regFlags32, 0x50df
and regFlags16, WORD PTR [regHeapBase + regField * 8 + OFFSET_modifiers]
movzx regFlags32, regFlags16
mov regValueSlot, regTop
dec regValueSlot ; top - 1
lea regReceiverSlot, [regTop - 2] ; shared by the one-slot arms
lea regNextBCI, [regBCI + 3]
; ... ordinal range check and jump-table dispatch ...The same can happen to arithmetic. Across the one-slot cases — boolean, byte, char, and others — the same stack.top-2 calculation appears for the outgoing stack top. If these calculations become one shared node, it must execute before the switch, even though not every arm needs the result.
Graal’s partial-redundancy-aware scheduler can undo this sharing by duplicating a computation into the branches that need it. Its general heuristic favors splitting when doing so reduces execution frequency. The stack.top-3 calculation shared by the long and double arms illustrates this: the scheduler splits it back into those arms, avoiding the calculation on the more frequent one-slot paths. But the shared stack.top-2 calculation serves most of the execution, so splitting it offers too little frequency saving — even though it would shorten live ranges.
Get Yudi Zheng’s stories in your inbox
Join Medium for free to get updates from this writer.
Keeping the computation shared is a reasonable choice until registers become scarce. The jump-table dispatch generated from the switch needs its selector and a register for the target address, alongside the interpreter state carried to the next handler. Hoisted loads and slot calculations add to that live set even though only one arm will execute. If their overlapping lifetimes exceed the available registers, the allocator must spill something, which is unfortunately the case in many handlers with complicated branching logic. The compiler’s good intentions in sharing computations then turn into extra stores and reloads.
How do we train the compiler to reduce spilling?
We can start with compiler directives. One of my favorites is GraalDirectives.opaque, which returns the same runtime value but hides it behind a distinct compiler node. Wrapping field and stack.top separately in each switch arm prevents the compiler from simply treating the resulting expressions as shared work. With each copy used only in its own arm, the scheduler can keep the dependent reads and arithmetic there.
Our int case now looks like this:
case Int -> {
field = GraalDirectives.opaque(field);
long top = GraalDirectives.opaque(stack.top);
long receiverSlot = top - 2;
long valueSlot = top - 1;
// ... receiver check, field store, and stack update using field and these slots ...
}It works. It’s just ugly.
Every arm now describes both the bytecode’s behavior and how we want it compiled. Worse, the desired placement depends on the compiler’s treatment of these directives, so a compiler change may send us back to rearrange the source.
We can instead teach the compiler the tradeoff directly: accept a little more generated code in exchange for shorter live ranges.
We restrict read hoisting and sharing, and teach the scheduler to favor separate copies of arithmetic in the consuming branches. The field metadata loads and slot calculations now stay inside the arms, without the source directives:
; No spills before the switch; dispatch omitted.
int_case:
▶ mov QWORD PTR [rsp + 0x18], regKindObject ; spill decompressed JavaKind reference
▶ mov QWORD PTR [rsp + 0x20], regFieldObject ; spill decompressed field reference
mov regReceiver32, DWORD PTR [regRefs + regTop * 4]
lea regReceiverSlot, [regTop - 2]
test regReceiver32, regReceiver32
je null_receiver_int
; ... value load and modifier handling omitted ...
mov regOffset32, DWORD PTR [regHeapBase + regField * 8 + OFFSET_fieldOffset]
; ... volatility check, receiver decompression, offset widening, and field store ...The long arm computes top-3 instead. Neither arm carries the other’s stack update through the switch.
These choices deliberately work against optimizations we built for general-purpose compilation. Sharing work is usually sensible, but our tiny handlers have tightly constrained register interfaces. We therefore special-case handler stubs, putting this policy in the compiler instead of repeating it through directives in every handler.
Das Leben der Anderen
Even then, spills can remain.
The two stores inside the int arm preserve values for the null-check exception path and a later safepoint slow path, omitted from the assembly excerpt.
Neither slow path uses these values for its actual work. Why keep them alive?
At a call site — including the runtime call on a safepoint poll’s slow path — the compiler may need to describe the caller’s Java execution state for deoptimization, debugging, or runtime stack inspection. Graal records this execution state in a frame state. That can include local variables the call itself never uses, potentially keeping them alive beyond their last executable use. It can even force a reference to be decompressed solely for bookkeeping, although subsequent field accesses need only its compressed form.
In our putfield example, both slow paths retain the decompressed field and JavaKind references alongside other parameters and locals.
Keeping a value alive does not require spilling it immediately. There are enough registers for the live values just before this null check. But these references remain live alongside the temporaries needed for the field store, increasing register pressure inside the arm.
On handler fast paths, Graal’s register allocator favors spilling values with no remaining register-requiring uses when it needs to free a register. Neither decompressed reference has another executable use, making both attractive spill candidates. Here, the resulting interval assignments place the stores before the null check, so the fast path pays too.
One source-level response is to stop giving intermediate results their own locals. For putfield, we could remove the kind local and write the switch directly:
switch (field.kind()) {
// ... same arms ...
}The same approach becomes less pleasant in invokestatic:
static long invokeStaticHandler(long bci, Frame frame, Stack stack) {
CallInfo callInfo = resolveCall(bci, frame);
Object retObj = VM.dispatchCall(
popArguments(
callInfo.argumentCount == 0
? EMPTY_ARRAY
: new Object[callInfo.argumentCount],
frame,
stack,
callInfo),
bci,
frame,
callInfo);
// ... return handling and next BCI omitted ...
}To avoid an intermediate local, we have crammed argument allocation, preparation, and invocation into one expression — an anti-pattern adopted to satisfy the compiler’s bookkeeping.
But our handler stubs cannot deoptimize. Unless debugging or runtime inspection requires their local values, we can discard that bookkeeping.
We therefore teach the compiler which frame-state values these stubs actually need, and prune the rest before register allocation. We retain the BCI used to report the guest’s execution position during stack walking, along with lock information. Live references needed by the GC remain tracked.
The two spill stores disappear from the int arm:
int_case:
mov regReceiver32, DWORD PTR [regRefs + regTop * 4]
test regReceiver32, regReceiver32
je null_receiver_int
; ... value load, field store, stack update, and continuation ...Now a local variable can go back to being a name that helps us read the code.
One Spill Flew Over the Sibling’s Nest
Return to the version before frame-state pruning. The remaining mystery is why the int arm spills both references as soon as it begins.
At the null check, 11 live intervals overlap, against 12 allocatable general-purpose registers in our x64 build. Pressure rises later in the field-store sequence, where 13 intervals overlap.
Why pay for two spills on entry?
The answer starts in another arm.
Graal’s linear-scan allocator lays out basic blocks and instructions along a one-dimensional axis, then represents each value’s lifetime as ranges on that axis. Those intervals can contain gaps and be split into children with different register or stack assignments. Allocation follows this linear order, while liveness analysis and reconciliation between blocks still use the control-flow graph.
In the putfield compilation, the allocator visits the Object arm before the int arm. There, 14 live intervals compete for 12 registers. The decompressed kind and field references become spill victims, freeing registers for the remaining work. The stores appear in the blocks where that pressure rises.
But their stack assignments extend into the later int arm too. The switch still supplies both references in registers. The Object arm’s stores never execute on the int path, so data-flow resolution must insert new stores to satisfy the int arm’s inherited stack assignments. It places them at the arm’s entrance, before the null check.
The int arm has register pressure of its own, but these two stack assignments originate in its sibling. It inherits both spills — and pays for them on entry, even though it may not need to free both registers.
if (safepoint_poll) { safepoint(); } as safepoint(), omitting the conditional branch to keep the control flow focused on the switch arms.We therefore teach the compiler to give a sibling another chance at a register.
When a spilled interval extends into another block, the allocator checks the locations supplied by its actual predecessors. If the stack assignment does not arrive on every incoming path, it splits the interval at that block’s entrance and allocates the new child separately.
The new child prefers the register already supplied by an incoming path. This lets the arms make different choices for the same value.
That independence costs compilation time. Each split requires predecessor checks and creates another interval to allocate, which may itself be split again. Across a large method, this extra work can erode the speed that makes linear scan attractive.
Yet again, we give handler stubs special treatment. After so many amendments to the compiler’s usual rules, an uncomfortable question presents itself.
Why not write assembly?
Act IV: Everything Everywhere All at Once
Well, that is not for me to decide.
When my boss Doug Simon and the team started the Maxine VM project, the idea was Java on Java.
I never convinced them to add __asm__ to Java, so I need an excuse to convince myself.
Luckily, there is one.
Our iconst_0 and putfield examples already use a Stack object, but the handlers still access the backing arrays directly.
Let us encapsulate those accesses in the stack abstraction. Instead of:
class Stack {
long top;
}
static long iconst0Handler(long bci, Frame frame, Stack stack) {
long slot = stack.top;
UNSAFE.putLong(frame.prims, slot * 8L + OFFSET_arrayStart, 0);
stack.top = slot + 1;
return bci + 1;
}We can write:
class Stack {
long top;
void pushInt(Frame frame, int value) {
UNSAFE.putLong(frame.prims, top * 8L + OFFSET_arrayStart, value);
top++;
}
int popInt(Frame frame) {
top--;
return (int) UNSAFE.getLong(frame.prims, top * 8L + OFFSET_arrayStart);
}
}
static long iconst0Handler(long bci, Frame frame, Stack stack) {
stack.pushInt(frame, 0);
return bci + 1;
}Stack now provides an overlay on the frame, while virtual argument expansion keeps that overlay in the source without carrying an object into the emitted assembly.
The handler no longer needs to know where zero goes.
We can now subclass Stack and override pushInt, caching values in instance fields instead of writing them immediately to the backing array.
Again, argument expansion carries these fields between handlers in registers.
That gives us top-of-stack caching.
One Field, Many Roles
Let us return to the simplest cache from the previous article, with two primitive fields, tos1 and tos2, and a template variable tosDepth counting the cached JVM slots. We can view these fields as an extension of Frame.prims: they hold stack slots just like the array elements, except they live in registers. top marks the end of the materialized stack; the cached slots extend it. Object references remain in the backing frame, so before pushing one, we write back the cached values and reset tosDepth to zero.
The following table shows a selection of handlers, how they interpret the cached slots, and what they leave for the next handler.
The types represented by tos1 and tos2 change across handlers, but we need no runtime type tests. The bytecode verifier establishes the operand types, and the cache protocol preserves their representation. The remaining question is how to use those bits without paying for a conversion.
We declare tos1 and tos2 as long, wide enough to hold the bits of any primitive value. Reading an int through an ordinary Java cast introduces a narrowing operation into the compiler graph. Depending on how it is lowered and allocated, that operation can introduce a register move and a separate register for the narrowed value.
For an integer operand, however, our protocol already places the value in the register’s lower 32 bits. We use a compiler directive to read those bits as an int, without introducing a separate narrowing operation. The upper bits need not be zero; a 32-bit use simply does not read them. Writing an integer back into a cache field needs the matching treatment: preserve its low 32 bits without requesting Java’s signed widening to long. unpackInt ignores the upper bits; packInt leaves them unspecified. On x64, the 32-bit arithmetic already supplies the bits we need.
That leaves two more mysterious entries in the table: Preserved and Unconstrained. A preserved value must reach the next handler unchanged, but passing it through requires no knowledge of its type.
For nop(tosDepth=2), both fields have Schrödinger’s type: their bits have a meaning, but we have no reason to look. Matching incoming and outgoing ABI registers lets the allocator carry these values through without moves, provided it keeps those registers intact.
Passing the bits through is enough until we need to write them back to the frame. Then we must know how much stack space they occupy. Our current state machine avoids that ambiguity by counting slots rather than values: a fully cached long uses both cache fields, including its padding.
Caching two longs would require a state that tells us more. We will come back to this later.
An unconstrained field need not preserve even its bits. For nop(tosDepth=1), we can pass whatever bits happen to occupy the register for tos2. We express this freedom through compiler directives, allowing the allocator to reuse that register inside the handler without preserving or restoring its contents. We need not even put zero there before the next handler.
We have already complained about directives cluttering the source.
Fortunately, we can sweep this machinery under the Stack overlay’s carpet. The handler compilation knows the concrete stack implementation, so it can inline the overridden operations.
// Directive names in this illustrative example describe their intended contracts.
class CachedStack extends Stack {
static final int NO_SLOTS_CACHED = 0;
static final int ONE_SLOT_CACHED = 1;
static final int TWO_SLOTS_CACHED = 2;
long tos1;
long tos2;
int tosDepth = NO_SLOTS_CACHED;
@Override
void pushInt(Frame frame, int value) {
switch (tosDepth) {
case ONE_SLOT_CACHED -> {
tos2 = GraalDirectives.packInt(value);
tosDepth = TWO_SLOTS_CACHED;
}
case TWO_SLOTS_CACHED -> {
UNSAFE.putLong(frame.prims, top * 8L + OFFSET_arrayStart, tos1);
top++;
tos1 = tos2;
tos2 = GraalDirectives.packInt(value);
// The cache stays full: 2 → 2.
}
default -> {
assert tosDepth == NO_SLOTS_CACHED;
tos1 = GraalDirectives.packInt(value);
tosDepth = ONE_SLOT_CACHED;
}
}
}
@Override
int popInt(Frame frame) {
return switch (tosDepth) {
case ONE_SLOT_CACHED -> {
tosDepth = NO_SLOTS_CACHED;
long result = tos1;
tos1 = GraalDirectives.unconstrainedValue();
yield GraalDirectives.unpackInt(result);
}
case TWO_SLOTS_CACHED -> {
tosDepth = ONE_SLOT_CACHED;
long result = tos2;
tos2 = GraalDirectives.unconstrainedValue();
yield GraalDirectives.unpackInt(result);
}
default -> {
assert tosDepth == NO_SLOTS_CACHED;
top--;
yield (int) UNSAFE.getLong(frame.prims, top * 8L + OFFSET_arrayStart);
}
};
}
}The handler sees only stack operations:
static long iadd(long bci, Frame frame, Stack stack) {
int right = stack.popInt(frame);
int left = stack.popInt(frame);
stack.pushInt(frame, left + right);
return bci + 1;
}One Handler, Many Variants
Let us manually inline the two pops and the push in iadd, keeping only the operations that affect tosDepth:
static long iadd(long bci, Frame frame, CachedStack stack) {
// First popInt.
switch (stack.tosDepth) {
case ONE_SLOT_CACHED -> stack.tosDepth = NO_SLOTS_CACHED;
case TWO_SLOTS_CACHED -> stack.tosDepth = ONE_SLOT_CACHED;
default -> { }
}
// Second popInt.
switch (stack.tosDepth) {
case ONE_SLOT_CACHED -> stack.tosDepth = NO_SLOTS_CACHED;
case TWO_SLOTS_CACHED -> stack.tosDepth = ONE_SLOT_CACHED;
default -> { }
}
// pushInt.
switch (stack.tosDepth) {
case ONE_SLOT_CACHED -> stack.tosDepth = TWO_SLOTS_CACHED;
case TWO_SLOTS_CACHED -> { }
default -> stack.tosDepth = ONE_SLOT_CACHED;
}
return bci + 1;
}The previous article introduced handler specialization: give the incoming template variable a constant value and let constant folding do its job. For iadd(tosDepth=2), each state update determines which arm of the next switch survives. The markers below trace that sequence:
static long iadd_2(long bci, Frame frame, CachedStack stack) {
// Represents the incoming template constant, not a runtime assignment.
stack.tosDepth = TWO_SLOTS_CACHED;
// First popInt.
switch (stack.tosDepth) {
case ONE_SLOT_CACHED -> stack.tosDepth = NO_SLOTS_CACHED;
▶ case TWO_SLOTS_CACHED -> stack.tosDepth = ONE_SLOT_CACHED;
default -> { }
}
// Second popInt.
switch (stack.tosDepth) {
▶ case ONE_SLOT_CACHED -> stack.tosDepth = NO_SLOTS_CACHED;
case TWO_SLOTS_CACHED -> stack.tosDepth = ONE_SLOT_CACHED;
default -> { }
}
// pushInt.
switch (stack.tosDepth) {
case ONE_SLOT_CACHED -> stack.tosDepth = TWO_SLOTS_CACHED;
case TWO_SLOTS_CACHED -> { }
▶ default -> stack.tosDepth = ONE_SLOT_CACHED;
}
return bci + 1;
}Restoring the operand operations, constant folding and dead-code elimination leave:
static long iadd_2(long bci, Frame frame, CachedStack stack) {
int right = GraalDirectives.unpackInt(stack.tos2);
int left = GraalDirectives.unpackInt(stack.tos1);
stack.tos1 = GraalDirectives.packInt(left + right);
stack.tos2 = GraalDirectives.unconstrainedValue();
stack.tosDepth = ONE_SLOT_CACHED;
return bci + 1;
}The corresponding assembly is compact:
; regBCI: incoming bci (long)
; regFrame: incoming frame (non-null)
; regPrims: incoming frame.prims (non-null)
; regRefs: incoming frame.refs (non-null)
; regCode: incoming frame.code (non-null)
; regTop: incoming stack.top (long)
; regTos1: incoming stack.tos1 (long)
; regTos2: incoming stack.tos2 (long)
add regTos1_32, regTos2_32 ; Java int addition
inc regBCI
jmp tail_call_target(regCode, regBCI, ONE_SLOT_CACHED)The same process derives iadd(tosDepth=0) and iadd(tosDepth=1), which fetch two operands or one operand from the backing frame, respectively.
The handlers describe bytecode behavior; the stack implementation supplies caching behavior wherever those operations occur.
Inlining brings them together, and constant folding derives each specialized variant. This is an aspect-oriented way to implement the optimization.
We still design the transitions in the stack abstraction, but need not calculate the outgoing tosDepth separately for every handler. The compiler fills in that column for us. That state selects the next already-specialized handler at the threaded tail call.
That only works while the compiler can determine the outgoing state on each path. Different branches may lead to different constant states, but invocation bytecodes are less cooperative: they may pop arguments in a loop whose iteration count depends on the target method. Our implementations flush the cache before that loop and finish with a known empty-cache state. Code that operates directly on the backing frame likewise needs the cached values written back first.
Flushing is less alarming than it sounds. These are the stack stores the uncached interpreter would have performed earlier; caching merely postpones them, and may eliminate some before a flush is needed.
Writing cached values back does not require allocating the overlay object. It can remain virtual, provided we do not let it escape through a call.
One Interpreter, Many Stack Implementations
Now we can change the stack implementation without rewriting the handlers. A different subclass can encode a different state machine.
Our first model spends two cache fields on one long, including its padding, because the state counts slots rather than values.
But the state could tell us more.
Let <CAT1> mean an int or float, occupying one JVM slot, and <CAT2> mean a long or double, occupying two. We then have six populated states: <CAT1>, <CAT2>, <CAT1,CAT1>, <CAT1,CAT2>, <CAT2,CAT1>, and <CAT2,CAT2>, with pairs ordered from bottom to top. Seven, including the empty cache.
Each cache field now holds a complete payload. The state tells materialization how much space to reserve for it in the backing frame.
For <CAT1,CAT1>, we write two integers or floats and advance top by two slots. For <CAT2,CAT2>, we write two longs or doubles and advance top by four. The fields still extend the backing array, but the number of slots they represent depends on the state.
Two registers. Now room for two longs.
We can go further and distinguish the actual primitive types: <INT>, <FLOAT>, <LONG>, <DOUBLE>, and ordered pairs such as <INT,LONG> or <DOUBLE,FLOAT>. That lets the stack implementation choose a register representation as well as a slot width. We keep integer payloads in long fields and add double fields for floating-point payloads, which can travel in XMM registers on x64. A float retains its raw bits rather than being numerically widened to a double. Consecutive floating-point handlers can then operate on those payloads without shuttling them between general-purpose and floating-point registers.
The state determines which fields contain valid payloads and must be written back during materialization; unused representations must not be treated as stack values. The expansion configuration exposes that validity to the compiler, so those unused fields need not occupy registers either.
There is a bill for all this information. Four single-value states and sixteen ordered pairs, plus the empty state, give us twenty-one states. Each incoming state creates another handler variant to compile, and each viable variant contributes generated code.
Some handler/state combinations are impossible, and we can say so. The typed popInt rejects a cached operand whose kind is not INT using an unreachable-path directive. For iadd entering in <DOUBLE,INT>, the first pop succeeds, but the second encounters the cached double. Once specialization exposes that contradiction, the compiler propagates the unreachable marker back to the entry, discarding the preceding work and reducing the variant to a tiny trap. This reduces both the remaining compilation effort and the emitted code size.
To reduce the number of variants further, we can also choose not to cache some perfectly legal combinations.
For example, we could keep only the empty state, the four single-type states, and the four same-type pairs: nine states. A push that would create a mixed-type pair would instead write the existing cache and the incoming value to the frame, then return to the empty-cache state. <INT,DOUBLE> is legal for DASTORE, but we can still omit it and flush instead.
Whether we add states or drop them, the changes stay in the stack subclass and its expansion configuration.
The handlers still push, pop, and compute.
The Fast and the Curious
With those pieces in place, let us step back and see how far we have come.
We threaded the handlers, then used the verifier’s knowledge to remove checks and widen their indices. We taught the compiler to make different tradeoffs for these tiny handlers. Finally, we cached the top of the stack, keeping more interpreter state in registers.
To see how these changes translate into performance, we selected Sieve and Mandelbrot from the Are We Fast Yet suite. Both are arithmetic-heavy workloads that exercise the handler and operand-stack operations discussed above. All measurements are interpreter-only, with JIT compilation disabled.
Our starting point is Crema before threaded tail calls. For TOS, we compare the basic slot cache, the category cache, and the typed cache.
Threading gives both workloads a substantial boost, but neither catches HotSpot until we teach the compiler the handler-specific tradeoffs.
With the basic cache, Sieve runs about 48% faster than HotSpot; Mandelbrot, about 34%. Sieve heavily exercises baload and bastore.
Two differences likely help here. We quicken these bytecodes for the array type; in particular, bastore no longer needs HotSpot’s byte-versus-boolean check. Our two-entry primitive cache can also keep both the index and value in registers, whereas HotSpot caches only the top operand.
The category cache brings Mandelbrot to 41% faster. Mandelbrot is heavy on double arithmetic: the basic slot cache holds only one double, while the category cache can keep both operands in registers.
The typed cache is the interesting twist. Although llvm-mca estimates 7.5% fewer cycles than the category-cache version for the interpreter instruction sequence corresponding to Mandelbrot’s hot loop, the typed cache runs slower in the benchmark. One possible culprit is code size: more handler variants mean a larger instruction footprint, potentially increasing pressure on the instruction cache.
Two arithmetic-heavy workloads do not settle the performance of an entire interpreter. They do show that readable Java and competitive interpreter code can coexist — even when getting there takes quite a bit of teaching.
The End Credits
We started by asking how close ordinary Java could get to the assembly we wanted. The answer depended on how we divided the work between the interpreter’s source code and the compiler. The source describes the execution model and supplies knowledge the compiler cannot discover on its own. The compiler turns that description into machine code, sometimes with policies tailored to the interpreter.
The harder question is where the design should live. When the design lives only in comments or in the programmer’s head, the implementation can drift away from it. Expressed through the right abstraction, it becomes something we can maintain and extend — and something the compiler can turn into efficient code.
Escape analysis deserves a credit of its own. Together with scalar replacement, it lets an object remain a useful source abstraction without requiring an object in the generated code. Its fields become independent compiler values that can live in registers. Argument expansion carries that idea across handler boundaries. The design stays in the source. Its objects need not survive into the assembly.
We still care about every load, move, and spill. We would just rather not write each one ourselves.
There is still call overhead to address.
That is where we go next.
