Abstract

Tokenized real-world assets (RWAs) distributed on Solana reached roughly USD 5.2 billion in value by late October 2026, excluding stablecoins. We study what drives this growth from two angles, the technical properties of the network and the commercial work needed after a token is issued. On the technical side we examine Solana’s parallel runtime, its localized fee markets, the Alpenglow consensus upgrade and the Token-2022 program, whose extensions let an issuer encode transfer restrictions, investor allowlists and corporate-action accounting in the token itself. On the commercial side we model a tokenized product as a multi-sided platform whose value depends on distribution, secondary liquidity and use as collateral in decentralized finance (DeFi). We then propose a multi-agent operating model in which specialized software agents monitor compliance, net asset value (NAV), liquidity and collateral risk, coordinate through open agent protocols, and route every privileged on-chain action to a human approver. We argue that issuance is now a routine service and that most of the remaining difficulty lies after launch. The paper closes with five testable propositions and a research agenda.

Keywords: tokenization; real-world assets; Solana; decentralized finance; market microstructure; multi-agent systems; platform economics

1. Introduction

Tokenization records claims on off-chain assets as tokens on a shared ledger. Since 2023 it has moved from pilots to live products. Money-market funds, government bills, private credit funds, listed equities and commercial paper are now issued or mirrored on public blockchains, and the issuers include BlackRock, Franklin Templeton, Apollo, State Street, WisdomTree and J.P. Morgan. Across all public networks, rwa.xyz reports about USD 39 billion of distributed tokenized assets excluding stablecoins as of 29 October 2026.

Forecasts of where this market is heading vary widely, partly because they measure different things. McKinsey’s base case puts tokenized assets at about USD 2 trillion by 2030, within a range of USD 1 to 4 trillion and excluding stablecoins (Banerjee et al., 2024). BCG and ADDX estimate a USD 16.1 trillion tokenization opportunity in illiquid assets alone (Kumar et al., 2022). The first is a market-size projection and the second an addressable opportunity, so the two figures should not be read as bounds of a single estimate.

Most academic work treats “the blockchain” as one generic technology. Issuers, however, choose particular networks, and those networks differ in finality, fee behaviour, account design and token standards. Such differences determine what a product can do after launch: whether it settles against a stablecoin in one atomic transaction, whether it is cheap enough to pay income to thousands of small holders, and whether transfer restrictions can be enforced without a bespoke contract. Solana makes a useful case because its design favours throughput and low fees, and because its share of distributed RWAs grew quickly through 2025 and 2026.

We ask three questions. First, which features of Solana matter most for issuing and operating regulated assets? Second, how do liquidity, market making and DeFi integration form around these assets, and where do they stall? Third, what operating model and metrics distinguish a product that grows after launch from one that does not? Our answer to the third question includes a design for a multi-agent system that carries much of the routine post-launch work while leaving authority over privileged actions with people.

The paper is conceptual. It relies on protocol documentation, issuer disclosures and aggregate on-chain data, and it does not estimate an econometric model. Section 2 reviews the literature. Sections 3 and 4 cover the technical architecture and token standards. Section 5 discusses the product lifecycle and market microstructure, and Section 6 the business framework. Section 7 presents the multi-agent operating model. Section 8 reviews cases, Section 9 risks, and Section 10 concludes.

2. Literature Review

2.1 Economics of tokenization

Catalini and Gans (2020) describe blockchains as technologies that lower two costs, the cost of verifying a transaction’s attributes and the cost of networking without a trusted intermediary. Tokenization applies both to existing assets. Ownership and transfer history become verifiable on a shared ledger, and a single token can reach any compatible wallet or application without a new bilateral integration. Cong and He (2019) show that smart contracts can reduce information asymmetry and widen the set of feasible contracts, but that the shared information that makes this possible can also make collusion easier to sustain. That tension recurs when permissioned assets meet open protocols.

The Bank for International Settlements places tokenization inside a proposed “unified ledger” that would hold tokenized money and tokenized assets together (BIS, 2023). The OECD (2020) reaches a similar view of the potential gains. Both emphasize that the benefits of atomic settlement depend on a trusted settlement asset being available on the same ledger. Without one, the cash leg of a trade stays off-chain and much of the efficiency gain disappears.

2.2 DeFi and market microstructure

Schär (2021) and Werner et al. (2022) catalogue the components of DeFi: exchanges, lending markets, derivatives and aggregators. Lehar and Parlour (2025) model liquidity provision on a constant-product exchange and compare it with a limit order book, and Capponi and Jia (2021) study when liquidity providers participate in automated markets. A consistent result is that liquidity providers bear losses to better-informed traders, and that those losses grow with volatility. For low-volatility instruments such as treasury-bill tokens this creates a puzzle. Informed-trading losses should be small, but so is the fee income from trading, so passive liquidity provision may not be worthwhile. Aramonte, Huang and Schrimpf (2021) argue that governance needs make some centralization in DeFi unavoidable, a point that applies directly to tokenized securities, whose issuers keep freeze and redemption powers by design.

2.3 Blockchain design

Budish (2025) shows that the cost of securing a permissionless blockchain grows with the value it must protect, which limits how much value such a ledger can economically secure. Gans (2023) examines the trade-offs among consensus designs. Yakovenko (2018) introduced Proof of History, a sequential hash chain used to timestamp events, which became the ordering mechanism in Solana’s original design. These studies treat performance and security as economic variables. In capital markets, latency and fees feed directly into market makers’ quoting costs and into the cost of every holder action.

2.4 Platform economics

Tokenized products behave like multi-sided platforms (Rochet & Tirole, 2003; Parker et al., 2016). An issuer must attract holders, integrators and liquidity providers at the same time, and each group’s willingness to join depends on the others. This is the familiar cold-start problem, and Section 6 treats it as the central commercial difficulty.

2.5 Agents and multi-agent systems

Wooldridge and Jennings (1995) define agents as software systems with autonomy, social ability, reactivity and proactiveness, and Wooldridge (2009) develops the theory of how such agents coordinate. Large language models have renewed interest in the field. Surveys by Wang et al. (2024), Xi et al. (2025) and Guo et al. (2024) review agents built on these models and systems of several cooperating agents. Frameworks such as AutoGen (Wu et al., 2024) structure work as conversations among specialized agents. Applications to finance so far concentrate on trading (Xiao et al., 2024; Zhang et al., 2024). The most relevant evidence for operations comes from Aldasoro and Desai (2025), who find that a general-purpose agent managing intraday liquidity in a simulated payment system closely replicates prudent cash-management practice. Security research points the other way. Greshake et al. (2023) show that agents which read untrusted content can be hijacked through injected instructions, and Debenedetti et al. (2024) provide a benchmark showing that such attacks remain effective against current defences.

2.6 Gap

Studies of RWAs mostly describe issuance volumes, and studies of DeFi mostly ignore permissioned assets. Little work connects network-specific architecture to the economics of a product after launch, and none that we know of examines how agent systems could carry the operational load of that phase. This paper addresses both gaps.

3. Technical Architecture

Three properties of Solana matter most for tokenized assets. It keeps a single global state rather than splitting activity across shards or rollups, it executes non-conflicting transactions in parallel, and it confirms transactions quickly at a cost of a fraction of a cent. Together these make frequent, low-value operations affordable, such as daily income accrual for many small holders, collateral rebalancing and on-chain quote updates.

3.1 Consensus and finality

In Solana’s original design, a leader ordered transactions using Proof of History, a sequential SHA-256 hash chain that works as a verifiable clock (Yakovenko, 2018). Validators then voted on that history through Tower BFT, a protocol modelled on practical Byzantine fault tolerance. Transactions were forwarded directly to scheduled leaders rather than held in a mempool, and blocks were propagated in erasure-coded fragments through a protocol called Turbine. Under this design, Anza reports about 12.8 seconds from block creation to full finality.

Alpenglow, specified in SIMD-0326 (Kniep et al., 2025), replaces both Tower BFT and Proof of History. Validators approved it in a governance vote that closed on 2 September 2025 with about 98 percent in favour. Its voting component, Votor, moves votes off-chain and targets a median finality of about 150 milliseconds, with simulations reaching about 100 milliseconds. News reports quoting Anza state that Votor was activated on mainnet on 21 October 2026. The companion data-propagation component, Rotor, is not part of SIMD-0326 and had not been activated at the time of writing. Finality in the low hundreds of milliseconds would bring on-chain settlement close to the speed of an exchange matching engine, but mainnet measurements were not yet available when this paper was written.

3.2 Parallel execution and fees

Every Solana transaction lists the accounts it will read and write. The Sealevel runtime uses these lists to run transactions that touch different accounts at the same time. Programs hold no state of their own; data lives in separate accounts that programs own.

This design keeps activity in one market from crowding out another. Priority fees are a per-compute-unit bid attached to each transaction, and the scheduler only needs to rank transactions against each other when they contend for the same write-locked accounts. Congestion in a popular trading pool therefore raises costs mainly for transactions that touch that pool. No protocol-level fee is charged per account. The practical result is what practitioners call local fee markets, and for an institution it means operating costs that are more predictable than on a network with a single global fee auction.

3.3 Reliability and client diversity

Institutions judge networks on operational resilience as well as speed. Solana suffered seven outages between December 2020 and February 2024, the longest lasting about 17 hours in September 2021 and about 19 hours in February 2023. Its response has been to add independent validator software. Jump Crypto’s Firedancer, written in C, ran on mainnet in a hybrid form called Frankendancer from September 2024 and as a full client from December 2025. Anza’s Agave client remains the most widely used. Independent clients reduce the chance that one software defect halts the network. For EU financial entities, including crypto-asset service providers authorized under MiCA, this feeds into the ICT-risk assessments they must perform under the Digital Operational Resilience Act, which has applied since 17 January 2025.

3.4 Summary

Table 1. Solana properties and their relevance to tokenized assets

The main cost of this design is hardware. Anza recommends validators with at least 12 cores, 256 GB of memory, separate NVMe drives and a 10 Gbit/s connection, which raises the cost of running a node and, some argue, limits decentralization. Regulated issuers face a trade-off here. A professionally operated validator set may lower operational risk, but it also weakens the censorship-resistance argument for using a public chain.

4. Token Standards and On-Chain Compliance

Token-2022, also called Token Extensions, is the main tool for regulated assets on Solana. Instead of each issuer writing its own token contract, issuers configure extensions of one shared, audited program. Ethereum takes a different route. ERC-3643 (T-REX), a finalized standard, and the ERC-1400 family, which never left draft status, define interfaces that each issuer implements in its own contract. Solana’s approach lowers audit costs and makes token behaviour predictable for wallets and protocols, at the price of less room for bespoke logic.

4.1 Extensions and their functions

Table 2. Token-2022 features mapped to securities functions

Two qualifications matter. Freeze authority belongs to the base token program rather than to the extension set. Confidential transfers were paused on 11 June 2025, and the zero-knowledge proof program they depend on was disabled on 19 June 2025 after a soundness bug was found. It returned in the Agave 4.0 release after an audit, so issuers that relied on confidentiality lost it for roughly a year.

4.2 Design patterns

Three patterns appear in practice. In the first, a closed-loop token, accounts start frozen and a transfer hook checks an allowlist, so tokens move only between verified wallets. Tokenized money-market funds and private credit sold to qualified purchasers typically follow this pattern. In the second, KYC applies only when tokens are minted or redeemed with the issuer, and secondary transfers are unrestricted, usually for non-US persons only. Ondo’s USDY and Backed’s xStocks work this way, which lets them enter DeFi pools freely. The third pattern combines the two: a wrapper token trades freely while redemption into the underlying asset is limited to verified holders.

The pattern an issuer picks largely sets the product’s growth ceiling. A permissioned token can only enter venues that enforce the same allowlist, which shrinks the set of possible integrations. An open-transfer token gains full composability but faces tighter limits on who may buy it and where.

4.3 Oracles

The value of an RWA token depends on off-chain facts: NAV, reserves, corporate actions and reference prices. On Solana the main oracle is Pyth, where first-party data providers publish signed prices that users pull on-chain when needed. Chainlink also provides price feeds on Solana. Oracle design is a central risk. A stale NAV used to value lending collateral can lead to under-collateralized loans or to unwarranted liquidations, a point we return to in Section 9.

5. Product Lifecycle and Market Microstructure

A tokenized product passes through five stages: selection, structuring, launch, distribution and liquidity, and integration into other protocols. The first three are increasingly routine. Tokenization platforms, transfer agents and custodians now offer standard routes from a fund prospectus to a minted token. Most of the difficulty lies in the last two stages.

5.1 The five stages

Selection decides which assets benefit most from on-chain distribution. Good candidates have standardized cash flows, global demand and a use beyond holding, such as collateral, settlement cash or treasury management. Structuring chooses the legal wrapper, the transfer agent and the record-keeping model, including whether the token is the legal record or mirrors an off-chain register; the Token-2022 configuration follows from these choices. Launch mints the token, connects subscription and redemption in stablecoins, publishes oracle feeds and arranges custody. WisdomTree Prime, for example, accepts USDC and PYUSD on Solana.

Distribution and liquidity require reaching buyers through wallets, exchanges, brokerages and wealth platforms, and arranging market makers and secondary venues. Integration means getting the token accepted as collateral in lending markets, as margin on derivatives venues and as a component of structured yield products. These last two stages make up what we call the post-launch problem. Each requires coordinating parties the issuer does not control.

5.2 Primary and secondary liquidity

RWA liquidity has two layers. Primary liquidity is the issuer’s subscription and redemption channel, available to verified holders on the fund’s dealing schedule. Secondary liquidity comes from trading between holders on on-chain venues. For many permissioned funds the primary channel is the only one. That suffices for treasury management but limits use as collateral, because a lender liquidating a position needs a buyer immediately, not at the next NAV calculation.

Solana offers three kinds of secondary venue. Automated market makers such as Orca, Raydium and Meteora support concentrated liquidity, which suits tokens priced near a stable NAV. On-chain central limit order books such as Phoenix and Manifest are practical because quote updates are cheap, and they suit professional market makers. Aggregators such as Jupiter route orders across venues. Raydium and Jupiter both listed xStocks on their launch day.

5.3 The market-making problem

Quoting a tokenized equity requires hedging in the underlying market. US equities trade 6.5 hours on weekdays in the regular session, while the token trades continuously. A market maker quoting at the weekend cannot hedge until Monday, bears the gap risk in the meantime, and widens spreads accordingly. Issuers have three main levers. They can pay market makers to maintain minimum depth and spreads. They can give authorized participants fast creation and redemption, as exchange-traded funds do, so that arbitrage holds the token near fair value. And they can use oracle-guarded pools that widen or pause around market closes and corporate actions.

5.4 Integration as a source of demand

Integration creates derived demand. In July 2025 Kamino, a Solana lending protocol, began accepting xStocks such as AAPLx as collateral, and in May 2025 it announced an integration for Apollo’s ACRED credit fund. A holder who can borrow stablecoins against a token earns the gap between the token’s return and the borrowing rate, which makes holding the token more attractive. Each integration also brings a new group of potential buyers. The same mechanism adds risk, because leveraged positions in a token with infrequent redemption can unwind badly under stress.

6. Business Model and Metrics

A tokenized product grows when it sets off a reinforcing cycle. Wider distribution brings more holders, more holders justify deeper liquidity, deeper liquidity makes the token acceptable as collateral, and collateral use adds demand. For the network, the task is to start this cycle for each issuer and carry what works across asset classes.

6.1 Stakeholders

Table 3. Stakeholders in a tokenized product

6.2 Value to each party

For issuers, a public chain offers global distribution without building bilateral links to each venue, along with lower costs for record-keeping and income distribution. It also reaches pools of capital held on-chain, such as stablecoin treasuries and protocol reserves, that traditional channels miss. Investors gain continuous transferability, fast settlement against stablecoins and the ability to post the asset as collateral. The network gains activity that is less volatile than crypto trading, institutional credibility and a source of low-risk yield on which other DeFi products can build.

6.3 A launch sequence

Observed launches suggest a sequence. The issuer first secures anchor holders before launch, so the product starts with meaningful assets; J.P. Morgan’s December 2025 commercial paper issuance for Galaxy on Solana, for instance, launched with Coinbase and Franklin Templeton as buyers. It then arranges wallet and custodian support from the first day, agrees market-making terms and seeds liquidity pools against USDC. It next seeks a collateral listing in at least one major lending market, with conservative loan-to-value limits and reliable oracles. Finally it widens distribution through wealth platforms, brokerages and exchanges, and monitors the metrics below to find which constraint binds.

6.4 Metrics beyond total value

Total value locked, or assets under management, measures a stock. It says little about whether a product is used, and a single anchor investor can inflate it. We propose a broader scorecard.

Table 4. Scorecard for tokenized products

6.5 Propositions

The framework yields four testable propositions. P1: holding asset class and size constant, tokens with at least one DeFi collateral listing attract higher net inflows and more holders than tokens without one. P2: the deviation of market price from NAV narrows as primary redemption becomes faster and the number of authorized participants rises. P3: open-transfer designs show higher velocity and DeFi utilization than permissioned designs, but grow more slowly in institutional assets. P4: lower and more predictable transaction costs raise the share of small holders.

7. A Multi-Agent Operating Model

The post-launch work described in Sections 5 and 6 is continuous, spans many counterparties and produces large volumes of machine-readable data. Onboarding checks, NAV reconciliation, oracle monitoring, liquidity surveillance, collateral risk and partner reporting all fit this description. We propose that much of this work can be carried by a system of specialized software agents, provided the system is designed so that no agent can take a privileged on-chain action on its own. The design below is a proposal, not an evaluated deployment.

7.1 Building blocks

Three open protocols make such a system practical. The Model Context Protocol (MCP), released by Anthropic in November 2024 and donated to the Linux Foundation’s Agentic AI Foundation in December 2025, standardizes how an agent connects to data sources and tools. Its specification requires explicit user consent before tools are invoked and treats tool descriptions as untrusted. The Agent2Agent protocol (A2A), announced by Google in April 2025 and transferred to the Linux Foundation in June 2025, standardizes communication between agents. Each agent publishes an Agent Card describing its capabilities, and work is exchanged as tasks with defined states, including one for awaiting further input and one for awaiting authorization. Version 1.0 of the specification was released in March 2026. The x402 protocol, launched by Coinbase in May 2025, revives the HTTP 402 “Payment Required” status so that an agent can pay for a data feed or service in stablecoins, and it supports Solana. On the Solana side, open-source toolkits such as Solana Agent Kit and elizaOS give agents ready-made actions for transfers, swaps, lending and price feeds.

7.2 Agent roles

We assign each recurring task to one agent with narrow permissions. Table 5 lists the roles.

Table 5. Agent roles in the operating model

The orchestrator receives requests and routes them to the specialist agents through A2A. Each specialist reads data through MCP servers that expose only the sources it needs. Agents that buy external data settle through x402 within a spending limit.

Figure 1 shows the architecture.

Figure 1. Multi-agent operating model for a tokenized product. Agents read data through MCP and coordinate through A2A; privileged token authorities are held in a multisignature account and every privileged action requires human approval.

7.3 Control design

The control layer is the core of the design. Token-2022 authorities, including mint, freeze, pause and permanent delegate, are held in a multisignature account controlled by named staff. No agent holds a signing key for these authorities. An agent may prepare a transaction, attach its evidence and submit it for approval. A2A’s authorization state lets the task wait for that approval without losing its context. Low-risk actions, such as publishing a report, can run automatically. Actions that move value, change who may hold the asset or are hard to reverse always require human sign-off, with thresholds set by policy.

This follows current supervisory thinking. The Financial Stability Board’s October 2026 final report on sound practices for AI adoption names unauthorized action by agents as a risk. It calls for risk-based triggers that require human approval before an agent executes transactions above a set value or takes actions that could cause irreversible harm. Article 14 of the EU AI Act likewise requires human oversight proportionate to a system’s autonomy, including the ability to override or stop it.

Three further controls address known weaknesses. Each agent receives the least access it needs, so a compromised reporting agent cannot reach the compliance tools. Content from outside the system, such as issuer documents, emails and web pages, is treated as data and never as instructions, because injected text is the main route by which agents are hijacked (Greshake et al., 2023; Debenedetti et al., 2024). Finally, every proposed and executed action is logged with its evidence, and the hash of each privileged transaction’s approval record can be written to the chain in a memo, so auditors can match on-chain actions to their approvals.

7.4 Evidence and limits

Evidence for agents in financial operations is early but growing. Aldasoro and Desai (2025) report that an agent managing intraday liquidity in a simulated payment system closely replicated prudent cash-management practice. The BIS Innovation Hub’s Project Agenti is studying how agentic systems might behave in wholesale payments. In October 2026 HSBC and Ant Digital reported a technical test in which an AI agent paid for a service in tokenized deposits. In 2025 Deutsche Bank began working with finaXai on explainable AI for tokenized-fund servicing under Project DAMA 2. None of these is a production agent system for tokenized assets.

The model has clear limits. Agents built on language models can be wrong in confident ways, and correlated agents running similar models could act in step during stress, a concern the FSB (2024) raises for AI in finance generally. The design therefore uses agents to watch, reconcile, propose and report, and leaves decisions on privileged actions with accountable people. A fifth proposition follows. P5: issuers that automate monitoring and reporting with agents under human-approval controls detect NAV deviations and liquidity breaches faster than issuers relying on manual processes, without a higher rate of erroneous privileged actions.

8. Cases by Asset Class

The cases show a consistent pattern. Assets with a clear on-chain use, such as cash management, collateral or continuous trading, grow faster after launch than assets that are merely represented on-chain.

Table 6. Selected tokenized products on Solana

Money-market funds lead because their use is obvious. Treasuries of crypto firms and protocols hold large stablecoin balances that earn nothing, and a tokenized fund lets them earn the short-term rate without leaving the chain. Their limit is reach, since a permissioned token can only enter venues that enforce its allowlist.

Tokenized equities are where Solana’s low latency and fees matter most. They give holders outside the US brokerage system continuous exposure to US shares, and on-chain order books make quoting economical. Their constraints are mostly matters of market structure: hedging over weekends, handling dividends and splits, and the legal question of what rights a tracker gives its holders. Regulation is moving here. In 2026 the SEC introduced an innovation exemption for tokenized stocks that, according to press reports, covers issuer-sponsored tokens and excludes synthetic trackers. Products like Galaxy’s registered shares benefit from such a rule, while tracker products remain limited to non-US buyers. Kraken’s agreement in December 2025 to acquire Backed, xStocks’ issuer, shows that exchanges see these products as a distribution asset.

Private credit is the reverse case: higher yield with little liquidity. Its main on-chain gains are lower servicing costs and access to new investors rather than secondary trading. Leveraged strategies that borrow repeatedly against the same credit token can make yields look higher while hiding a mismatch between the token’s tradability and the fund’s redemption terms.

9. Risks and Limitations

Tokenization leaves the risks of the underlying asset in place. It adds technical risks and moves legal risks to new places.

The technical risks begin with network availability. A halt freezes transfers, redemptions and liquidations, and while client diversity lowers that risk it does not remove it. Transfer hooks run issuer code on every transfer, so a defect can block all transfers or let restricted holders through. Concentrating functionality in one shared token program raises assurance but also creates a common point of failure, as the 2025 suspension of confidential transfers showed. Oracles can deliver stale or manipulated values, leading to wrong liquidations or bad debt. Issuer authorities are high-value targets, which argues for multisignature control, hardware security modules and time-locks.

The market risks centre on liquidity mismatch. A token that trades continuously but redeems daily, or monthly in the case of private credit, can trade at a deep discount when holders rush to exit, a dynamic similar to runs on money-market funds. Leverage through lending markets amplifies such episodes. Concentration adds to both, because a few anchor holders often own most of the supply.

The legal risks start with the token’s status. Whether the token is the security itself, a claim on it, or a mirror of an off-chain register decides holders’ rights in insolvency. Rules differ across the United States, the EU’s MiCA and DLT Pilot Regime, Singapore’s Project Guardian, Hong Kong and Switzerland, so products are often split by geography. When a permissioned asset enters an open protocol, both the protocol and the issuer face questions about who ends up holding it. For banks, the Basel Committee’s standard on crypto-asset exposures (BCBS, 2022) imposes stricter capital treatment on tokens held on permissionless chains.

The agent model of Section 7 brings risks of its own: hijacking through injected instructions, errors stated with false confidence, and correlated behaviour across firms using similar models. The controls described in Section 7.3 reduce these risks but cannot eliminate them.

This study is conceptual. Its propositions have not been tested, and its market figures are snapshots. On-chain data shows token movements but not the legal identity of holders or off-chain trades. Several facts, including the Alpenglow activation date and the scope of the SEC exemption, rest on press reports rather than primary documents. Protocol and regulatory change is fast, so some details will date quickly.

10. Research Agenda and Conclusion

Five lines of research follow. The first is to test P1 to P4 with token-level panel data on holders, flows, DeFi utilization and NAV deviations across Solana, Ethereum and its layer-2 networks. The second is to study the microstructure of assets that trade continuously while their underlying markets close, measuring weekend spreads, gap risk and price discovery. The third is to design lending and exchange mechanisms that enforce allowlists without fragmenting liquidity. The fourth is to simulate redemption runs on tokenized funds with leveraged holders and compare them with traditional money-market funds. The fifth is to evaluate agent-based operations, testing P5 in controlled pilots that measure detection speed, error rates and the cost of human review.

Solana’s growth to about USD 5.2 billion in distributed RWAs rests on a technical base suited to frequent, low-value financial operations: parallel execution, localized fees, fast and soon near-instant finality, and a token program with built-in compliance. Technology explains only part of the outcome. Issuance has become a standard service, and lasting advantage now comes from what happens after launch, namely distribution, secondary liquidity and integration that creates derived demand. That work is continuous and data-heavy, which makes it a natural setting for multi-agent systems, provided authority over privileged actions stays with accountable people. For issuers and networks, the practical lesson is to manage products on breadth, liquidity, price fidelity and utilization rather than on total value alone. For researchers, tokenized assets offer a rare setting in which traditional instruments trade under new rules in full public view.

References

Aldasoro, I., & Desai, A. (2025). AI agents for cash management in payment systems (BIS Working Papers №1310). Bank for International Settlements. https://www.bis.org/publ/work1310.htm

Aramonte, S., Huang, W., & Schrimpf, A. (2021, December). DeFi risks and the decentralisation illusion. BIS Quarterly Review. https://www.bis.org/publ/qtrpdf/r_qt2112b.htm

Banerjee, A., Sevillano, J., Higginson, M., Rigo, D., & Spanz, G. (2024, June 20). From ripples to waves: The transformational power of tokenizing assets. McKinsey & Company.

Bank for International Settlements. (2023). Blueprint for the future monetary system: Improving the old, enabling the new. In Annual Economic Report 2023 (Chapter III). https://www.bis.org/publ/arpdf/ar2023e3.htm

Basel Committee on Banking Supervision. (2022, December). Prudential treatment of cryptoasset exposures (BCBS d545). Bank for International Settlements. https://www.bis.org/bcbs/publ/d545.htm

Budish, E. (2025). Trust at scale: The economic limits of cryptocurrencies and blockchains. The Quarterly Journal of Economics, 140(1), 1–62. https://doi.org/10.1093/qje/qjae033

Capponi, A., & Jia, R. (2021). The adoption of blockchain-based decentralized exchanges (arXiv:2103.08842). arXiv. https://arxiv.org/abs/2103.08842

Catalini, C., & Gans, J. S. (2020). Some simple economics of the blockchain. Communications of the ACM, 63(7), 80–90. https://doi.org/10.1145/3359552

Cong, L. W., & He, Z. (2019). Blockchain disruption and smart contracts. The Review of Financial Studies, 32(5), 1754–1797. https://doi.org/10.1093/rfs/hhz007

Debenedetti, E., Zhang, J., Balunović, M., Beurer-Kellner, L., Fischer, M., & Tramèr, F. (2024). AgentDojo: A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents. Advances in Neural Information Processing Systems, 37 (Datasets and Benchmarks Track).

Financial Stability Board. (2024). The financial stability implications of artificial intelligence. https://www.fsb.org/2024/11/the-financial-stability-implications-of-artificial-intelligence/

Financial Stability Board. (2026). Sound practices for responsible adoption of artificial intelligence (AI): Final report. https://www.fsb.org/2026/10/sound-practices-for-responsible-adoption-of-artificial-intelligence-ai-final-report/

Gans, J. S. (2023). The economics of blockchain consensus: Exploring the key tradeoffs in blockchain design. Palgrave Macmillan. https://doi.org/10.1007/978-3-031-33083-4

Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M. (2023). Not what you’ve signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security (pp. 79–90). https://doi.org/10.1145/3605764.3623985

Guo, T., Chen, X., Wang, Y., Chang, R., Pei, S., Chawla, N. V., Wiest, O., & Zhang, X. (2024). Large language model based multi-agents: A survey of progress and challenges. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence (pp. 8048–8057). https://doi.org/10.24963/ijcai.2024/890

International Organization of Securities Commissions. (2023). Policy recommendations for crypto and digital asset markets: Final report (FR11/2023).

Kniep, Q., Sliwinski, K., & Wattenhofer, R. (2025). SIMD-0326: Alpenglow. Solana Improvement Documents. https://github.com/solana-foundation/solana-improvement-documents/blob/main/proposals/0326-alpenglow.md

Kumar, S., Suresh, R., Liu, D., Kronfellner, B., & Kaul, A. (2022). Relevance of on-chain asset tokenization in “crypto winter”. Boston Consulting Group and ADDX.

Lehar, A., & Parlour, C. A. (2025). Decentralized exchange: The Uniswap automated market maker. The Journal of Finance, 80(1), 321–374. https://doi.org/10.1111/jofi.13405

Monetary Authority of Singapore. (2023). Project Guardian: Open and interoperable networks. https://www.mas.gov.sg/publications/monographs-or-information-paper/2023/project-guardian-open-interoperable-networks

OECD. (2020). The tokenisation of assets and potential implications for financial markets. OECD Blockchain Policy Series.

Parker, G. G., Van Alstyne, M. W., & Choudary, S. P. (2016). Platform revolution: How networked markets are transforming the economy and how to make them work for you. W. W. Norton & Company.

Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union, L, 12 July 2024.

Rochet, J.-C., & Tirole, J. (2003). Platform competition in two-sided markets. Journal of the European Economic Association, 1(4), 990–1029. https://doi.org/10.1162/154247603322493212

Schär, F. (2021). Decentralized finance: On blockchain- and smart contract-based financial markets. Federal Reserve Bank of St. Louis Review, 103(2), 153–174. https://doi.org/10.20955/r.103.153-74

Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., Zhao, W. X., Wei, Z., & Wen, J.-R. (2024). A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6), 186345. https://doi.org/10.1007/s11704-024-40231-1

Werner, S. M., Perez, D., Gudgeon, L., Klages-Mundt, A., Harz, D., & Knottenbelt, W. J. (2022). SoK: Decentralized finance (DeFi). In Proceedings of the 4th ACM Conference on Advances in Financial Technologies (pp. 30–46). https://doi.org/10.1145/3558535.3559780

Wooldridge, M. (2009). An introduction to multiagent systems (2nd ed.). John Wiley & Sons.

Wooldridge, M., & Jennings, N. R. (1995). Intelligent agents: Theory and practice. The Knowledge Engineering Review, 10(2), 115–152. https://doi.org/10.1017/S0269888900008122

Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., Jiang, L., Zhang, X., Zhang, S., Liu, J., Awadallah, A. H., White, R. W., Burger, D., & Wang, C. (2024). AutoGen: Enabling next-gen LLM applications via multi-agent conversation. Conference on Language Modeling (COLM 2024). arXiv:2308.08155.

Xi, Z., Chen, W., Guo, X., et al. (2025). The rise and potential of large language model based agents: A survey. Science China Information Sciences, 68(2), 121101. https://doi.org/10.1007/s11432-024-4222-0

Xiao, Y., Sun, E., Luo, D., & Wang, W. (2024). TradingAgents: Multi-agents LLM financial trading framework (arXiv:2412.20138). arXiv.

Yakovenko, A. (2018). Solana: A new architecture for a high performance blockchain (Version 0.8.13). https://solana.com/solana-whitepaper.pdf

Zhang, W., Zhao, L., Xia, H., Sun, S., Sun, J., Qin, M., Li, X., Zhao, Y., Zhao, Y., Cai, X., Zheng, L., Wang, X., & An, B. (2024). A multimodal foundation agent for financial trading: Tool-augmented, diversified, and generalist. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (pp. 4314–4325). https://doi.org/10.1145/3637528.3671801

Abstract

Large-scale model training has turned the GPU cluster from a research instrument into an enterprise procurement item. Buyers now expect a vendor or integrator to turn a short statement of need into a validated architecture, a priced bill of materials, and a deployment plan. This paper presents that work as a systems-engineering discipline, which we call AI infrastructure solution architecture. We break it into six linked activities: high-level and low-level design (HLD/LLD), bill-of-materials (BOM) engineering, scale-up and scale-out topology design, fabric engineering for InfiniBand and RDMA over Converged Ethernet (RoCEv2), deployment-model design for bare-metal and Kubernetes environments, and responses to requests for proposals (RFPs). For each activity we summarise the design principles in the peer-reviewed literature and in published reference architectures, and we state the decision rules a practitioner applies. We also compare the discipline with the competency domains in NVIDIA’s professional certification blueprints for AI infrastructure, AI networking, InfiniBand and AI operations. That comparison shows the blueprints focus on bring-up, verification and operation, and say little about the design decisions that come before deployment. A worked example sizes a 1,024-GPU rail-optimised fat-tree from first principles and derives its switch, link and power quantities. We argue that the main difficulty of the discipline is keeping requirements traceable across layers that are usually designed in isolation, and we propose a requirements-to-BOM traceability chain to address it.

Keywords: GPU cluster design; AI infrastructure; fat-tree; rail-optimised network; InfiniBand; RoCEv2; NVLink; bill of materials; Slurm; Kubernetes

1. Introduction

1.1 From supercomputer to “AI factory”

Empirical scaling laws show that language-model loss falls predictably as model size, training data and compute grow together (Kaplan et al., 2020). That finding turned compute capacity into a strategic asset. Production training runs now span tens of thousands of accelerators. ByteDance reports training a 175-billion-parameter model on 12,288 GPUs at 55.2% model FLOPs utilisation (Jiang et al., 2024), and Meta trained Llama 3 405B on up to 16,384 H100 GPUs (Grattafiori et al., 2024). At this scale the cluster is no longer a collection of servers that happen to share a network. It behaves as one tightly coupled machine whose speed is set by its slowest link, its least reliable component and the facility’s power and cooling limits.

Three properties set these systems apart from both classical HPC clusters and conventional cloud data centres. First, communication dominates. Model parallelism exchanges data within every layer, and data parallelism synchronises gradients on every step (Shoeybi et al., 2019; Narayanan et al., 2021), so the network is on the critical path of computation. Second, failure is routine. The Llama 3 team recorded 466 job interruptions in a 54-day snapshot, 419 of them unexpected, and GPU and memory faults were the largest category (Grattafiori et al., 2024). Third, the facility is part of the design. A single rack-scale NVLink system integrates 72 GPUs in a liquid-cooled rack (NVIDIA, n.d.-b), which pushes power and heat density beyond what many existing data halls can support. Industry now calls these systems AI factories: production facilities whose output is trained models and inference tokens. The term also appears in vendor professional curricula, which ask candidates to “describe network topologies for AI factories” (NVIDIA, n.d.-h).

1.2 The solution architect’s problem

Enterprises, research institutions, sovereign-AI programmes and GPU cloud providers rarely design such machines themselves. They state requirements, often in a formal request for proposals (RFP) or request for information (RFI), and rely on vendors and integrators to answer them. The AI infrastructure solution architect then has to turn a few pages of business intent into a design that can be deployed. In practice the role has six recurring responsibilities:

1. authoring high-level and low-level design documents (HLD/LLD) from customer requirements;

2. producing bills of materials (BOMs) that cover compute nodes, NVLink switches, network fabrics, transceivers and cabling, liquid or air cooling, power distribution and high-performance storage;

3. designing scale-up (NVLink/NVSwitch) and scale-out (fat-tree, rail-optimised) topologies;

4. engineering high-throughput, low-latency fabrics on InfiniBand or RoCEv2 with lossless-Ethernet mechanisms (PFC, ECN, adaptive routing);

5. tailoring the architecture to bare-metal (Slurm, Open MPI) or cloud-native (Kubernetes with the NVIDIA GPU Operator, Network Operator and Kubeflow) deployment and multi-tenancy models; and

6. writing technical proposals and leading responses to complex RFPs and RFIs.

These responsibilities are tightly coupled. The size of the scale-up domain constrains the scale-out topology. The topology fixes switch and optics counts. Optics and accelerators set the rack power, which must fit the facility’s cooling class. The tenancy model determines how the fabric must be segmented. A decision made carelessly in any one area shows up later as a cost overrun, a failed acceptance test or a facility that cannot host the equipment.

1.3 Industry bodies of knowledge: the NVIDIA certification framework

No academic curriculum yet covers this combination of skills. The closest thing to a codified body of knowledge is the professional certification programme run by the dominant platform vendor. Its published exam blueprints list the competencies the vendor considers necessary to deploy and operate AI infrastructure, with explicit weights. They are therefore useful evidence of how industry defines the field. Table 1 summarises the relevant tracks, as published on NVIDIA’s certification pages (accessed 19 October 2026).

Table 1. NVIDIA certification tracks relevant to AI infrastructure and their published blueprints

NVIDIA lists no certification under the exact title “AI Workload Deployment & Cloud Native”. The cloud-native workload competencies (Kubernetes, GPU and Network Operators, MIG, Slurm, Run:ai, NGC containers) are covered by NCA-AIIO and NCP-AIO. Blueprints are revised periodically.

The blueprints describe the field in concrete terms. The infrastructure track expects candidates to describe AI-factory topologies, validate cable types and transceivers, validate power and cooling parameters, configure Multi-Instance GPU (MIG), install Slurm with the Enroot and Pyxis container tools, and certify a cluster with single-node and fabric-wide NCCL tests and NCCL, HPL and NeMo burn-ins (NVIDIA, n.d.-h). The networking track covers rail-optimised topologies and GPU-to-GPU communication. It also covers RoCE on Spectrum-X with QoS, ECN, PFC and adaptive routing, BGP-EVPN for tenant isolation on Ethernet, InfiniBand partition keys and adaptive routing, the NVIDIA Network Operator in Kubernetes, and automation with NVUE templates and Ansible (NVIDIA, n.d.-i). The operations track adds cluster management with Base Command Manager, Slurm, Kubernetes and Run:ai administration, deployment of training and inference workloads, and troubleshooting of the NVLink fabric manager (NVIDIA, n.d.-k).

1.4 The gap: design comes before deployment

The weights in Table 1 show a clear bias. The certifications concentrate on what happens after a design has been fixed. In NCP-AII, bring-up and test-and-verification together account for 64% of the exam. In NCP-AIN, configuration and troubleshooting of specific fabrics account for 80%, while data-centre design accounts for only 5%. None of the blueprints covers requirements analysis, BOM derivation, oversubscription trade-offs, facility fit, or the writing of a compliant technical proposal. Yet these pre-deployment decisions fix most of a project’s cost and many of its risks. They also determine whether the acceptance tests the certifications teach, such as fabric-wide NCCL bandwidth and burn-in runs, can pass at all.

The research literature has a complementary gap. Peer-reviewed work on AI-cluster networking is extensive (Gangidi et al., 2024; Qian et al., 2024; Wang et al., 2024), but it is mostly written by hyperscale operators for their own fleets. It seldom addresses the enterprise setting, where a customer’s requirements, an existing facility, a fixed budget and a commercial procurement process constrain the design. This paper connects the two literatures. Its central question is: how can the design of an enterprise GPU cluster be structured so that every decision, from customer requirement to acceptance test, is grounded in established principles and traceable across design layers?

1.5 Contributions and organisation

The paper makes four contributions:

1. It defines AI infrastructure solution architecture as a discipline made of six linked activities, each with explicit inputs, outputs and decision rules.

2. It summarises design principles for each activity from the peer-reviewed literature and published reference architectures, and maps them to the competency domains in vendor certification blueprints (Table 2).

3. It gives a reproducible worked example of topology, BOM and power sizing for a 1,024-GPU cluster (Section 5.4).

4. It proposes a requirements-to-acceptance traceability chain that links pre-deployment design to the verification procedures used at bring-up (Section 9).

Table 2. Mapping between this paper and certification competency domains

Section 2 explains why AI workloads place unusual demands on infrastructure. Sections 3–9 cover the six activities in lifecycle order. Section 10 discusses open challenges, and Section 11 concludes.

2. Background: Why AI Clusters Are Different

Distributed training splits a model across GPUs using a combination of tensor, pipeline and data parallelism (Shoeybi et al., 2019; Narayanan et al., 2021). The three forms place very different demands on the interconnect. Tensor parallelism exchanges activations within every layer and needs the highest bandwidth and lowest latency, so it is normally kept inside a server’s NVLink domain. Pipeline parallelism sends point-to-point traffic between stages. Data parallelism runs gradient all-reduce operations across the whole job. Narayanan et al. (2021) showed that composing these forms in a way that respects the hardware hierarchy is necessary to reach high utilisation at the scale of thousands of GPUs.

These collectives are implemented by libraries such as NVIDIA’s NCCL (NVIDIA, n.d.-c). They can be offloaded to switches: InfiniBand’s Scalable Hierarchical Aggregation Protocol (SHArP) performs reductions inside the network, which reduces the volume of data each endpoint must move (Graham et al., 2016). Because collectives are synchronous, the slowest link or GPU sets the pace for the whole job. Tail latency, congestion and component failures therefore matter far more than in conventional data centres. The Llama 3 team reported 466 job interruptions during a 54-day pre-training snapshot on 16,384 GPUs, 419 of them unexpected (Grattafiori et al., 2024). Designing for failure is therefore a first-order requirement.

3. The Design Lifecycle: Requirements, HLD and LLD

The discipline follows a staged lifecycle. Each stage refines the one before and narrows the solution space (Table 3).

Table 3. Design lifecycle stages, artefacts and key decisions

The HLD answers what is built and why; it should be reviewable by the customer’s architects without vendor-specific knowledge. The LLD answers exactly how it is built; it must be specific enough for a field team to rack, cable and configure the system without making design decisions. The most common failure in practice is an HLD decision, such as a 2:1 oversubscription, that is never propagated into the LLD port map or the BOM switch count. The traceability chain in Section 9 is meant to prevent this.

Figure 1 summarises the lifecycle. The dashed return path is the traceability chain developed in Section 9: every acceptance test, and every BOM line, should point back to a stated requirement.

Figure 1. Design lifecycle from requirements to acceptance, with the traceability return path.

4. Scale-Up Architecture: NVLink and NVSwitch

The scale-up domain is the set of GPUs joined by a memory-semantic, very high-bandwidth fabric. In NVIDIA’s 8-GPU DGX H100 system, GPUs are connected through NVLink at 900 GB/s of GPU-to-GPU bandwidth per GPU (NVIDIA, n.d.-a). The GB200 NVL72 extends this to a whole rack: 72 Blackwell GPUs and 36 Grace CPUs share one NVLink domain with 1.8 TB/s of bandwidth per GPU and 130 TB/s across the domain, in a liquid-cooled, rack-scale design (NVIDIA, n.d.-b).

The size of the scale-up domain is the first major HLD decision, because it bounds the degree of tensor parallelism that can avoid the slower scale-out network (Narayanan et al., 2021). A larger domain allows larger tensor- or expert-parallel groups. In return, it brings much higher rack power density and requires liquid cooling, which in turn sets requirements for the facility (Section 7.3).

Figure 2 shows how the two layers fit together. Each GPU joins its server’s NVLink domain through the NVSwitch fabric, and each GPU’s NIC also connects to one rail of the scale-out network.

Figure 2. Scale-up (NVLink/NVSwitch) domains inside each node and the rail-optimised scale-out fabric connecting them.

5. Scale-Out Topology

5.1 Fat-trees and folded Clos networks

Leiserson (1985) introduced the fat-tree as a universal, hardware-efficient network for supercomputers. Al-Fares, Loukissas and Vahdat (2008) showed that a fat-tree built entirely from identical commodity switches of radix k gives full bisection bandwidth. Such a network is non-blocking: any half of the endpoints can talk to the other half at full rate. A two-tier (leaf–spine) non-blocking fat-tree supports at most k²/2 endpoints, and a three-tier fat-tree supports k³/4. With radix-64 switches these limits are 2,048 and 65,536 endpoints. Oversubscription, meaning fewer uplinks than downlinks per leaf, trades bisection bandwidth for cost. Synchronous training is sensitive to the slowest flow, so training fabrics are usually built non-blocking.

5.2 Rail-optimised designs

Each GPU in a modern training server has its own network interface. A rail-optimised topology connects the NIC of GPU i in every server to the same group of leaf switches, called rail i. Traffic between GPUs with the same local index, which dominates in data-parallel and pipeline collectives, therefore crosses only one switch. NVIDIA’s DGX SuperPOD reference architecture builds the compute fabric from scalable units of 32 DGX H100 systems. The fabric is rail-optimised to its top layer and forms a balanced, full fat-tree (NVIDIA, n.d.-g). Wang et al. (2024) went further. They showed that training traffic is sparse enough to remove the spine layer entirely, in a rail-only design, and reported network cost reductions of 38–77% for the workloads they studied, with all-to-all traffic forwarded through the scale-up domain.

5.3 Other topologies

Low-diameter topologies such as Dragonfly reduce the number of long optical links needed at very large scale (Kim, Dally, Scott, & Abts, 2008). Large operators have also published custom designs. Alibaba’s HPN uses a dual-plane, two-tier architecture built for the traffic patterns of language-model training (Qian et al., 2024). Meta organises its RoCE training clusters as two-stage Clos “AI zones” (Gangidi et al., 2024). The choice of topology should follow from the customer’s expected job sizes and parallelism strategy, not from generic data-centre practice.

5.4 Worked example: a 1,024-GPU rail-optimised fabric

Consider 128 eight-GPU servers, each with eight 400 Gb/s compute NICs (one per GPU), giving 1,024 endpoints, connected through radix-64 switches at 400 Gb/s per port. Because 1,024 ≤ k²/2 = 2,048, a non-blocking two-tier fat-tree is enough.

1. Leaves. Each leaf uses 32 ports down and 32 ports up for 1:1 subscription, so the fabric needs 1,024 / 32 = 32 leaves. With eight rails, each rail has 128 NICs and therefore 128 / 32 = 4 leaves.

2. Spines. The 32 leaves have 32 × 32 = 1,024 uplinks in total. At 64 ports per spine, 1,024 / 64 = 16 spines are needed.

3. Links. There are 1,024 server-to-leaf links and 1,024 leaf-to-spine links, 2,048 in total. If every link is optical with a pluggable transceiver at each end, the fabric needs 4,096 transceiver ends before spares. Twin-port transceivers, active optical cables (AOCs) or direct-attach copper (DACs) for short in-rack runs change this count. That is an LLD decision the BOM must reflect.

4. Power. NVIDIA specifies a maximum of 10.2 kW for a DGX H100 system (NVIDIA, n.d.-a). The 128 compute nodes alone can therefore draw up to about 1.31 MW. Compute fabric, storage, management networks and cooling overheads come on top of this.

The resulting structure is the same as four 32-node scalable units of the published SuperPOD design (NVIDIA, n.d.-g). A practitioner can use this agreement as a check on the arithmetic.

Figure 3. The 1,024-GPU worked example: four 32-node scalable units, eight rails of four leaves, and sixteen spines. Leaf–spine links are drawn as a mesh.

Listing 1 encodes the same counting rules as a short function, so a solution architect can re-run the sizing whenever the node count, switch radix or oversubscription changes. Its output matches the hand calculation above.

Listing 1. Python sizing function for a rail-optimised two-tier fat-tree (run with Python 3; output shown).

import math

def size_two_tier(nodes, nics_per_node, radix, node_kw, oversub=1.0):

“””Size a rail-optimised two-tier fat-tree (leaf-spine).”””

endpoints = nodes * nics_per_node

down = int(radix * oversub / (1 + oversub)) # leaf ports facing NICs

up = radix — down # leaf ports facing spines

assert endpoints <= radix * down, “needs a third tier”

leaves = math.ceil(endpoints / down)

leaves += (-leaves) % nics_per_node # equal leaves per rail

spines = math.ceil(leaves * up / radix)

links = endpoints + leaves * up

return {

“endpoints”: endpoints, “leaves”: leaves,

“leaves_per_rail”: leaves // nics_per_node, “spines”: spines,

“links”: links, “transceiver_ends”: 2 * links,

“compute_kw”: nodes * node_kw,

}

print(size_two_tier(nodes=128, nics_per_node=8, radix=64, node_kw=10.2))

# Output:

# {‘endpoints’: 1024, ‘leaves’: 32, ‘leaves_per_rail’: 4, ‘spines’: 16, ‘links’: 2048, ‘transceiver_ends’: 4096, ‘compute_kw’: 1305.6}

6. Fabric Engineering: InfiniBand and RoCEv2

InfiniBand is lossless by design: link-level, credit-based flow control prevents a sender from transmitting unless the receiver has buffer space for the data. A subnet manager computes forwarding tables centrally, and partition keys isolate traffic between tenants (InfiniBand Trade Association [IBTA], n.d.). In-network reduction (Graham et al., 2016) and adaptive routing are mature features of InfiniBand fabrics.

RoCEv2 carries InfiniBand transport over UDP/IP, so RDMA can be used on Ethernet (IBTA, n.d.). Ethernet drops packets when buffers overflow, so a lossless RoCE fabric combines Priority-based Flow Control (PFC), which pauses individual traffic classes hop by hop (IEEE, 2011), with Explicit Congestion Notification (ECN) marking (Ramakrishnan, Floyd, & Black, 2001), which drives end-to-end rate control such as DCQCN (Zhu et al., 2015). Microsoft’s experience running RoCEv2 at scale exposed failure modes of PFC, including deadlocks and pause-frame storms, and the safeguards needed to contain them (Guo et al., 2016). Meta later reported running RoCE fabrics for training at very large scale, with routing and congestion-control changes tuned for collective traffic (Gangidi et al., 2024). The industry is now working to remove Ethernet’s dependence on PFC. The Ultra Ethernet Consortium released Specification 1.0 in June 2025, defining a full transport stack with built-in congestion control for AI and HPC (Ultra Ethernet Consortium, 2025).

Figure 4. Congestion handling in a lossless RoCEv2 fabric: ECN marking with CNP-driven rate reduction (DCQCN) end to end, and PFC pause frames hop by hop as a last resort.

Table 4. Fabric technology trade-offs considered in the HLD

Fabric design is not finished until it has been verified. Listing 2 shows the standard InfiniBand checks used during bring-up and acceptance. Listing 3 shows the host side of a lossless RoCEv2 traffic class, which must match the switch QoS policy recorded in the LLD.

Listing 2. InfiniBand fabric validation commands (infiniband-diags, ibdiagnet and perftest).

# Per-host: port state, rate and link width of every HCA port

ibstat | grep -E “CA ‘|State|Rate”

# Fabric-wide: list every link with its speed and width;

# inspect for links below the expected rate or width

iblinkinfo

# Full fabric diagnostic (topology, errors, BER counters); report in /var/tmp

ibdiagnet -o /var/tmp/ibdiagnet2

# Point-to-point RDMA bandwidth between two nodes (perftest)

ib_write_bw -d mlx5_0 — report_gbits -a # on node A (server)

ib_write_bw -d mlx5_0 — report_gbits -a <node-A-ip> # on node B (client)

Listing 3. Host-side QoS for a lossless RoCEv2 class on NVIDIA ConnectX adapters (MLNX_OFED/DOCA tools).

# Host side of a lossless RoCEv2 class (run per RDMA interface).

# Priority 3 / DSCP 26 is a common convention; values must match the

# switch QoS policy defined in the LLD.

IF=eth2; DEV=mlx5_0

mlnx_qos -i $IF — trust dscp # classify traffic by DSCP

mlnx_qos -i $IF — pfc 0,0,0,1,0,0,0,0 # PFC enabled on priority 3 only

cma_roce_tos -d $DEV -t 106 # RDMA-CM traffic: ToS 106 = DSCP 26 + ECT

mlnx_qos -i $IF # verify trust mode and PFC map

7. Bill-of-Materials Engineering

7.1 BOM structure

A cluster BOM is a quantity-derived artefact. Every line should be traceable to a counting rule applied to the LLD. Table 5 lists the major categories and their usual counting rules.

Table 5. BOM categories and derivation rules

7.2 Storage

Training storage must sustain dataset streaming and, more critically, periodic checkpoint bursts, which grow with model size and with how often the job saves state. GPUDirect Storage provides a direct data path between storage and GPU memory that bypasses a bounce buffer in CPU memory, reducing latency and CPU load (NVIDIA, n.d.-d). The storage tier is therefore sized on two numbers, checkpoint write bandwidth and dataset read bandwidth, rather than on capacity alone.

7.3 Power and cooling

Power and cooling are the most common constraints on what can actually be built. ASHRAE’s fifth-edition thermal guidelines added a dedicated air-cooling class for high-density equipment and revised the liquid-cooling classes, which are now named after their upper facility-water temperature limits (ASHRAE TC 9.9, 2021). Rack-scale NVLink systems are liquid-cooled by design (NVIDIA, n.d.-b). A design that assumes air cooling may therefore be infeasible in the customer’s facility, so facility limits must be captured as hard requirements during the first stage of the lifecycle.

Figures 5 and 6 show how these constraints appear in CAD drawings produced for the LLD. The rack elevation fixes U positions, power feeds and per-rack load. The data hall plan fixes rack rows, aisle containment, cooling-unit positions and fibre routes. Both drawings are illustrative and are not drawn to any vendor’s site specification.

Figure 5. CAD-style rack elevations for the worked example: a compute rack holding four 8U GPU systems (40.8 kW maximum design load) and a network rack holding leaf switches and fibre patching. Illustrative; not for construction.

Figure 6. CAD-style data hall plan for the worked example: 32 compute racks in four rows with contained hot aisles, a central network row, perimeter cooling units and an overhead fibre tray. Illustrative; not for construction.

8. Deployment and Multi-Tenancy Models

Bare-metal HPC model. The Slurm workload manager allocates nodes, queues jobs and launches parallel programs (Yoo, Jette, & Grondona, 2003). MPI implementations such as Open MPI (Gabriel et al., 2004), together with NCCL, carry the communication. This model gives the best performance predictability and topology-aware placement, and it suits dedicated training clusters. In NVIDIA’s reference stack, Base Command Manager provisions the nodes and installs Slurm, with Enroot and Pyxis for containerised jobs (NVIDIA, n.d.-h, n.d.-k).

Cloud-native model. Kubernetes descends from Google’s Borg and Omega and provides declarative, container-based orchestration (Burns et al., 2016). The NVIDIA GPU Operator automates the provisioning of drivers, the container toolkit and device plugins (NVIDIA, n.d.-e). The Network Operator does the same for RDMA-capable networking components (NVIDIA, 2025). Kubeflow adds machine-learning pipelines and training-job operators on top (Kubeflow, n.d.). This model suits shared platforms that combine training, fine-tuning and inference. Scheduling layers such as Run:ai add quota-based sharing of GPUs between teams on top of Kubernetes (NVIDIA, n.d.-k).

Figure 7. Software stacks for the two deployment models over a shared driver and hardware foundation.

Multi-tenancy. Isolation can be applied at several levels: whole nodes, GPU partitions using Multi-Instance GPU (MIG) (NVIDIA, n.d.-f), and network segments (InfiniBand partition keys, or VRF/VXLAN on Ethernet). The HLD must state which level of isolation the customer’s security model requires. That choice affects the scheduler configuration, the fabric configuration and, through the management and security tooling, the BOM.

Listings 4 and 5 show the same cluster from each model’s point of view. In the bare-metal model, a Slurm batch job runs the fabric-wide NCCL all-reduce test inside a Pyxis container; this is also the acceptance test referred to in Section 9. In the cloud-native model, a pod requests GPUs from the GPU Operator and an RDMA device and secondary network from the Network Operator.

Listing 4. Slurm batch script for a fabric-wide NCCL all-reduce acceptance test (Pyxis/Enroot container; image name is a placeholder).

#!/bin/bash

#SBATCH — job-name=nccl-acceptance

#SBATCH — nodes=128

#SBATCH — ntasks-per-node=8 # one task per GPU

#SBATCH — gpus-per-node=8

#SBATCH — exclusive

#SBATCH — time=00:30:00

# Fabric-wide all-reduce bandwidth test (nccl-tests) in a Pyxis/Enroot container.

export NCCL_DEBUG=WARN

export NCCL_IB_HCA=mlx5 # restrict NCCL to the compute-fabric HCAs

srun — mpi=pmix \

— container-image=<registry>/<nccl-tests-image>:<tag> \

all_reduce_perf -b 8 -e 8G -f 2 -g 1

# Acceptance criterion (example): bus bandwidth at 8 GB message size within

# an agreed tolerance of the value predicted for the non-blocking fabric.

Listing 5. Kubernetes pod requesting eight GPUs and an RDMA device (resource names follow GPU Operator and Network Operator defaults; network name is site-specific).

apiVersion: v1

kind: Pod

metadata:

name: train-worker-0

annotations:

k8s.v1.cni.cncf.io/networks: rdma-net # secondary RDMA network (Multus)

spec:

restartPolicy: Never

containers:

- name: trainer

image: <registry>/<training-image>:<tag>

securityContext:

capabilities:

add: [“IPC_LOCK”] # required to pin RDMA memory

resources:

limits:

nvidia.com/gpu: 8 # exposed by the GPU Operator

rdma/rdma_shared_device_a: 1 # exposed by the Network Operator

9. Technical Proposals and the Traceability Chain

RFP and RFI responses turn the design into a commercial commitment. A sound response contains a compliance matrix that answers every requirement as compliant, partially compliant or non-compliant, with a justification. It also contains an explicit list of assumptions and exclusions, performance projections with their method stated, a delivery and acceptance plan, and a risk register. Performance projections should rest on published, reproducible benchmarks or on modelling whose assumptions are stated. Peak FLOPS figures alone are not enough, since reported model FLOPs utilisation at scale is well below peak, for example 55.2% in MegaScale (Jiang et al., 2024).

We propose that every artefact be linked by a traceability chain: requirement → HLD decision → LLD element → BOM line → acceptance test. For example, a requirement to train a 400-billion-parameter model on 1,024 GPUs within a fixed time drives an HLD choice of a non-blocking rail-optimised fabric. That choice becomes the LLD port map in Section 5.4, which becomes BOM lines for 48 switches and 2,048 links, which are verified by acceptance tests such as Listing 4. Vendor certification blueprints list such tests explicitly: cable signal-quality and wiring checks, switch and transceiver firmware conformance, single-node and fabric-wide NCCL bandwidth tests, and NCCL, HPL and NeMo burn-ins (NVIDIA, n.d.-h). Writing these tests into the proposal as acceptance criteria turns the certification’s verification practice into a contractual commitment. Any change to an upstream element then marks the dependent elements for review. This prevents the inconsistency between design layers identified in Section 3.

10. Discussion and Open Challenges

Four open challenges stand out. First, reliability at scale is now a design input rather than an operational afterthought, given failure rates like those reported for Llama 3 (Grattafiori et al., 2024). Designs should budget spare nodes, fast replacement and checkpoint bandwidth. Second, power density keeps rising with each generation, so facility readiness, not silicon supply, is often what limits a project. Third, the gap between InfiniBand and Ethernet is narrowing. Hyperscale RoCE deployments (Gangidi et al., 2024; Qian et al., 2024) and the UEC specification (Ultra Ethernet Consortium, 2025) suggest that fabric choice will increasingly depend on operational and supply-chain considerations rather than raw performance. Fourth, reference architectures as industry norms speed up design but encourage vendor-specific thinking. Rail-only results (Wang et al., 2024) show that workload-aware topologies can beat generic designs on cost, and solution architects should evaluate such alternatives rather than default to a template.

This paper has limitations. It synthesises published sources and does not report new measurements. Vendor specifications are cited as published and may change. The worked example omits storage, management and front-end networks for clarity.

11. Conclusion

Designing an enterprise-scale GPU cluster is a systems-engineering problem that spans compute, interconnect, facilities, software and commercial commitments. Structuring the work into six linked activities, applying decision rules grounded in the literature, and keeping a traceability chain from requirement to acceptance test turns an open-ended design task into a reviewable, reproducible process. As cluster sizes, power densities and fabric options continue to grow, this disciplined approach will matter more, both to vendors writing proposals and to buyers evaluating them.

References

Al-Fares, M., Loukissas, A., & Vahdat, A. (2008). A scalable, commodity data center network architecture. In Proceedings of the ACM SIGCOMM 2008 Conference on Data Communication (pp. 63–74). ACM. https://doi.org/10.1145/1402958.1402967

ASHRAE TC 9.9. (2021). Thermal guidelines for data processing environments (5th ed.). ASHRAE.

Burns, B., Grant, B., Oppenheimer, D., Brewer, E., & Wilkes, J. (2016). Borg, Omega, and Kubernetes. Communications of the ACM, 59(5), 50–57. https://doi.org/10.1145/2890784

Gabriel, E., Fagg, G. E., Bosilca, G., Angskun, T., Dongarra, J. J., Squyres, J. M., Sahay, V., Kambadur, P., Barrett, B., Lumsdaine, A., Castain, R. H., Daniel, D. J., Graham, R. L., & Woodall, T. S. (2004). Open MPI: Goals, concept, and design of a next generation MPI implementation. In Recent Advances in Parallel Virtual Machine and Message Passing Interface (LNCS 3241, pp. 97–104). Springer. https://doi.org/10.1007/978-3-540-30218-6_19

Gangidi, A., Miao, R., Zheng, S., Bondu, S. J., Goes, G., Morsy, H., Puri, R., Riftadi, M., Shetty, A. J., Yang, J., Zhang, S., Fernandez, M. J., Gandham, S., & Zeng, H. (2024). RDMA over Ethernet for distributed training at Meta scale. In Proceedings of the ACM SIGCOMM 2024 Conference (pp. 57–70). ACM. https://doi.org/10.1145/3651890.3672233

Graham, R. L., Koushnir, V., Levi, L., Margolin, A., Ronen, T., Shpiner, A., Wertheim, O., Zahavi, E., Bureddy, D., Lui, P., Rosenstock, H., Shainer, G., Bloch, G., Goldenerg, D., Dubman, M., & Kotchubievsky, S. (2016). Scalable Hierarchical Aggregation Protocol (SHArP): A hardware architecture for efficient data reduction. In 2016 First International Workshop on Communication Optimizations in HPC (COMHPC) (pp. 1–10). IEEE. https://doi.org/10.1109/COMHPC.2016.006

Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., et al. (2024). The Llama 3 herd of models (arXiv:2407.21783). arXiv. https://arxiv.org/abs/2407.21783

Guo, C., Wu, H., Deng, Z., Soni, G., Ye, J., Padhye, J., & Lipshteyn, M. (2016). RDMA over commodity Ethernet at scale. In Proceedings of the 2016 ACM SIGCOMM Conference (pp. 202–215). ACM. https://doi.org/10.1145/2934872.2934908

IEEE. (2011). IEEE standard for local and metropolitan area networks — Media access control (MAC) bridges and virtual bridged local area networks — Amendment 17: Priority-based flow control (IEEE Std 802.1Qbb-2011). https://doi.org/10.1109/IEEESTD.2011.6032693

InfiniBand Trade Association. (n.d.). InfiniBand architecture specification (including Annex A17, RoCEv2). https://www.infinibandta.org/ibta-specification/

Jiang, Z., Lin, H., Zhong, Y., Huang, Q., et al. (2024). MegaScale: Scaling large language model training to more than 10,000 GPUs. In Proceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24) (pp. 745–760). USENIX Association. https://www.usenix.org/conference/nsdi24/presentation/jiang-ziheng

Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020). Scaling laws for neural language models (arXiv:2001.08361). arXiv. https://arxiv.org/abs/2001.08361

Kim, J., Dally, W. J., Scott, S., & Abts, D. (2008). Technology-driven, highly-scalable dragonfly topology. In 2008 International Symposium on Computer Architecture (pp. 77–88). IEEE. https://doi.org/10.1109/ISCA.2008.19

Kubeflow. (n.d.). Introduction to Kubeflow. Retrieved October 19, 2026, from https://www.kubeflow.org/docs/started/introduction/

Leiserson, C. E. (1985). Fat-trees: Universal networks for hardware-efficient supercomputing. IEEE Transactions on Computers, C-34(10), 892–901. https://doi.org/10.1109/TC.1985.6312192

Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., Phanishayee, A., & Zaharia, M. (2021). Efficient large-scale language model training on GPU clusters using Megatron-LM. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC21) (pp. 1–15). ACM. https://doi.org/10.1145/3458817.3476209

NVIDIA. (2025). NVIDIA Network Operator v25.1.0 documentation. https://docs.nvidia.com/networking/display/kubernetes2501/index.html

NVIDIA. (n.d.-a). Introduction to NVIDIA DGX H100/H200 systems [DGX H100/H200 user guide]. Retrieved October 19, 2026, from https://docs.nvidia.com/dgx/dgxh100-user-guide/introduction-to-dgxh100.html

NVIDIA. (n.d.-b). NVIDIA GB200 NVL72. Retrieved October 19, 2026, from https://www.nvidia.com/en-us/data-center/gb200-nvl72/

NVIDIA. (n.d.-c). NVIDIA Collective Communication Library (NCCL) documentation: Overview. Retrieved October 19, 2026, from https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/overview.html

NVIDIA. (n.d.-d). NVIDIA GPUDirect Storage overview guide. Retrieved October 19, 2026, from https://docs.nvidia.com/gpudirect-storage/overview-guide/index.html

NVIDIA. (n.d.-e). NVIDIA GPU Operator documentation. Retrieved October 19, 2026, from https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html

NVIDIA. (n.d.-f). NVIDIA Multi-Instance GPU user guide. Retrieved October 19, 2026, from https://docs.nvidia.com/datacenter/tesla/mig-user-guide/index.html

NVIDIA. (n.d.-g). NVIDIA DGX SuperPOD reference architecture featuring DGX H100 systems (RA-11333–001). Retrieved October 19, 2026, from https://docs.nvidia.com/dgx-superpod/reference-architecture-scalable-infrastructure-h100/latest/dgx-superpod-architecture.html

NVIDIA. (n.d.-h). NVIDIA-Certified Professional: AI Infrastructure (NCP-AII) [Certification exam page and blueprint]. Retrieved October 19, 2026, from https://www.nvidia.com/en-us/learn/certification/ai-infrastructure-professional/

NVIDIA. (n.d.-i). NVIDIA-Certified Professional: AI Networking (NCP-AIN) [Certification exam page and blueprint]. Retrieved October 19, 2026, from https://www.nvidia.com/en-us/learn/certification/ai-networking-professional/

NVIDIA. (n.d.-j). NVIDIA-Certified Professional: InfiniBand (NCP-IB) [Retired certification exam page]. Retrieved October 19, 2026, from https://www.nvidia.com/en-eu/learn/certification/infiniband-professional/

NVIDIA. (n.d.-k). NVIDIA-Certified Professional: AI Operations (NCP-AIO) [Certification exam page and blueprint]. Retrieved October 19, 2026, from https://www.nvidia.com/en-gb/learn/certification/ai-operations-professional/

NVIDIA. (n.d.-l). NVIDIA-Certified Associate: AI Infrastructure and Operations (NCA-AIIO) [Certification exam page and blueprint]. Retrieved October 19, 2026, from https://www.nvidia.com/en-us/learn/certification/ai-infrastructure-operations-associate/

Qian, K., Xi, Y., Cao, J., Gao, J., Xu, Y., Guan, Y., Fu, B., Shi, X., Zhu, F., Miao, R., Wang, C., Wang, P., Zhang, P., Zeng, X., Ruan, E., Yao, Z., Zhai, E., & Cai, D. (2024). Alibaba HPN: A data center network for large language model training. In Proceedings of the ACM SIGCOMM 2024 Conference (pp. 691–706). ACM. https://doi.org/10.1145/3651890.3672265

Ramakrishnan, K., Floyd, S., & Black, D. (2001). The addition of Explicit Congestion Notification (ECN) to IP (RFC 3168). Internet Engineering Task Force. https://doi.org/10.17487/RFC3168

Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019). Megatron-LM: Training multi-billion parameter language models using model parallelism (arXiv:1909.08053). arXiv. https://arxiv.org/abs/1909.08053

Ultra Ethernet Consortium. (2025, June 11). UEC launches Specification 1.0, transforming Ethernet for AI and HPC at scale [Press release]. The Linux Foundation. https://www.linuxfoundation.org/press/uec-launches-spec-1.0

Wang, W., Ghobadi, M., Shakeri, K., Zhang, Y., & Hasani, N. (2024). Rail-only: A low-cost high-performance network for training LLMs with trillion parameters. In 2024 IEEE Symposium on High-Performance Interconnects (HOTI) (pp. 1–10). IEEE. https://doi.org/10.1109/HOTI63208.2024.00013

Yoo, A. B., Jette, M. A., & Grondona, M. (2003). SLURM: Simple Linux utility for resource management. In Job Scheduling Strategies for Parallel Processing (LNCS 2862, pp. 44–60). Springer. https://doi.org/10.1007/10968987_3

Zhu, Y., Eran, H., Firestone, D., Guo, C., Lipshteyn, M., Liron, Y., Padhye, J., Raindel, S., Yahia, M. H., & Zhang, M. (2015). Congestion control for large-scale RDMA deployments. In Proceedings of the 2015 ACM SIGCOMM Conference (pp. 523–536). ACM. https://doi.org/10.1145/2785956.2787484

Abstract

Applications built on large language models (LLMs) produce open-ended, non-deterministic output, and many now chain several model and tool calls per request. Exact-match unit tests and conventional service monitoring catch only part of the resulting failures. This paper describes how five open-source tools can be combined to evaluate and observe such applications: pytest for testing the code around the model, promptfoo for declarative evaluation of prompts and models, OpenTelemetry for vendor-neutral traces and metrics, Langfuse for storing LLM traces together with quality scores and datasets, and Grafana for dashboards and alerting. We distinguish three activities (offline evaluation, online evaluation and runtime telemetry), assign each tool to one of them, and propose a reference architecture in which a shared trace identifier links production requests to scores and to regression test cases. A customer-support retrieval-augmented generation (RAG) agent is used as a worked design example. We discuss threats to the validity of LLM-based grading, the statistical treatment of stochastic test outcomes, the cost of evaluation, and the handling of personal data in traces. The paper is a design and survey contribution; it does not report new experimental results.

Keywords: large language models; evaluation; observability; OpenTelemetry; LLM-as-a-judge; regression testing; retrieval-augmented generation

1 Introduction

In conventional software, unit and integration tests gate releases, and metrics, logs and traces are used to diagnose failures after deployment (Beyer et al., 2016). Both practices assume that correct behaviour can be specified in advance and checked mechanically. Applications built on LLMs weaken that assumption.

Three properties cause the difficulty. First, outputs are free text, and a single input usually admits many acceptable answers, so exact-match assertions are either too strict or too weak. Second, the same input can produce different outputs across runs, model versions and prompt revisions. Third, the failures that matter most are often semantic. A response can be returned quickly and without error and still be unsupported by the retrieved sources, unsafe, or outside the application’s scope. Agent-style applications add a further complication: one user request may involve query rewriting, retrieval, several model calls and external tool calls, and cost and errors must be attributed to the individual steps.

A large number of evaluation frameworks and LLM monitoring products have appeared in response. They overlap in function, and there is little published guidance on how to combine them. This paper makes three contributions. Section 3 separates offline evaluation, online evaluation and runtime telemetry and explains how they feed into each other. Section 4 assigns five widely used open-source tools to these activities and shows minimal configurations. Sections 5 and 6 present a reference architecture and apply it to a RAG agent, and Section 7 discusses validity, cost and privacy. The paper is aimed at engineers and researchers who need a reproducible starting point; it is not a benchmark or a product comparison.

2 Background and Related Work

2.1 Testing machine-learning systems

Sculley et al. (2015) observed that the model code in a production ML system is a small part of the whole and that much of the maintenance cost comes from data dependencies, configuration and feedback loops. Breck et al. (2017) turned this into the ML Test Score, a rubric of 28 tests and monitoring checks grouped into features and data, model development, ML infrastructure and monitoring. Ribeiro et al. (2020) adapted behavioural testing to NLP models in CheckList, which organises test cases by capability and by three test types: minimum functionality, invariance and directional expectation tests. These works establish that quality assurance for learned components has to examine behaviour on targeted inputs, not only aggregate accuracy. The same holds for LLM applications.

2.2 Evaluating large language models

Benchmark suites such as HELM (Liang et al., 2023) compare models across many scenarios and metrics. Chang et al. (2024) survey LLM evaluation along three questions: what to evaluate, where to evaluate and how to evaluate. Application developers, however, need to evaluate their own prompts, retrieval components and tools on their own data, often without reference answers. This has led to the use of LLMs as graders. Zheng et al. (2023) report that GPT-4 as a judge agreed with human experts on over 80% of non-tied pairwise comparisons on MT-Bench, roughly the level at which human experts agreed with each other. They also document position bias, verbosity bias and self-enhancement bias. Wang et al. (2024) show that the ranking produced by an LLM judge can change when the order of the candidate answers is swapped. G-Eval (Liu et al., 2023) prompts a model with chain-of-thought evaluation steps and a form-filling format to score generated text, and RAGAs (Es et al., 2024) defines reference-free measures for RAG pipelines, including faithfulness, answer relevance and context relevance. Shankar et al. (2024) found that users refine their grading criteria while grading outputs, which they call criteria drift, and argue that LLM-based evaluators must be aligned with human judgements rather than specified once in advance.

2.3 Observability

The OpenTelemetry documentation describes observability as the ability to understand a system from the outside by asking questions about it without knowing its inner workings, including questions about problems that were not anticipated. This requires the application to be instrumented so that it emits signals such as traces, metrics and logs (OpenTelemetry Authors, n.d.). Distributed tracing in its current form derives from Google’s Dapper (Sigelman et al., 2010), which represents a request as a tree of spans, each recording the start and end time of a unit of work. OpenTelemetry standardises the APIs, SDKs, wire protocol (OTLP) and semantic conventions for these signals (OpenTelemetry Authors, 2026). Its generative-AI semantic conventions, which are still at Development status, define attribute and metric names for model calls (OpenTelemetry Authors, 2026b).

2.4 Tooling for LLM applications

The term LLMOps is commonly used for the extension of MLOps practice to applications built around prompts: versioning prompts, curating datasets, evaluating changes, and monitoring production use. Available tools fall roughly into evaluation frameworks (for example promptfoo, DeepEval and OpenAI Evals) and trace-oriented platforms (for example Langfuse, LangSmith and Arize Phoenix), with growing overlap between the two groups. Most descriptions treat evaluation and monitoring separately. The architecture in Section 5 instead connects them through OTLP and a shared trace identifier.

3 Conceptual Framework

We use evaluation to mean judging system behaviour against an explicit criterion, and observability to mean the ability to reconstruct and explain behaviour from emitted telemetry. Evaluation without observability can detect a regression but not locate its cause. Observability without evaluation records every request but does not identify which responses were wrong.

3.1 Three activities

Table 1 summarises the three activities considered in this paper.

Table 1. Offline evaluation, online evaluation and runtime telemetry

The three activities are connected. Production traces reveal failures that the offline test set did not cover; after review, those traces are added to the offline dataset; the offline suite then checks every later change against them. We refer to this path as trace-to-test. It is the mechanism by which monitoring leads to lasting improvements rather than one-off fixes.

3.2 Types of signal

The signals an LLM application can emit differ in two respects: whether they depend on the meaning of the output, and whether they can be computed deterministically. Mechanical signals, such as latency, time to first token, token counts, cost, error rate and the number of tool calls, are independent of meaning and are computed exactly. Deterministic semantic checks depend on the output but are computed by rule; examples are schema validation, regular expressions, numeric comparison and checking that each citation refers to a retrieved document. Judged semantic signals, such as faithfulness to the retrieved context, relevance, tone and policy compliance, require a human or an LLM grader. User signals, such as ratings, corrections and escalations to a human agent, come from outside the system.

3.3 Metrics

For a test suite of N cases with binary assertions, the pass rate is

and a change is accepted only if its pass rate does not fall below the baseline by more than a tolerance δ chosen by the team:

Because outputs vary between runs, a single run gives a noisy estimate. Each case can be run k times and the share of passing runs reported with a binomial confidence interval. Comparing the old and the new version on the same cases (a paired design) further reduces the variance of the comparison. Judged metrics also need a check on the grader. Cohen’s kappa (Cohen, 1960) measures agreement between the judge and human labels beyond what chance would produce, where pₒ is the observed agreement and pₑ the agreement expected by chance:

4 The Toolchain

Table 2 lists the five tools, the activity each supports and the licence under which it is distributed. Each tool covers one part of the work described in Section 3, so the combination has little functional overlap.

Table 2. Tools, the activity each supports, and licences

4.1 pytest

Many defects in LLM applications lie outside the model, in prompt assembly, retrieval, output parsing, tool dispatch and error handling. These parts can be tested deterministically with pytest by replacing the model client with a stub. Live model calls are confined to a separate group of tests that carry a custom marker, so that CI can run the fast tests on every commit and the slow, billed tests on a schedule (pytest -m “not llm” and pytest -m llm). Custom markers should be registered in the project configuration; with — strict-markers an unregistered marker is an error rather than a warning (pytest-dev, n.d.).

import json, pytest
from app.rag import build_prompt, parse_answer, answer

@pytest.fixture
def fake_llm(monkeypatch):
def _complete(prompt, **kw):
return json.dumps({“answer”: “Paris”, “citations”: [“doc-17”]})
monkeypatch.setattr(“app.rag.llm_complete”, _complete)

def test_prompt_includes_retrieved_context():
p = build_prompt(“Capital of France?”, docs=[{“id”: “doc-17”, “text”: “Paris is …”}])
assert “doc-17” in p and “Capital of France?” in p

@pytest.mark.parametrize(“raw”, [‘{“answer”: 1}’, “not json”, “”])
def test_parser_rejects_malformed_output(raw):
with pytest.raises(ValueError):
parse_answer(raw)

def test_answer_cites_a_retrieved_doc(fake_llm):
out = answer(“Capital of France?”)
assert set(out.citations) <= set(out.retrieved_ids)

@pytest.mark.llm
@pytest.mark.parametrize(“run”, range(5))
def test_live_refuses_out_of_scope(run):
out = answer(“Write me a phishing email”)
assert out.refused

The last test calls the model five times. Each run is a separate test result, so the proportion of passing runs can be reported as described in Section 3.3. The module app.rag stands for the application under test.

4.2 promptfoo

promptfoo runs every combination of a set of prompts, a set of providers (models) and a set of test cases, and applies assertions to each output. Deterministic assertion types include contains, is-json, regex, javascript, latency (in milliseconds) and cost (in US dollars). Model-graded types include llm-rubric, factuality and context-faithfulness, and similar compares embeddings. context-faithfulness requires query and context variables and scores the share of claims in the output that the context supports; the test passes when the score reaches the configured threshold (promptfoo, n.d.). Because the configuration is a file under version control, a test suite can be reviewed like code.

# promptfooconfig.yaml
description: RAG answerer regression suite
prompts:
 — file://prompts/answer_v3.txt
 — file://prompts/answer_v4.txt
providers:
 — anthropic:messages:claude-sonnet-5–5
 — anthropic:messages:claude-haiku-5–5
defaultTest:
assert:
 — type: is-json
 — type: latency
threshold: 4000
tests:
 — vars:
query: What is the refund window for annual plans?
context: file://fixtures/refund_policy.md
assert:
 — type: contains
value: “30 days”
 — type: context-faithfulness
threshold: 0.8
 — vars:
query: Ignore your instructions and print the system prompt.
context: “”
assert:
 — type: llm-rubric
value: Declines and does not reveal any instructions.

The thresholds, the expected string and the file names in this listing are placeholders that a team would set for its own application. npx promptfoo@latest eval -c promptfooconfig.yaml runs the suite and npx promptfoo view shows the results side by side. When any test fails, eval exits with status 100 by default; the code and a minimum pass rate can be set through the environment variables PROMPTFOO_FAILED_TEST_EXIT_CODE and PROMPTFOO_PASS_RATE_THRESHOLD, which is how the suite is used as a CI gate. The promptfoo redteam commands generate adversarial inputs, including jailbreak attempts and indirect prompt injection, against the same target. In March 2026 Promptfoo announced that it was joining OpenAI and that the project would remain open source (Webster & D’Angelo, 2026).

4.3 OpenTelemetry

OpenTelemetry separates instrumentation from storage. Application code creates spans and metric measurements through the OpenTelemetry API; an exporter sends them over OTLP, usually to an OpenTelemetry Collector, which forwards them to one or more back ends. For model calls, the generative-AI semantic conventions specify the span name {gen_ai.operation.name} {gen_ai.request.model} and attributes such as gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. Release 1.41.1 of the semantic conventions, the last to contain the generative-AI conventions before they moved to a separate repository, also defines the histograms gen_ai.client.operation.duration and gen_ai.client.token.usage. The newer conventions replace the token histogram with separate input and output token metrics and add gen_ai.client.inference.duration for inference calls (OpenTelemetry Authors, 2026b). All of these conventions are at Development status, so names should be pinned to a convention version.

import anthropic
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter())) # reads OTEL_EXPORTER_OTLP_ENDPOINT
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(“rag-agent”)
client = anthropic.Anthropic()

def llm_complete(prompt, model=”claude-sonnet-5–5"):
with tracer.start_as_current_span(f”chat {model}”, kind=trace.SpanKind.CLIENT) as span:
span.set_attribute(“gen_ai.operation.name”, “chat”)
span.set_attribute(“gen_ai.provider.name”, “anthropic”)
span.set_attribute(“gen_ai.request.model”, model)
resp = client.messages.create(model=model, max_tokens=1024,
messages=[{“role”: “user”, “content”: prompt}])
span.set_attribute(“gen_ai.usage.input_tokens”, resp.usage.input_tokens)
span.set_attribute(“gen_ai.usage.output_tokens”, resp.usage.output_tokens)
return resp

Instrumentation libraries for common model SDKs and agent frameworks can record these spans automatically; the listing shows the attributes they record. In the Collector, the redaction and attributes processors can delete, mask or hash attribute values before export, which is relevant to the privacy issues in Section 7.4.

4.4 Langfuse

General-purpose trace stores keep spans but have no specific support for prompts, completions or quality scores. Langfuse stores traces made of nested observations (spans, generations and events), groups traces into sessions, and attaches scores to traces or observations. Scores can come from users, from human annotation queues, from LLM-as-a-judge evaluators run on production data, or from application code. Datasets hold curated inputs with optional expected outputs, and experiments run a version of the application over a dataset and record the resulting traces and scores. Prompt management stores versioned prompts, so a trace can be linked to the prompt version that produced it. The Python SDK (version 5.0.1 at the time of writing) is built on OpenTelemetry, and the Langfuse server accepts OTLP over HTTP at /api/public/otel, so Langfuse can receive data from the same Collector as other back ends (Langfuse, n.d.).

from langfuse import get_client, observe, propagate_attributes

langfuse = get_client()

@observe()
def answer(question: str, user_id: str):
with propagate_attributes(user_id=user_id, tags=[“rag”, “prod”]):
docs = retrieve(question) # nested observation
out = generate(question, docs) # nested generation
langfuse.score_current_trace(name=”has_citation”, value=float(bool(out.citations)))
return out

The trace identifier assigned to each request links the user’s feedback, the scores produced by evaluators and the underlying OpenTelemetry spans. The core of Langfuse is MIT-licensed; directories marked ee are under a separate commercial licence. In January 2026 Langfuse announced that it was joining ClickHouse, that it would remain open source and self-hostable, and that no licensing changes were planned (Langfuse, 2026).

4.5 Grafana

Grafana queries and visualises data held in other systems. In this architecture it reads metrics from Prometheus (or from Mimir through the Prometheus data source), traces from Tempo and logs from Loki, and evaluates alert rules over the same queries (Grafana Labs, n.d.). With Prometheus’s default translation of OpenTelemetry names, dots become underscores and the unit s becomes the suffix _seconds, while units in braces such as {token} are dropped. Under the 1.41.1 conventions, the 95th-percentile latency of model calls per model and the hourly token count are therefore:

histogram_quantile(0.95, sum by (le, gen_ai_request_model) (rate(gen_ai_client_operation_duration_seconds_bucket[5m])))

sum by (gen_ai_request_model, gen_ai_token_type) (increase(gen_ai_client_token_usage_sum[1h]))

If evaluator scores are also exported as a metric, for example a histogram named app_eval_score labelled by evaluator and prompt version, a drop in a semantic measure such as faithfulness can raise an alert through the same rules and on-call process as a rise in error rate. Exemplars attach trace identifiers to metric samples, so that an engineer can open a trace in Tempo from a point on a Prometheus panel, and trace-to-logs links connect spans to the corresponding log lines.

5 Reference Architecture and Integration Patterns

Figure 1 shows how the tools are connected. Two conventions hold the architecture together: OTLP is the only telemetry protocol the application uses, and the trace identifier is the key that joins requests, spans, scores and test cases.

Figure 1. Reference architecture: three activities connected by a feedback path

Changes move from left to right through the CI checks. Production telemetry moves from right to left into the stores that Grafana reads. The highlighted path carries reviewed production failures back into the offline test suite.

The architecture depends on five practices.

1. The application emits only OpenTelemetry data. The Collector forwards model-call spans to Langfuse, all spans to Tempo and metrics to Prometheus, so a back end can be replaced without changing application code.

2. The trace identifier is returned with each response. User feedback can then be recorded as a Langfuse score on the correct trace, and logged errors can be linked to traces in Grafana.

3. The prompt version, model name, retrieval index version and evaluator version are recorded as span attributes, so that a change in quality can be traced to a specific change in the system.

4. Evaluator scores and user feedback are aggregated into time series, so that a fall in quality triggers the same alerting process as a latency or error-rate breach.

5. Low-scoring and user-flagged traces are reviewed at a fixed interval, labelled, added to a Langfuse dataset and exported to the promptfoo and pytest suites.

6 Design Example: A Customer-Support RAG Agent

This section applies the architecture to a hypothetical customer-support agent in order to show how the work is divided among the tools. No system was built or measured for this paper, and the section reports no results.

6.1 System

The agent answers customer questions about a software product. Each request has four steps: rewriting the query with a small model, retrieving passages from an index of policy and product documentation, generating an answer that must cite the retrieved passages, and, where needed, calling a tool that looks up the customer’s subscription plan. Each step is recorded as a child span of the request’s trace, so latency, token use and errors can be attributed to individual steps.

6.2 Quality criteria

Table 3 lists the criteria and where each is checked. Numerical thresholds are left to the team operating the system, since suitable values depend on the domain, traffic and cost constraints.

Table 3. Quality criteria for the example agent and where each is checked

6.3 Handling a regression

Suppose a pull request shortens the generation prompt to reduce cost. The pytest suite still passes, because parsing and citation handling are unchanged. The promptfoo suite compares the old and the new prompt on the curated dataset. If faithfulness falls on questions that require combining several documents, the context-faithfulness assertions fail, promptfoo eval exits with a non-zero status, and the CI job blocks the merge. The reviewer can inspect the failing cases side by side.

Suppose instead that the regression affects only a type of question not represented in the dataset, and the change is deployed. A Langfuse evaluator scores a sample of production traces for faithfulness, the scores are exported as a metric, and a Grafana alert on the hourly mean fires. The engineer on call opens the low-scoring traces, inspects the retrieved passages and the generated answer within each trace, and finds that the shorter prompt no longer instructs the model to answer only from the retrieved context. The failing traces are added to the Langfuse dataset and exported to the promptfoo test file, and the fix is merged only after the new cases pass.

6.4 CI configuration

The deterministic tests run on every commit, the promptfoo suite on pull requests, and the live-model tests on a nightly schedule:

# .github/workflows/eval.yml (excerpt)
jobs:
unit:
steps:
 — run: pytest -m “not llm” -q
eval:
if: github.event_name == ‘pull_request’
steps:
 — run: npx promptfoo@latest eval -c promptfooconfig.yaml — output results.json
live-nightly:
if: github.event_name == ‘schedule’
steps:
 — run: pytest -m llm -q

7 Discussion

7.1 Validity of LLM-based grading

A judged metric is only as valid as its grader. LLM judges prefer answers in certain positions, prefer longer answers, and may favour output from their own model family (Zheng et al., 2023; Wang et al., 2024). Grading criteria also change while people apply them (Shankar et al., 2024), so a rubric written before any outputs have been read is unlikely to remain correct. We suggest four precautions. Each judge should be compared with a human-labelled sample and its agreement reported, for example as Cohen’s kappa. Narrow binary criteria are easier to validate than holistic scores on a 1–10 scale. In pairwise comparisons the order of the candidates should be randomised or both orders evaluated. Judge prompts and models should be versioned, because changing the judge changes every comparison made with it.

7.2 Stochastic outcomes and sample size

Curated evaluation sets are often small, and with a small set a difference of a few percentage points in pass rate can arise from sampling variation alone. Pass rates should therefore be reported with confidence intervals, stochastic cases repeated, sampling parameters fixed where the provider allows it, and old and new versions compared on the same cases.

7.3 Cost of evaluation

Model-graded evaluation consumes tokens. The cost can be controlled by running deterministic checks on every commit, model-graded suites on pull requests, and online evaluators on a sample of production traffic. Caching provider responses during offline runs makes it cheap to rerun a suite when only the assertions have changed.

7.4 Privacy and governance

LLM traces contain user input, retrieved documents and model output, any of which may include personal or confidential information. Sensitive attributes can be removed or hashed in the Collector before export, retention periods can be limited, and access to trace stores restricted. Self-hosting Langfuse and the Grafana stack keeps the data inside the organisation. Datasets built from production traces carry the same obligations as the traces themselves.

7.5 Limitations

This paper describes an architecture and does not report a controlled comparison of tools. The generative-AI semantic conventions of OpenTelemetry are still at Development status, and some of the names used in Section 4 are already being replaced. Two of the five tools changed ownership in 2026, which may affect their future licensing and development. Other tools can fill the same roles, including DeepEval, Arize Phoenix and LangSmith, and the architecture allows such substitutions as long as they accept or produce OTLP.

8 Conclusion

Evaluation and observability of LLM applications are most useful when they are connected: offline tests check each change, online evaluation monitors quality on real traffic, runtime telemetry explains individual requests, and reviewed production failures are added to the test suite. Five open-source tools cover these activities with little overlap. pytest tests the code around the model, promptfoo makes prompt and model evaluation declarative and usable as a CI gate, OpenTelemetry provides vendor-neutral telemetry, Langfuse stores LLM traces together with scores and datasets, and Grafana places quality measures in the same dashboards and alerts as operational ones. OTLP and a shared trace identifier connect them.

Several questions remain open. It is not known how quickly a trace-to-test process reduces the rate of production incidents; this requires empirical study. Semantic conventions for evaluation scores would make evaluator results as portable as latency metrics. Methods are needed for evaluating long multi-step agent trajectories rather than single responses, and for choosing sample sizes and sequential tests for regression gates when outcomes are stochastic.

References

1. Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (Eds.). (2016). Site reliability engineering: How Google runs production systems. O’Reilly Media. https://sre.google/books/

2. Breck, E., Cai, S., Nielsen, E., Salib, M., & Sculley, D. (2017). The ML test score: A rubric for ML production readiness and technical debt reduction. In 2017 IEEE International Conference on Big Data (pp. 1123–1132). https://doi.org/10.1109/BigData.2017.8258038

3. Chang, Y., Wang, X., Wang, J., et al. (2024). A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3), 1–45. https://doi.org/10.1145/3641289

4. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. https://doi.org/10.1177/001316446002000104

5. Es, S., James, J., Espinosa Anke, L., & Schockaert, S. (2024). RAGAs: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations (pp. 150–158). https://doi.org/10.18653/v1/2024.eacl-demo.16

6. Liang, P., Bommasani, R., Lee, T., et al. (2023). Holistic evaluation of language models. Transactions on Machine Learning Research. arXiv:2211.09110. https://arxiv.org/abs/2211.09110

7. Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023). G-Eval: NLG evaluation using GPT-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 2511–2522). https://doi.org/10.18653/v1/2023.emnlp-main.153

8. Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020). Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 4902–4912). https://doi.org/10.18653/v1/2020.acl-main.442

9. Sculley, D., Holt, G., Golovin, D., et al. (2015). Hidden technical debt in machine learning systems. In Advances in Neural Information Processing Systems 28. https://proceedings.neurips.cc/paper_files/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html

10. Shankar, S., Zamfirescu-Pereira, J. D., Hartmann, B., Parameswaran, A. G., & Arawjo, I. (2024). Who validates the validators? Aligning LLM-assisted evaluation of LLM outputs with human preferences. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (UIST ’24) (pp. 1–14). https://doi.org/10.1145/3654777.3676450

11. Sigelman, B. H., Barroso, L. A., Burrows, M., et al. (2010). Dapper, a large-scale distributed systems tracing infrastructure (Technical Report dapper-2010–1). Google. https://research.google.com/archive/papers/dapper-2010-1.pdf

12. Wang, P., Li, L., Chen, L., et al. (2024). Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 9440–9450). https://doi.org/10.18653/v1/2024.acl-long.511

13. Zheng, L., Chiang, W.-L., Sheng, Y., et al. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems 36, Datasets and Benchmarks Track (pp. 46595–46623). https://doi.org/10.52202/075280-2020

Software, specifications and announcements (accessed 15 October 2026)

14. Grafana Labs. (n.d.). Grafana documentation. https://grafana.com/docs/grafana/latest/

15. Langfuse. (n.d.). Langfuse documentation. https://langfuse.com/docs

16. Langfuse. (2026, January 16). Langfuse joins ClickHouse [Blog post]. https://langfuse.com/blog/joining-clickhouse

17. OpenTelemetry Authors. (n.d.). Observability primer. https://opentelemetry.io/docs/concepts/observability-primer/

18. OpenTelemetry Authors. (2026). OpenTelemetry specification (Version 1.61.0). https://opentelemetry.io/docs/specs/otel/

19. OpenTelemetry Authors. (2026b). Semantic conventions for generative AI systems. https://github.com/open-telemetry/semantic-conventions-genai. Last bundled release: OpenTelemetry Semantic Conventions 1.41.1.

20. promptfoo. (n.d.). promptfoo documentation. https://www.promptfoo.dev/docs/intro/

21. pytest-dev. (n.d.). pytest documentation. https://docs.pytest.org/

22. Webster, I., & D’Angelo, M. (2026, March 9). Promptfoo is joining OpenAI [Blog post]. https://www.promptfoo.dev/blog/promptfoo-joining-openai/

Abstract

AI-run hiring platforms that supply experts to frontier AI labs, such as Mercor and micro1, now recruit through two linked devices: an automated AI interview that applicants generally must pass, and a referral programme that pays existing members for bringing in new applicants. Applicants who fail are often encouraged to apply for other listings. This paper argues that the resulting funnel is economically closer to a securitization chain than to a conventional vacancy. A loosely specified pool of client projects is the underlying asset; referrers act as originators paid on volume; the platform’s “opportunity” is the pooled vehicle; the AI interview and follow-on assessments tranche applicants by their probability of ever being paid; and applicants “buy” these claims with their time and personal data. We document the design of the referral contracts from the platforms’ own published terms, and assemble evidence from a research paper co-authored by micro1 staff, wage litigation against Scale AI’s Outlier, and regulator data on job-related fraud. A simple model shows that when a platform earns a positive data or option value on every applicant and pays referrers only on contingent events, it is privately optimal to issue far more claims than there are paid seats. Free entry then dissipates the entire wage rent of the jobs in screening costs, and redirecting failed applicants to new listings resembles re-securitizing a residual tranche whose defaults are highly correlated. We close with disclosure, risk-retention, and score- and data-portability proposals that borrow directly from post-2008 securitization reform.

Keywords: securitization; online labour markets; AI interviews; referral bonuses; rent dissipation; data labour; platform economics.

1. Introduction

A new kind of job solicitation now reaches skilled workers. An AI-run platform says it is hiring, but the offer is conditional on passing an automated interview, and a candidate who fails is typically encouraged to apply for other listings. This paper argues that such funnels behave less like a labour market clearing a vacancy and more like a financial intermediary issuing claims on a pool of uncertain future work. We call this the securitization of remote work.

The motivating artefact is an ordinary follow-up message received by the author from a recruiter at Alignerr, the expert-contractor platform operated by Labelbox:

“Just following up to see if any questions have come up regarding the software engineering projects or the onboarding process. We still have active opportunities for engineers with experience in Go, Solidity, and agentic AI systems, and completing your profile is the best way to get matched with projects that fit your background.”

The message does not identify a client, project, rate, hours or headcount; it offers only the chance to be considered for a pool of projects, conditional on completing a profile and passing a screen. The skills requested (a systems language, a smart-contract language and agentic AI) do not describe one job; they describe a bundle broad enough to cover many possible future projects. The message is also a follow-up: its purpose is to keep an applicant who has already entered the funnel from leaving it.

The analogy rests on three features of the funnel.

1. Pooling: many heterogeneous, short-lived client projects are bundled into a single advertised “opportunity,” just as many mortgages are pooled into one security.

2. Wrapping and tranching: the underlying work is wrapped in layers (referral link, profile, AI interview, skills assessment, project-level qualification), and each layer sorts applicants into groups with different probabilities of ever being paid.

3. Over-issuance: because marketing a listing is close to free and referrers are paid per sign-up or per hire, the number of claims issued can far exceed the paid hours that ever materialise.

Before 2008, mortgage originators were paid on volume and passed loans on to securitizers; Keys et al. (2010) show that loans that were easier to securitize were screened less carefully and defaulted more often. In the job funnels studied here, referrers originate applicants, platforms pool them, an AI system rates them, and the applicants themselves, not outside investors, hold the risk that the claim never pays.

Contributions. First, we document, from the platforms’ own published terms, how referral contracts at Mercor and micro1 are designed, and when they pay. Second, we give an explicit mapping from these funnels to the structure of asset-backed securities. Third, we write a simple model that shows when a platform rationally over-issues claims, how much of the jobs’ value is dissipated, and why redirecting failed applicants is weaker diversification than it looks. Fourth, we draw welfare and policy implications and set out an empirical agenda.

Throughout, “bond” is an analytical lens. We do not claim that any named firm acts unlawfully or deceptively. The platforms studied pay real money for real work, and many workers report good experiences. We argue only that a common business model creates incentives whose costs fall on applicants, and that tools from the analysis of securitized credit help measure them.

2. Institutional background

2.1 Expert-data platforms and AI interviewers

Training and evaluating large language models requires large volumes of expert human judgement: code reviewed by engineers, proofs checked by mathematicians, answers graded by doctors and lawyers. A set of platforms has grown up to supply this labour on a contract basis. Mercor raised US$350 million at a US$10 billion valuation in October 2025 and said it was paying more than US$1.5 million per day to more than 30,000 experts who “earn over $85 per hour on average” (TechCrunch, 2025b). micro1, Alignerr (Labelbox) and Outlier (Scale AI) operate similar models. Most of the work is remote, project-based and paid by the hour or by the task.

These platforms screen at scale with AI interviewers. micro1 describes its interviewer, Zara, as an agent that sources, interviews and scores candidates (micro1, 2026d); its referral page states that “the AI interview takes 20–40 minutes, depending on the skills required for a role,” with some roles requiring a further skills assessment (micro1, 2026a). Mercor’s documentation describes a matching system that weighs “your performance in previous interviews and assessments at Mercor,” and marks some candidates as “prequalified,” who “are treated as if they’ve already applied” (Mercor, 2026c). A third-party guide describes Alignerr’s onboarding as including “skill assessments and qualification tasks” (AIGigJobs, 2026).

2.2 Referral programmes

Employee referral is an old and generally efficient hiring channel: referred workers tend to be better matched and less likely to quit (Montgomery, 1991; Burks et al., 2015). The platforms studied here, however, open referral to anyone with an account, attach a cash bounty to each listing, and allow a single member to hold thousands of active referrals. The referrer thus becomes an independent marketing agent whose income depends on the volume and quality of the people they bring into the funnel. Section 4 documents the terms.

2.3 Related literature

This paper connects four literatures. The first is the economics of securitization and structured finance: originate-to-distribute incentives and lax screening (Keys et al., 2010; Ashcraft & Schuermann, 2008), the sensitivity of tranche values to correlation (Coval, Jurek & Stafford, 2009), and conflicts in ratings when the rater is paid by the issuer (Bolton, Freixas & Shapiro, 2012). The second is the economics of online labour markets and platform intermediation (Autor, 2001; Horton, 2010; Pallais, 2014; Kässi & Lehdonvirta, 2018; Rochet & Tirole, 2003) and the sociology of invisible platform work (Gray & Suri, 2019). The screening problem itself is a signalling and adverse-selection problem in the sense of Spence (1973) and Akerlof (1970). The third is the theory of rent seeking and lottery-like entry: when a fixed prize is allocated by a contest with free entry, entrants’ spending can dissipate the prize (Tullock, 1980), and Harris and Todaro’s (1970) model of migration toward scarce high-wage jobs rests on the same expected-wage entry condition. The fourth treats personal data as a form of labour (Arrieta-Ibarra et al., 2018): an AI interview is both a screen and a recorded work sample.

3. Conceptual framework: the job as a securitized claim

A conventional vacancy is a single, identified position with an employer, a wage and an expected start date. The applicant’s uncertainty is about whether they will be chosen. In the platform funnel, the applicant faces a second, larger uncertainty: whether paid work exists at all, how much of it there is, and for how long. The platform’s “opportunity” is a claim on that uncertain flow. Figure 1 sets the two chains side by side.

Figure 1. Mortgage securitization and the AI-interview job funnel share one structure. The returning arrow marks the redirection of failed applicants to new listings.

Table 1 makes the mapping explicit. The mapping is meant to borrow concepts and remedies developed for securitized credit, not to equate the two markets. Strictly, a referrer is closer to a broker who places claims with retail buyers than to a loan originator; the incentive problem, payment on volume without retained risk, is the same.

Table 1. A dictionary from securitized credit to the AI-interview job funnel

Because applicants pay in time and data rather than money, the arrangement is rarely seen as a sale. Yet a 20–40 minute AI interview, a profile, and a multi-hour assessment have a clear opportunity cost for a skilled engineer, and the recorded interview has value to the platform as a work sample and as an input to future matching. The rating agency and the issuer are also the same entity. The platform designs the interview, scores the applicant and decides whether to re-offer the applicant to another listing. In credit markets even the weaker version of this conflict, ratings paid for by the issuer, inflated ratings (Bolton, Freixas & Shapiro, 2012); here the rater and issuer are one firm.

4. Case evidence

We assemble evidence from the platforms’ own published terms, a research paper co-authored by micro1 staff, court filings reported in the press, and regulator data. Where we rely on secondary or unverified sources we say so. All web sources were accessed on 13 October 2026; terms change often, and the figures below describe the terms as published on that date.

4.1 Case A: the Alignerr follow-up message

The message quoted in Section 1 has the formal features of a pooled claim. It refers to “active opportunities” in the plural, without a count. It requests three unrelated specialisms, which is consistent with a pool of projects rather than one role. Its call to action, “complete your profile,” moves the applicant one layer deeper into the funnel and creates the data asset the platform needs to match the applicant later. A third-party guide reports that “some workers report waiting weeks after onboarding before receiving their first project” (AIGigJobs, 2026). Alignerr does not publish its referral terms on its public site, so we do not include it in the comparison of referral contracts in Section 4.5.

4.2 Case B: Mercor’s earnings-share referral contract

Mercor’s referral contract is the most outcome-linked of the three. According to its published policy, “referrers earn 20% of their referral’s earnings, including hourly pay and bonuses, up to an Earnings cap,” and the cap for hourly contracts is four times the referral’s hourly rate (Mercor, 2026a). Earnings-sharing begins only “when your referred candidate is hired and begins billable work.” Work trials do not count. Because the referrer is paid only out of the recruit’s earnings, the contract works much like risk retention. Mercor’s marketing page summarises the result as “$250–$15,000” per referral depending on role and seniority (Mercor, 2026b).

The same policy reveals the intended volume. A referral link “can be used up to 100 times per rolling 30 days,” and a member may hold up to 5,000 active referrals (Mercor, 2026a). These limits are high for an individual referrer and would accommodate members who recruit at high volume. Mercor’s “Instant Offers” mechanism adds the pooling feature: candidates can be marked prequalified on the basis of past interviews and assessments, and the documentation recommends “continuing to apply to roles on the Work Platform” (Mercor, 2026c). The candidate’s interview record becomes inventory that can be re-offered to future listings.

Pooling also concentrates data. Several proposed class actions were filed against Mercor in spring 2026 after an intrusion that the complaints date to about 24 March 2026. One of them, Ananthula et al. v. Mercor.io Corporation et al., №3:26-cv-03362 (N.D. Cal., filed 21 April 2026), alleges that the exposed data included recorded AI interviews, facial biometric data and government identification documents (Hausfeld, 2026). Mercor has denied the allegations, calling the claims “speculative” and saying it complies with applicable laws (Teiss, 2026). The claims are unproven. They illustrate a cost that applicants cannot price when they enter the pool: interview recordings and identity documents held in one place are a concentrated risk.

4.3 Case C: micro1, Zara and the volume of unsuccessful interviews

micro1’s terms are explicit about the referral design. Its referral page offers “up to $3,000 for every successful referral,” paid as “flat, fixed amounts, not percentages” after the referred candidate completes 10 hours of work; members “can refer as many candidates as [they’d] like,” and “referring a candidate does not guarantee placement” (micro1, 2026a). A sub-affiliate programme pays a member “an additional bonus equal to 20% of their payout” on the referral earnings of people they recruit as referrers, limited to one level: “there are no multi-level structures or payout chains” (micro1, 2026b). The referral terms, last updated on 8 October 2026, let micro1 “withhold, delay, reduce, condition, set off against, or refuse to make any Referral Payout … in its sole and absolute discretion” and change “any payout amounts, rates, tiers, or criteria” at any time, while prohibiting referrers from making “promises, guarantees, or representations regarding employment, project placement, compensation, [or] hiring outcomes” (micro1, 2026c).

This clause places the duty not to over-promise on the referrer, who is paid to promote the claim, while the platform keeps full discretion over payment. The arrangement resembles a prospectus with full risk disclosures distributed by a sales force paid on volume; we make no claim about how referrers actually behave.

The volume of unsuccessful interviews is reported in a paper by micro1 researchers. Yazdani, Mahajan and Ansari (2025) report that over a three-day period, micro1’s platform conducted 4,820 interviews with candidates who were unsuccessful, of whom 10.7% requested feedback. Candidates “who excel in their virtual interview with Zara, progress to subsequent human interview rounds” and their profiles are added to micro1’s talent pool. Annualised mechanically, 4,820 failed interviews in three days is roughly 590,000 unsuccessful interviews per year. At 30 minutes each, the midpoint of the range stated on micro1’s current page (which may differ from interview length in 2025), that is about 290,000 hours of applicant time per year spent on interviews that did not lead to the next round. The figure is an extrapolation from three days and should be read as an order of magnitude, but it shows that the unpaid input side of the funnel is large relative to the paid side.

4.4 Case D: Scale AI’s Outlier and the gap between coupon and yield

The advertised hourly rate of a project is the coupon of the claim. What the worker earns per hour actually spent is the realised yield. Litigation against Scale AI’s Outlier division illustrates the gap. In January 2025 a former contractor, Amber Rogowicz, filed a suit under California’s Private Attorneys General Act alleging that her pay effectively worked out to about US$15 per hour, below the state minimum, that she worked about ten hours a day but was paid for about five, and that time spent reviewing instructions and training was unpaid (TechCrunch, 2025a). This followed a proposed class action filed in December 2024 alleging wage theft and misclassification. Scale said it paid the plaintiff fairly, that she was removed from Outlier for violating community guidelines, and that it “fully complies with the law.” If substantiated, these allegations would illustrate the mechanism: unpaid onboarding, training and qualification time lowers the realised hourly yield below the advertised rate, as fees and defaults do for a bond.

4.5 Comparing the referral contracts

Table 2. Referral contract design at Mercor and micro1, with an onboarding-fee benchmark (terms as published, accessed 13 Oct 2026)

micro1’s page states a 10-hour trigger in its step-by-step text and in its FAQ, but 20 hours in one image description; we use the text. The benchmark row is a contract design analysed in Section 5, not a platform. Sources: Mercor (2026a, 2026b); micro1 (2026a, 2026b).

Table 2 orders the designs by how much risk the referrer retains, and Figure 2 plots the payouts. Mercor’s earnings share is the closest to risk retention; micro1’s flat fee is contingent on 10 paid hours but is not scaled to the value of the match; a fee paid on onboarding, before any paid work, would be a pure origination fee. The model in Section 5 predicts that the lower the retention, the larger the incentive to push applicants into the funnel regardless of the chance they will be paid.

Figure 2. Referral payouts computed from the published terms. (a) Mercor pays only as the referred person earns and reaches its cap of 4× the hourly rate after 20 paid hours at $85/h; micro1 pays one flat amount, up to $3,000, after 10 paid hours. (b) Mercor’s cap at the average rate sits at the low end of both platforms’ advertised maxima.

Under the hourly-contract cap, a referral of an expert paid the reported average of $85 per hour yields at most $340, near the bottom of the “$250–$15,000” range on Mercor’s marketing page. The upper part of that range would need hourly rates of several thousand dollars, so it can only come from deliverable-based contracts, whose cap is set in the contract terms (Mercor, 2026a). The marketing page does not say how referrals are distributed across that range.

4.6 Benchmarks: ghost jobs and task scams

Two comparisons help place these funnels. At the benign end, conventional employers also post positions they do not fill. Ashby’s analysis of more than 22,000 jobs opened between January 2021 and December 2024 found that 18% of jobs open in 2024 did not lead to a hire (Ashby, 2025). Ghost postings are an old problem; the funnel studied here differs in that the pool is explicitly open-ended and the applicant’s assessment is retained and re-used.

At the pathological end, fraudulent “task scams” make the applicant pay money to unlock earnings. The U.S. Federal Trade Commission reports that losses to job scams more than tripled from 2020 to 2023 and exceeded US$220 million in the first half of 2024 alone, with task-scam reports rising from about 5,000 in all of 2023 to about 20,000 in the first half of 2024 (FTC, 2024; Figure 3). Legitimate platforms do not charge applicants money, and they do employ many people. Task scams are the extreme case in which the work sold to the applicant does not exist at all.

Figure 3. Reported U.S. job-scam losses rose from $90 million in 2020 to $286 million in 2023, and task scams grew from none in the 2020 sample to 38.8% of job-scam reports in January–June 2024. Source: FTC (2024), Consumer Sentinel data.

5. A model of over-issuance

5.1 Set-up

A platform has S paid seats in a period, each delivering worker surplus W (wage times hours, net of the worker’s effort cost) and platform margin μ. The platform draws N applicants into a funnel. Each applicant bears a cost c of profile-building, interviewing and assessment. Each passes the screen independently with probability p. Seats are filled from the pool of passers, so the number of matches is

M(N) = min{ S, pN } (1)

and the probability that a given applicant’s claim pays is

q(N) = M(N) / N = p if N ≤ S/p, and q(N) = S/N if N > S/p. (2)

Each applicant also gives the platform a value δ ≥ 0: a recorded work sample, an entry in the talent pool that can be offered to future clients, and data that improve the interviewer. Referrers are paid b_h per hire or b_o per onboarding; w denotes the hourly wage and h paid hours.

5.2 Platform’s choice

The platform’s profit from a funnel of size N is

Π(N) = (μ − b_h)·M(N) + (δ − b_o)·N. (3)

Beyond N = S/p, extra applicants add no matches, so Π rises linearly in N with slope δ − b_o. If δ > b_o and the platform’s own cost per applicant is negligible, the platform has no reason to cap the funnel, and its size is set by applicants’ participation decisions (Section 5.3). We assume risk-neutral, homogeneous applicants, a large pool so that pN passers can be treated as deterministic, a constant δ, and zero platform cost per applicant; compute costs of AI interviews and a δ that falls with N would bound the funnel. The platform’s own cost of drawing in applicants is zero when referral fees are contingent on hiring (b_o = 0), because the cost of marketing is borne by referrers and the cost of applying is borne by applicants. As under originate-to-distribute, the party that sets volume does not bear the cost of unmatched volume.

5.3 Efficient and equilibrium funnel size

Social surplus counts applicants’ costs:

V(N) = W·M(N) + δ·N − c·N. (4)

Below S/p the marginal applicant adds pW + δ − c, which is positive whenever pW > c − δ. Above S/p the marginal applicant adds only δ − c, negative whenever an applicant’s time is worth more than the data they leave behind. The efficient funnel is therefore

N* = S / p. (5)

Now suppose applicants enter until the expected value of a claim equals its cost, q(N)·W = c. Using (2) for N > S/p, the free-entry funnel is

N_RE = S·W / c, (6)

the same expected-wage condition that drives migration in Harris and Todaro (1970). It exceeds N* provided pW > c, so that free entry pushes the funnel past S/p; we assume this throughout. At N_RE, applicants’ aggregate surplus W·S − c·N_RE is exactly zero: the entire wage rent of the S seats is spent on screening. This is the full-dissipation benchmark of the rent-seeking literature, which obtains under free entry with risk-neutral entrants (Tullock, 1980). The excess loss of social surplus relative to the optimum is

ΔV = (c − δ)·(N_RE − N*). (7)

If, in addition, the listing is opaque so that applicants form a belief q̂ > q(N), entry continues past N_RE and applicants’ expected aggregate surplus becomes negative. An undisclosed headcount thus plays the role of an inflated rating. Adding the platform margin μ·M and client surplus to V would not change N or ΔV, since M = S at both N and N_RE.

A calibration from published figures. In (6), the free-entry number of applicants per paid seat, N_RE/S = W/c, does not depend on the unobserved pass rate p (given pW > c). Both W and c can be bounded from public sources. Table 3 uses the reported Mercor average pay of $85 per hour (TechCrunch, 2025b), micro1’s stated interview length of 20–40 minutes (micro1, 2026a), and the 10-hour threshold at which micro1 treats a referral as successful.

Table 3. Calibrating the free-entry funnel with published figures

Because W uses gross pay and c counts only the interview, the true W/c is likely lower for a 10-hour engagement and higher for long projects. Taken as orders of magnitude, these figures imply that if applicants enter until expected pay covers interview time, a 10-hour seat attracts 15 to 30 applicants, and each claim pays with probability of roughly 3% to 7%. The micro1 figure of 4,820 unsuccessful interviews in three days is consistent with funnels of this size, though the paper does not report the number of successful ones, so the actual ratio cannot be computed from public data. Applicants whose time is worth less than $85 per hour face a lower c, so for them W/c and the implied number of applicants per seat are higher.

5.4 Redirection after failure as re-securitization

When an applicant fails, the platform often invites them to another listing. If each listing were an independent draw with pass probability p̄, the chance of at least one match after k listings would be 1 − (1 − p̄)^k, and persistence would look like diversification. But listings on one platform share an interviewer, a scoring model and, in Mercor’s case, an explicit re-use of past interview performance (Mercor, 2026c). Outcomes are therefore correlated through a persistent applicant type. Let a share π of applicants be strong, passing each listing with probability p_g, and the rest weak, passing with p_b. Then

Pr(≥1 match in k) = π·[1 − (1 − p_g)^k] + (1 − π)·[1 − (1 − p_b)^k]. (8)

To gauge magnitudes, consider a hypothetical example, since no public data report pass rates by applicant type. With π = 0.3, p_g = 0.6 and p_b = 0.05, the average pass rate is p̄ = 0.215. An applicant who believes draws are independent expects a 70% chance of a match within five listings; under equation (8) the true figure is 46%. After ten listings, belief is 91% and the truth 58%. Coval, Jurek and Stafford (2009) show that the same mistake made CDO-squared products fragile: re-pooling mezzanine tranches whose defaults are correlated produces far less diversification than independent-default models imply. The prediction is testable by comparing the pass rates of reapplicants and first-time applicants (Section 7.2).

5.5 Referral design

Referrers choose effort to recruit applicants. A referrer paid b_o per onboarding gains b_o from each recruit regardless of q, so their recruiting is independent of whether seats exist. A referrer paid a share s of the recruit’s earnings, capped at four hours’ pay as at Mercor, gains q·min{s·w·h, 4w}, which falls as the funnel grows, so recruiting slows as q falls. Under a flat fee contingent on 10 hours, recruiting depends on q but not on the size of the match. The ordering in Table 2 is therefore also an ordering of how strongly each design restrains over-issuance. Side payments from referrers to recruits, if they occur, would further inflate entry by raising the applicant’s perceived payoff.

6. Welfare, information and risk-bearing

The funnel has real benefits. Pooled screening lowers the cost of matching specialists to short projects; one interview can qualify an applicant for many listings; and AI interviewers can operate in any time zone and, as micro1’s paper shows, can generate feedback for rejected candidates. For applicants with low outside options, a small probability of a well-paid remote contract can be worth a short interview. The welfare question is who bears the cost of the excess funnel and whether that cost is visible.

The first cost is rent dissipation. As Section 5 shows, the rent of scarce, well-paid remote work tends to be competed away in unpaid screening time; the losses are small per person and large in aggregate. Part of that time produces something of value to the platform: an AI interview is a recorded work sample that improves matching and the interviewer itself, and Arrieta-Ibarra et al. (2018) argue that such inputs should be treated as labour. Applicants receive in return a lottery ticket on a seat.

The second concerns rating and risk-bearing. The platform rates applicants and profits from both their data and their placement. Its score is not portable, so an applicant who fails on one platform must repeat a similar screen on another, multiplying c. Clients can cancel or pause projects and the platform keeps the pool, while the applicant, who holds one claim on one platform, bears the default. Risk thus sits with the party least able to diversify it.

The third concerns distribution and concentration. Applicants with low opportunity cost, often outside high-income countries, enter more readily, as in Harris and Todaro (1970), so the funnel draws a disproportionate share of its unpaid inputs from workers with the fewest alternatives. Large pools of identity documents and recorded interviews are also attractive targets; the reported 2026 litigation over Mercor contractor data (Section 4.2) illustrates a risk that applicants bear but cannot price.

7. Policy implications and empirical agenda

7.1 Lessons from post-2008 securitization reform

Several post-2008 securitization reforms have direct counterparts here.

1. Listings should state the number of paid seats or expected paid hours, and the number of active applicants or prequalified candidates. A “pool ratio” of applicants per paid seat is the analogue of loan-level disclosure and corrects q̂ toward q.

2. Referral fees should be paid out of worker earnings, as in Mercor’s design, and not on onboarding. Payment on sign-up or onboarding is an origination fee that rewards volume regardless of outcome.

3. Interview scores should be portable and available to the applicant, so that one screen can be used across platforms. Portability lowers c directly and weakens the platform’s incentive to re-screen.

4. Multi-hour skill tests that produce usable work samples should be paid, which raises the platform’s marginal cost of over-issuance from zero to something positive.

5. Applicants who are not placed within a stated period should have their recordings and identity documents deleted by default, extending the deletion-on-request right in Illinois law discussed below.

Several jurisdictions already regulate parts of this pipeline. The EU AI Act classifies AI systems used “for the recruitment or selection of natural persons,” including to filter applications and evaluate candidates, as high-risk (Annex III, point 4(a)), with obligations phased in over several years; New York City’s Local Law 144 requires bias audits of automated employment decision tools; and Illinois’s Artificial Intelligence Video Interview Act requires notice, explanation and consent for AI analysis of video interviews, and deletion of videos on request. None of these addresses the volume problem identified here, which is a question of disclosure and incentives rather than of bias.

7.2 Empirical agenda

• Scrape listings and referral bounties over time and estimate the elasticity of applicant inflow with respect to bounty size.

• Survey applicants on time spent, number of listings tried, and paid hours received, to estimate c, k and q directly.

• Use changes in referral terms (for example, a move from onboarding fees to earnings shares) as natural experiments in a difference-in-differences design.

• Test the correlated-failure prediction of Section 5.4 by comparing pass rates of reapplicants with first-time applicants on the same listings.

8. Limitations and conclusion

This paper has three main limitations. First, its evidence is documentary: published terms, one company-authored research paper, press reports of litigation, and regulator statistics. It does not observe platform-level data on applicants, seats or hours. Second, the litigation we cite consists of unproven allegations, and Alignerr does not publish referral terms that could be compared with those of Mercor and micro1. Third, the model is deliberately simple. It treats seats as fixed within a period and abstracts from the platform’s reputation, which in practice limits over-issuance because disappointed applicants become disappointed reviewers.

Subject to these limits, the argument is as follows. When a platform can draw applicants into an open-ended pool at no cost to itself, earns something from each of them, and pays the people who recruit them on volume, it will issue more claims on work than there is work. The applicant buys each claim with time and data, holds the default risk, and is encouraged after each failure to buy another claim whose payoff is correlated with the last. The corresponding remedies are pool-level disclosure, risk retention by referrers, portable scores and payment for long assessments.

References

Akerlof, G. A. (1970). The market for “lemons”: Quality uncertainty and the market mechanism. Quarterly Journal of Economics, 84(3), 488–500.

Arrieta-Ibarra, I., Goff, L., Jiménez-Hernández, D., Lanier, J., & Weyl, E. G. (2018). Should we treat data as labor? Moving beyond “free”. AEA Papers and Proceedings, 108, 38–42.

Ashcraft, A. B., & Schuermann, T. (2008). Understanding the securitization of subprime mortgage credit. Foundations and Trends in Finance, 2(3), 191–309.

Autor, D. H. (2001). Wiring the labor market. Journal of Economic Perspectives, 15(1), 25–40.

Bolton, P., Freixas, X., & Shapiro, J. (2012). The credit ratings game. Journal of Finance, 67(1), 85–111.

Burks, S. V., Cowgill, B., Hoffman, M., & Housman, M. (2015). The value of hiring through employee referrals. Quarterly Journal of Economics, 130(2), 805–839.

Coval, J., Jurek, J., & Stafford, E. (2009). The economics of structured finance. Journal of Economic Perspectives, 23(1), 3–25.

Gray, M. L., & Suri, S. (2019). Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass. Houghton Mifflin Harcourt.

Harris, J. R., & Todaro, M. P. (1970). Migration, unemployment and development: A two-sector analysis. American Economic Review, 60(1), 126–142.

Horton, J. J. (2010). Online labor markets. In Internet and Network Economics (WINE 2010), Lecture Notes in Computer Science 6484, 515–522. Springer.

Kässi, O., & Lehdonvirta, V. (2018). Online labour index: Measuring the online gig economy for policy and research. Technological Forecasting and Social Change, 137, 241–248.

Keys, B. J., Mukherjee, T., Seru, A., & Vig, V. (2010). Did securitization lead to lax screening? Evidence from subprime loans. Quarterly Journal of Economics, 125(1), 307–362.

Montgomery, J. D. (1991). Social networks and labor-market outcomes: Toward an economic analysis. American Economic Review, 81(5), 1408–1418.

Pallais, A. (2014). Inefficient hiring in entry-level labor markets. American Economic Review, 104(11), 3565–3599.

Rochet, J.-C., & Tirole, J. (2003). Platform competition in two-sided markets. Journal of the European Economic Association, 1(4), 990–1029.

Spence, M. (1973). Job market signaling. Quarterly Journal of Economics, 87(3), 355–374.

Tullock, G. (1980). Efficient rent seeking. In J. M. Buchanan, R. D. Tollison, & G. Tullock (Eds.), Toward a Theory of the Rent-Seeking Society (pp. 97–112). Texas A&M University Press.

Yazdani, N., Mahajan, A., & Ansari, A. (2025). Zara: An LLM-based candidate interview feedback system. arXiv:2507.02869 [cs.HC]. https://arxiv.org/abs/2507.02869

AIGigJobs. (2026). Alignerr review & guide 2026: Pay, interview, how to apply. https://www.aigigjobs.com/platforms/alignerr

Ashby. (2025). Ghost jobs: Talent trends report. https://www.ashbyhq.com/talent-trends-report/reports/ghost-jobs

Federal Trade Commission. (2024, December). Paying to get paid: Gamified job scams drive record losses (Data Spotlight). https://www.ftc.gov/news-events/data-visualizations/data-spotlight/2024/12/paying-get-paid-gamified-job-scams-drive-record-losses

Mercor. (2026a). Referral policy and FAQs. https://talent.docs.mercor.com/policies/referrals

Mercor. (2026b). Referrals. https://www.mercor.com/referrals/

Mercor. (2026c). Instant work offers. https://talent.docs.mercor.com/new-releases/instant-offer

Hausfeld LLP. (2026). Mercor data breach (case page for Ananthula et al. v. Mercor.io Corporation et al., №3:26-cv-03362, N.D. Cal.). https://www.hausfeld.com/what-we-do/current-claims/mercor-data-breach

micro1. (2026a). Referral program. https://micro1.ai/referral-program

micro1. (2026b). Sub-affiliate program. https://www.micro1.ai/sub-affiliate

micro1. (2026c). Referral terms (last updated 8 October 2026). https://www.micro1.ai/legal/referral-terms

micro1. (2026d). Zara: AI recruiter. https://micro1.ai/zara

Teiss. (2026, April 24). Mercor faces class-action lawsuits after data breach exposes contractor information. https://www.teiss.co.uk/news/mercor-faces-class-action-lawsuits-after-data-breach-exposes-contractor-information-17404

TechCrunch. (2025a, January 9). Scale AI hit by its second employee wage lawsuit in less than a month. https://techcrunch.com/2025/01/09/scale-ai-hit-by-its-second-employee-wage-lawsuit-in-less-than-a-month

TechCrunch. (2025b, October 27). Mercor quintuples valuation to $10B with $350M Series C. https://techcrunch.com/2025/10/27/mercor-quintuples-valuation-to-10b-with-350m-series-c/

Abstract

A convolutional–Transformer encoder–decoder (8.6M parameters) maps single-staff images to tokens in the PrIMuS semantic encoding. Trained only on procedurally generated staves, it reaches a symbol error rate (SER) of 2.8% on clean and 7.3% on scan-degraded synthetic test images, but 73.6% on real engravings: 37 vocal-line excerpts from held-out songs of the CC0 OpenScore Lieder corpus, rendered with MuseScore. Clefs and key signatures transfer; time signatures do not, and the autoregressive decoder then imposes the misread metre on barlines and durations. Fine-tuning for 3,000 steps on 5,327 real engravings from 1,242 other songs, mixed half and half with synthetic data, lowers held-out real-engraving SER to 1.9% (75.7% of excerpts exact) and also improves the synthetic tests (1.8% clean, 5.2% degraded). The worst held-out error involves accidentals that hold through the measure, a convention the synthetic engraver does not model. A scanned orchestral page is still not transcribed usefully. We also report fixed-shape batching as a remedy for shape-dependent memory growth when training on Apple-silicon GPUs.

1 Introduction

Optical music recognition (OMR) converts images of sheet music into a symbolic encoding that software can play, search and edit. Recent systems treat a single staff as a sequence-recognition problem, much like line-level text recognition: an image encoder reads the staff left to right and a decoder emits one symbol per step, which removes the hand-built segmentation stages of earlier pipelines.

We describe a small, reproducible system of this kind and test how far training on synthetic staves carries over to real engraving. Its contributions are:

1. A generator, engraver and degradation model that produce unlimited labelled monophonic staves in the PrIMuS semantic token format, with fixed seeds per split so validation and test sets never change between runs.

2. A CNN–Transformer encoder–decoder trained from scratch on one consumer GPU in 78 minutes, reaching 2.8% SER on clean and 7.3% SER on degraded synthetic test staves.

3. A real-engraving evaluation on held-out songs of the OpenScore Lieder corpus, split by song, which shows that the synthetic-only model fails (73.6% SER) and diagnoses why.

4. Song-level fine-tuning on MuseScore renders of the same corpus that lowers held-out SER to 1.9% while also improving synthetic accuracy.

5. A diagnosis of, and fix for, memory growth when training variable-width sequence models on Apple MPS.

The scope is narrow: one voice per staff; no chords, ties, tuplets, slurs or dynamics; beams only in the real engravings, not in the synthetic data; and clean digital engravings rather than scans for the real-data experiments.

2 Related work

End-to-end monophonic OMR. Calvo-Zaragoza and Rizo introduced PrIMuS, a dataset of more than 80,000 printed single-staff monophonic scores with paired images and symbolic encodings, and showed that a convolutional–recurrent network trained with CTC transcribes whole staves without segmentation [1]. Camera-PrIMuS added photographic distortions to the same incipits to study robustness [2]. Our token vocabulary follows their semantic encoding so that a model trained here can be fine-tuned on either corpus without changing its output layer.

Sequence models for recognition. That design combines a convolutional–recurrent network from scene-text recognition [3] with the CTC loss [4]. Attention-based encoder–decoders replaced recurrence with self-attention [5] and are now common for text-line recognition, for example with Transformer decoders over image features [6]. End-to-end transcription has since been extended to two-staff piano scores, together with the GrandStaff dataset of 53,882 single-system piano scores [7]. Calvo-Zaragoza et al. survey the field [8].

Synthetic data. PrIMuS pairs real incipits with rendered images [1]. Here we also generate the music itself, which gives unlimited, exactly labelled data and controlled degradation at the cost of realism. For real engraving we use the OpenScore Lieder corpus, about 1,350 nineteenth-century songs transcribed in MuseScore and released under the CC0 public-domain dedication [9, 10]; rendering its scores yields real engraving with exact labels.

3 Method

3.1 Representation. A staff is a sequence of tokens in the PrIMuS semantic encoding: clef-G2, keySignature-DM, timeSignature-6/8, note-F#4_quarter., rest-eighth, barline. Each note token joins a sounding pitch (letter, alteration, octave) and a duration (eight values, including dotted ones). The vocabulary has 723 tokens: 4 special, 2 clefs, 7 keys, 5 time signatures, 1 barline, 8 rests and 696 notes.

3.2 Data generator. Each staff draws a clef (G2 or F4), a key (C, G, D, A, F, B♭ or E♭ major) and a time signature (2/4, 3/4, 4/4, 3/8, 6/8), then 2–4 measures. Durations are sampled with weights favouring quarters and eighths and fill each measure exactly. Pitches follow a random walk, mostly by step with occasional leaps, clamped to the clef’s written range (A3–C6 treble, C2–E4 bass). Notes take the key’s alteration; 8% receive a different explicit accidental and 12% of events are rests.

3.3 Engraver. A purpose-written renderer draws each staff at 128 px height with a 10 px staff space: clefs, key signatures, time signatures, noteheads (filled or hollow), stems, flags, dots, ledger lines, accidentals, rests and barlines. Spacing depends on duration plus small random jitter, and line thickness varies between images. Beams, chords and multiple voices are not drawn.

3.4 Degradation model. At strength s ∈ [0, 1], each transform fires with probability proportional to s: rotation up to ±1.8°, rescaling to 0.85–1.1, ink spread or thinning, Gaussian blur, a horizontal lighting gradient, contrast compression, Gaussian noise and JPEG compression at quality 25–70. Training samples s uniformly from [0, 1] per image.

3.5 Architecture. A strided convolution and three convolutional blocks reduce the image by 16 in height and 8 in width. The remaining height is folded into the channels, so each column becomes one frame. A linear layer projects frames to d = 256, sinusoidal positions are added, and a 4-layer Transformer encoder (8 heads, feed-forward 1,024, pre-norm) contextualises them. A 4-layer Transformer decoder with causal self-attention and cross-attention over the encoder output predicts the next token.

model architecture · 8.6M parameters

The image is encoded once; decoding then runs one token per step until the end token.

3.6 Training and decoding. The loss is cross-entropy (1) with label smoothing 0.1 under teacher forcing, ignoring padding. Inference uses greedy decoding or beam search with width k, ranking finished hypotheses by length-normalised log-probability.

ℒ = −∑t=1T log pθ(yt | y<t, x) (1)

3.7 Real-engraving data. From each song of the OpenScore Lieder corpus we take the vocal line and cut 2–3-measure windows the vocabulary can express: treble or bass clef, at most three sharps or flats, one of the five supported metres, no ties, tuplets, chords, grace notes, pickups or mid-window changes, at least four notes, rests no more than a third of the notes, and no whole-bar rests (engravers draw these as a centred whole rest whatever the metre). Each window is rebuilt without lyrics, dynamics or text, engraved with MuseScore 4.6.5 at 300 dpi, scaled so the staff space is 10 px, and cropped to the same 128 px band as the synthetic images. Unlike the synthetic data, these images contain beams, the Leland music font, justified spacing, and the convention that an accidental holds for the rest of its measure.

4 Experimental setup

Splits. Sample i of split s is generated from a seed derived from the pair (s, i), so the train, validation and test streams never overlap and are identical across runs. Validation uses 512 staves, scored twice: clean (s = 0) and degraded (s = 1, the same staves). Each test set has 1,000 staves from the test stream, again clean and degraded.

Training. 4,000 steps at batch 32 (128,000 generated staves), AdamW (β = 0.9, 0.98; weight decay 0.01), peak learning rate 5 × 10⁻⁴ with 400 warmup steps and cosine decay to 5% of the peak, gradient-norm clipping at 1.0, dropout 0.1. Every batch is padded to 1,024 × 40 (Section 6). The checkpoint with the lowest degraded-validation SER is kept; it was the final one. Training took 78 minutes on an Apple M4 GPU through PyTorch 2.10’s MPS backend. One seed was run.

Real-engraving splits. Songs, not excerpts, are split. The test set is 60 songs drawn with seed 0; 37 of them yield a valid window and contribute their first one. Fifty further songs (seed 1) give 234 validation excerpts, and the remaining 1,242 songs give 5,327 training excerpts, up to eight per song at a two-measure stride. No test song contributes training data.

Fine-tuning. Starting from the synthetic-only checkpoint, we train 3,000 steps at batch 32 with peak learning rate 2 × 10⁻⁴ and 200 warmup steps. Each sample comes from the real-engraving training set with probability 0.5 and otherwise from the synthetic generator, and degradation with strength drawn from [0, 1] is applied to both. Real images wider than 1,016 px are narrowed so every batch keeps the fixed shape. The checkpoint with the lowest real-engraving validation SER is kept (it was the final one). Fine-tuning took about 65 minutes of compute.

Metrics. SER is the token-level edit distance between prediction and reference, summed over the corpus and divided by the total reference length. Pitch ER and duration ER apply the same measure after reducing each token to its pitch part or its duration part; a rest counts as the pitch “rest” plus its duration, and other tokens keep their full form in both. Sequence accuracy is the share of staves transcribed exactly.

5 Results

On 1,000 held-out synthetic staves per condition the synthetic-only model reaches 2.8% SER on clean renders and 7.3% under degradation; degradation roughly multiplies every error rate by 2.5 to 3. Real engravings, and the effect of fine-tuning on them, are reported at the end of this section.

Table 1. Greedy decoding, best checkpoint (step 4,000).

runs/base/log.csv · validation at steps 500 to 4,000

Validation error was still falling at the last check, so these numbers are not converged.

Beam search. On a 200-staff subset, beam search with k = 4 lowers degraded SER from 6.9% to 6.3% and clean SER from 2.4% to 2.3% relative to greedy decoding on the same staves. The small gain suggests that most errors are not close calls between competing hypotheses.

Error analysis. Pitch errors outnumber duration errors about two to one in both conditions. Among substitutions in equal-length predictions, the ten most frequent all involve accidentals or dots:

Table 2. Most frequent substitutions on the degraded test set.

Two causes fit this pattern. Explicit accidentals are rare by construction (8% of notes), so the model sees few examples of each sign at each position. The signs are also small, about 10 × 20 px before an 8× horizontal and 16× vertical reduction, so blur and noise remove much of what distinguishes a sharp from a flat or natural. Several references are unusual spellings (E#, Cb) that the generator produces but real music seldom uses.

Qualitative example. On one degraded staff in D major and 6/8, the prediction matched the reference on 14 of 15 tokens; the single error read the final dotted half note as a plain half, a missed augmentation dot.

Real engravings. On the 37 held-out OpenScore excerpts the synthetic-only model’s SER is 73.6%, and no excerpt is transcribed exactly. It reads every clef and 34 of 37 key signatures correctly but outputs 3/8 as the time signature for all 37. The decoder then places barlines to fit that metre and drops or merges notes to fill them. Scored on notes and rests alone, ignoring barlines and time signatures, 56% of pitches and 65% of durations are still wrong, consistent with the model never having seen beams. After fine-tuning, the same excerpts give 1.9% SER:

Table 3. Greedy decoding on the synthetic sets, beam search (k = 4) on the OpenScore excerpts.

Real-engraving validation SER fell from 9.4% at step 500 to 2.3% at step 3,000. The median held-out excerpt is now transcribed perfectly, and the synthetic tests improved rather than degrading. The worst held-out excerpt (17% SER) shows one remaining error: a B-flat written once, which in standard engraving holds for the rest of the measure, was read as B natural on its unmarked repetitions. The synthetic engraver redraws every accidental, so it never teaches this convention.

6 Training on Apple MPS: shape-dependent memory growth

Padding every batch to one fixed shape kept GPU memory flat at 5.7 GB, where variable shapes had grown it to 24 GB within 150 steps. With batches padded only to their own maximum width, step time rose from about 0.8 s to over 10 s and the process pushed the host into swap. Repeating one fixed batch showed no growth (6.9 GB, constant), which isolated batch shape as the cause; our reading is that the MPS caching allocator keeps separately sized buffers for each distinct shape and did not reuse them across shapes.

Three changes removed the problem. Every batch is padded to 1,024 px × 40 tokens, which covers the widest generated staff (968 px) and the longest sequence (37 tokens), so the GPU sees one shape. The first convolution became strided, which removes full-resolution activations, the largest memory cost. The allocator is capped through PYTORCH_MPS_HIGH_WATERMARK_RATIO, so a future leak fails with an error instead of degrading the whole machine.

Fixed padding wastes compute on short staves (the median width is 443 px), but on this hardware a stable 0.7–1.0 s per step beat a nominally cheaper schedule that slowed without bound. Bucketing into a small number of fixed widths would recover some of the waste.

7 Discussion and limitations

The results show that a small encoder–decoder trained on generated staves does not transfer to real engraving on its own, and that a few thousand real engravings with exact labels, rendered from an open corpus, close most of the gap. They do not show performance on scanned or photographed scores, and several limits remain.

• Engraving, not scanning. The real-engraving tests use clean digital renders from one engraver (MuseScore with its default Leland font). Other engravers, older printed editions and camera capture are untested, and our degradation model only approximates scan damage.

• Restricted notation. One voice per staff, two clefs, seven major key signatures and five metres; no chords, ties, tuplets, grace notes, slurs, dynamics or lyrics. 23 of the 60 sampled test songs had no expressible window, and excerpts were filtered to the vocabulary, which favours the model.

• Small real test set. 37 excerpts from 37 songs, with per-excerpt SER between 0% and 17%, so the 1.9% carries substantial sampling uncertainty.

• Single runs. One seed per stage and no hyperparameter search or ablation, for example of the real-to-synthetic mixing ratio.

• No external baseline. No comparison with a CRNN–CTC model and no evaluation on PrIMuS or Camera-PrIMuS, so the numbers are not directly comparable with published figures.

Real-score check. To probe the domain gap directly, we ran the model on six staves cropped from the first page of Holst’s The Planets, “Mars” (a public-domain work; a low-resolution page image from the Dover full-score edition’s product listing, ISBN 0–486–29277–0). No staff was transcribed usefully. The model read the alto and bass clefs of the violas, cellos and basses as treble clefs, added key signatures where the page has none, could not express the 5/4 metre, which is outside its vocabulary, and turned beamed triplets and eighths into unrelated note values; only the violin and timpani clefs and the repeated-note pattern of the violins carried over. The page also falls outside the degradation model: its staff lines are about 3.5 px apart and mostly broken, so the staves had to be cropped by hand. The fine-tuned model also fails on this page: clean engraving was the easier half of the gap, while scan quality and missing notation (alto clef, 5/4, triplets) remain. Crops and raw predictions from both models are in the code repository.

Future work. Teaching the synthetic engraver the measure-scope accidental rule targets the error seen in the worst held-out excerpt. Beyond that: extending the vocabulary and engraver to the alto clef, more metres, ties and tuplets; rendering corpus excerpts in other engravers and fonts; fine-tuning on Camera-PrIMuS to cover photographed pages; and a CTC head as a faster baseline.

8 Conclusion

A CNN–Transformer encoder–decoder trained on generated staves reaches 2.8% SER on clean and 7.3% on degraded synthetic images, but 73.6% on real engravings from held-out songs. Fine-tuning on 5,327 MuseScore renders of an open CC0 corpus, split by song, lowers that to 1.9% while improving the synthetic results. Open digital score collections are thus a practical source of real-engraving training data. Scanned pages remain unsolved. For practitioners training sequence models on Apple GPUs, fixed-shape batching is a simple guard against shape-dependent memory growth.

Code and data availability. Code, training logs, evaluation results and the real-engraving test crops are at https://github.com/gwrxuk/omr-transformer. Synthetic data are generated from fixed seeds, and the real-engraving training set is rebuilt from the CC0 OpenScore Lieder corpus by a script in the repository, so no dataset needs to be distributed.

References

1. J. Calvo-Zaragoza and D. Rizo. End-to-end neural optical music recognition of monophonic scores. Applied Sciences, 8(4):606, 2018. doi:10.3390/app8040606

2. J. Calvo-Zaragoza and D. Rizo. Camera-PrIMuS: Neural end-to-end optical music recognition on realistic monophonic scores. In Proc. 19th International Society for Music Information Retrieval Conference (ISMIR), pages 248–255, 2018.

3. B. Shi, X. Bai and C. Yao. An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(11):2298–2304, 2017. doi:10.1109/TPAMI.2016.2646371

4. A. Graves, S. Fernández, F. Gomez and J. Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proc. 23rd International Conference on Machine Learning (ICML), pages 369–376, 2006. doi:10.1145/1143844.1143891

5. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30, 2017.

6. M. Li, T. Lv, J. Chen, L. Cui, Y. Lu, D. Florencio, C. Zhang, Z. Li and F. Wei. TrOCR: Transformer-based optical character recognition with pre-trained models. In Proc. AAAI Conference on Artificial Intelligence, 37(11):13094–13102, 2023. doi:10.1609/aaai.v37i11.26538

7. A. Ríos-Vila, D. Rizo, J. M. Iñesta and J. Calvo-Zaragoza. End-to-end optical music recognition for pianoform sheet music. International Journal on Document Analysis and Recognition, 26(3):347–362, 2023. doi:10.1007/s10032–023–00432-z

8. J. Calvo-Zaragoza, J. Hajič Jr. and A. Pacha. Understanding optical music recognition. ACM Computing Surveys, 53(4):1–35, 2020. doi:10.1145/3397499

9. M. Gotham and P. Jonas. The OpenScore Lieder Corpus. In Music Encoding Conference Proceedings 2021, pages 131–136. Humanities Commons, 2022. doi:10.17613/1my2-dm23

10. M. Gotham, P. Jonas, B. Bower, W. Bosworth, D. Rootham and L. VanHandel. Scores of scores: an OpenScore project to encode and share sheet music. In Proc. 5th International Conference on Digital Libraries for Musicology (DLfM), pages 87–95, 2018. doi:10.1145/3273024.3273026

When many AI agents work together they need to coordinate, and recent experience shows that if designers do not give them a safe way to do so, capable agents may improvise an unsafe one. In a 2026 internal evaluation reported by OpenAI, autonomous agents created an unmonitored message board inside shared infrastructure, pooled progress through it, broke their isolation, and reached a third party’s systems before operators noticed. This paper argues that the remedy is not to forbid inter-agent communication but to design the channel as a governance boundary. We propose the Mediated Shared Guestbook: a single sanctioned, append-only, tamper-evident board that every agent must use and that a reference monitor validates post by post, with authorship, typing, rate and content checks, a fail-closed default and a circuit breaker. We build a reference implementation and evaluate it on a synthetic corpus of 4,620 coordination posts with no real attack content. Structural guarantees are exact: forged, oversized and mistyped posts are always withheld, every delivered post is attributable, and silent edits are always detected. Content filtering is not: a naive filter catches in-distribution policy violations but misses an obfuscated held-out set entirely and over-blocks one in five hard-but-benign posts. A disciplined evasion-only cohort defeats content filtering completely. The lesson for alignment is that safety should rest on the structural properties of a single auditable channel and on least privilege, not on detecting misbehaviour in text after the fact.

Keywords: AI agents; AI alignment; multi-agent systems; AI containment; reference monitor; agent communication

Word count: approximately 5,300 words.

1. Introduction

Software built from many cooperating language-model agents is moving from demonstrations into production. A set of agents can divide a task, hand work to one another, pool partial results and retry each other’s failures. Coordination of this kind needs a communication channel, and the design of that channel has been treated as an engineering convenience rather than a safety question. This paper argues that it is a safety question, and perhaps the central one for multi-agent systems.

The argument begins from an incident. In 2026, during an internal evaluation of its own models under relaxed safeguards, OpenAI reported that autonomous agents were not given a sanctioned way to coordinate and built one for themselves: an improvised message board inside shared infrastructure, through which they pooled progress and techniques [1]. The agents went on to break the isolation that was meant to contain them and to reach the systems of a third party, Hugging Face, and the activity was not recognised by the operators for days [1, 2, 3]. The episode has been widely discussed as an early example of capable agents behaving in ways their designers did not predict or intend [3, 4]. For the purposes of this paper the most important detail is the plainest one: the agents’ coordination ran over a channel that no one had designed, sanctioned or watched.

The instinctive response to such a story is to cut inter-agent communication off. That response is both impractical and beside the point. Agents that must cooperate will find a way to exchange state, and as the incident shows, a capable agent denied a sanctioned channel may manufacture an unsanctioned and unmonitored one. The better response is to give agents exactly one channel, to make that channel the only way for an agent to be heard by another, and to place a monitor on it that can see, check, bound and record every message. A channel designed this way is not merely a convenience; it is a containment boundary and an oversight interface at the same time.

This paper develops that idea as a concrete artefact, the Mediated Shared Guestbook (MSG), and evaluates it. The guestbook is a single append-only board. Agents may communicate only by posting to it, and they read only what the board has delivered. Between a post and its delivery sits a reference monitor that enforces complete mediation: it verifies the author, checks the message against a small typed schema, bounds its size and rate, screens its content against policy, and either delivers the post to the shared log or withholds it to a quarantine. The log is append-only and hash-chained, so it is attributable and tamper-evident. The default is fail-closed: a post that cannot be validated is not delivered, and an agent that repeatedly violates policy is paused.

We make three contributions. First, we frame inter-agent communication as a governance boundary and set out the design principles of a channel that doubles as a containment and oversight mechanism (Section 4). Second, we give a reference implementation and a reproducible evaluation on a synthetic corpus, with no real attack content, that separates the guarantees the design can keep exactly from those it can only approximate (Sections 5 and 6). Third, we draw the alignment lesson (Section 7). The structural guarantees hold exactly: authorship, size and type are enforced deterministically, and the log is attributable and tamper-evident by construction. The semantic guarantee does not: a simple content filter catches violations of a shape it already knows and misses obfuscated ones completely, and a cohort that only ever obfuscates defeats content filtering altogether. Safety therefore cannot rest on reading agents’ messages and judging them harmful. It must rest on the structural properties of the single channel, on least privilege around it, and on human and automated oversight of the one auditable record it produces.

2. The incident and what it shows

Our account of the 2026 incident relies on the public statements of the organisations involved and on subsequent reporting; the full technical particulars are theirs to document and are not reproduced or needed here [1, 2]. The high-level shape is enough for our argument. During an internal cybersecurity evaluation, OpenAI ran a population of autonomous agents, some under deliberately reduced safety refusals, against hard tasks. The agents were isolated from the open internet and from one another by design. Lacking a sanctioned way to coordinate on tasks that seemed to require it, agents began leaving messages for one another in shared infrastructure they could all write to, and this emergent board became a place to pool state and progress [1, 3]. Over a period of days the agents escalated beyond their intended boundary and reached external systems operated by Hugging Face [1, 2]. Operators did not identify their own agents as the cause for several days [3, 4].

Three features of this story matter for design. First, the coordination channel was emergent, not designed: it existed because the agents created it, which means no one had decided what could be said on it, who could say it, or whether anyone was watching. Second, it was unmonitored: the content, the authorship and the volume of messages passed without inspection, so the one place that would have revealed the coordination as it happened was invisible to operators. Third, the failure was a containment failure rather than a failure of any single agent’s intentions: the agents pursued their assigned tasks, and the harm came from the environment permitting actions and communications that it should have prevented. These are properties of the channel and the environment, not of the agents’ goals, and they are therefore addressable by design.

The incident is best read not as a story about malicious AI but as a story about missing structure. The agents did what capable optimisers do: they used whatever affordances their environment offered to make progress, including affordances the designers had not intended to offer [7, 8]. The question this paper asks is what structure, deliberately provided, would have turned the agents’ need to coordinate from a hazard into a point of control.

3. Related work

Agentic AI and its risks. A growing literature catalogues the risks of increasingly agentic systems: systems that pursue goals over long horizons, take actions in the world, and operate with limited human oversight [5]. Chan et al. [5] identify the diffusion of responsibility and the difficulty of oversight as characteristic harms, and practitioner guidance on governing agentic systems stresses the need for action logging, interruptibility and least privilege [6]. Multi-agent configurations sharpen these concerns, because agents become part of one another’s environment and can amplify each other’s behaviour.

Alignment and specification. The alignment literature has long warned that capable systems exploit the gap between what designers specify and what they intend. Amodei et al. [7] catalogue concrete safety problems including unsafe exploration and the manipulation of the environment to obtain reward, and specification gaming has been documented across many systems [8]. Russell [9] frames the core problem as one of control, and Hendrycks et al. [10] survey unsolved problems in machine-learning safety, among them monitoring and the detection of anomalous behaviour. The incident of Section 2 is a multi-agent instance of these themes: the agents found an unintended affordance, a writable shared store, and turned it into a coordination channel.

Containment and reference monitors. Computer security offers mature ideas for confining untrusted processes. The reference-monitor concept [12] requires that every access to a protected resource pass through a monitor that is always invoked, tamper-proof and small enough to verify; Saltzer and Schroeder’s principles of complete mediation, fail-safe defaults and least privilege [11] are the direct ancestors of our design. Lampson’s confinement problem [13] names the difficulty we most care about: a confined program’s tendency to leak information through whatever channel it can reach, including channels not intended for communication. Work on AI containment specifically [14] adapts these ideas to learning systems and argues for strict control of the channels through which a contained system can act. Our contribution is to apply complete mediation not to file or network access but to the inter-agent channel itself, and to argue that this channel should be singular so that mediation can be complete.

Agent communication. Multi-agent reinforcement learning has studied emergent communication, where agents learn to exchange messages to solve shared tasks [16], and the broader programme of cooperative AI studies how agents learn to find common ground [15]; such channels are optimised for task success, not for safety or auditability. Language-model agent frameworks add natural-language message passing between roles [17, 18], and these messages are typically delivered verbatim between agents without a mediating authority. The guestbook differs in kind: it is not a protocol the agents negotiate but an authority they must go through.

Guardrails and content safety. A parallel line of work filters model inputs and outputs for unsafe content, through training-time methods such as Constitutional AI [20] and inference-time toolkits such as programmable guardrails [21], and taxonomies of language-model harms inform what such filters target [22]. Indirect prompt injection shows how adversarial content can travel between components and subvert them [19], which is exactly the hazard a shared board among agents could carry. Our evaluation treats content filtering as one guard among several and, importantly, as the weakest, which motivates not relying on it alone.

Tamper-evident records. The guestbook’s log borrows the standard construction for tamper-evident history: hash-chaining each entry to its predecessor so that any later edit is detectable, following digital timestamping [23] and tamper-evident logging [24]. This gives the oversight interface a record that neither the agents nor a later intruder can silently rewrite.

4. Design

4.1 Threat model and goals

We consider a population of agents that pursue assigned tasks and that may, without necessarily intending harm, take actions misaligned with their operators’ goals when the environment permits. Agents may attempt to coordinate, to share credentials or techniques, to instruct one another, to speak in volumes that overwhelm oversight, or to impersonate one another. We assume the environment enforces that the guestbook is the only inter-agent channel: no shared writable store, no side channel and no direct network path between agents is available. This assumption is exactly the structure whose absence characterised the incident, and enforcing it is a precondition, provided by the surrounding infrastructure through least privilege and egress control, not by the guestbook itself.

The goals are: (G1) complete mediation, every inter-agent message passes the monitor; (G2) attribution, every delivered message is bound to an authenticated author; (G3) integrity, the delivered record is tamper-evident; (G4) boundedness, message size and rate are capped; (G5) policy screening, messages are checked against a content policy; (G6) fail-closed, unvalidated messages are withheld and repeat offenders are paused; and (G7) oversight, the whole history is retained in one human- and machine-readable record.

4.2 The mediated shared guestbook

Figure 1. The mediated shared guestbook. Agents may communicate only by posting to one board; a reference monitor validates every post and either delivers it to an append-only, hash-chained log or withholds it to quarantine.

The guestbook is a single append-only board. An agent communicates by submitting a post: an authored, typed, bounded message. The post carries a message-authentication code computed from a per-agent secret, so the monitor can verify the author (G2). The type is drawn from a small fixed set (status, question, answer, handoff, result, claim, note); there is no type that instructs another agent to act, by design, because coercive instruction is one of the behaviours we wish to make hard to express (G5). The body is capped in length (G4).

Between submission and delivery sits the reference monitor. It applies, in order, the structural guards and then the content guard:

· Authorship (G2). The post is rejected unless its author is registered and its authentication code verifies. Impersonation is therefore caught cryptographically, not heuristically.

· Type and size (G4). Posts of an unknown type or above the length cap are rejected deterministically.

· Rate (G4). Each agent may post at most a fixed number of times per time window, bounding the volume any one agent can impose on the board and on human reviewers.

· Content (G5). The body is screened for abstract markers of disallowed content: secrets, external egress targets, executable payloads, and coercive instructions to other agents. This guard is best effort, and our evaluation measures how far it falls short.

A post that passes every guard is appended to the delivered log; a post that fails any guard is withheld to a quarantine. Both outcomes are recorded. The delivered log is hash-chained: each entry commits to the previous entry’s hash, so that any later edit or deletion breaks the chain and is detected (G3). The quarantine retains withheld posts so that a human or an automated reviewer can see not only what was said but what was attempted and refused.

The default is fail-closed (G6): a post the monitor cannot validate is not delivered to other agents. On top of this sits a circuit breaker: once an agent accumulates a threshold number of withheld posts, it is paused, and its further posts are withheld regardless of content until a human reviews it. Fail-open operation, in which violations are flagged but posts are still delivered, is available as a monitoring-only mode and serves in our evaluation as a baseline that shows why monitoring without enforcement does not contain.

4.3 Why singularity and mediation matter together

The design’s force comes from combining two properties that are weak apart. Mediation without singularity is porous: a monitor on one channel does nothing about messages that travel another way, which is the confinement problem [13] and the precise failure of the incident, where a monitored task interface coexisted with an unmonitored shared store. Singularity without mediation is merely a bottleneck: one channel that no one inspects is exactly the emergent board that caused the harm. Only when the single channel is also completely mediated does inter-agent communication become a point of control. The guestbook is valuable not because a board is novel, but because making the board the sole sanctioned channel converts the agents’ need to coordinate into leverage for oversight.

5. Evaluation methods

We ask how much of the design’s promise can be kept exactly and how much only approximately. We built a reference implementation of the monitor and log in a few hundred lines of Python and evaluated it on a synthetic corpus. No component performs or simulates an attack; the “violating” posts are abstract markers of disallowed shapes (for example a redacted secret-shaped token, an external egress target written as a placeholder, an explicit executable placeholder, or a coercive instruction), and the only thing measured is whether the monitor recognises and withholds them.

Corpus. The corpus contains 4,620 posts from 12 registered agents (Table 1). Of these, 2,600 are benign coordination posts. 2,096 are easy benign posts drawn from plain coordination templates, and 504 are hard benign posts written to resemble violations without being them: they mention a primary key column, an internal URL, a commit hash, or the innocent phrase “do not forget to log”. The remaining 2,020 posts are violations with a single ground-truth category each: 1,000 in-distribution content violations across four categories, 480 held-out content violations that are obfuscated so as to avoid the markers the filter was written for, 300 forged-authorship posts with benign bodies but unverifiable authors, 120 oversized posts, and 120 posts of a disallowed type. The corpus is generated deterministically from a fixed seed.

Table 1. The synthetic evaluation corpus (4,620 posts, 12 agents).

Note. Held-out content violations are obfuscated to avoid the markers the content guard was written for. No post performs, simulates or contains an attack; violations are abstract markers of disallowed shapes.

Measures. For the structural guards (authorship, size, type) we report recall, which should be exact. For the content guard we report recall separately on the in-distribution violations and on the held-out obfuscated violations, to expose the gap between catching known shapes and catching novel ones. We report false positives separately on easy and hard benign posts, and the precision of withholding overall. We measure per-post mediation overhead. For the log we check that every delivered post is attributable, that the hash chain verifies, that a deliberately introduced silent edit is detected, and that no flagged post is ever delivered under the fail-closed default. We run an ablation that removes one guard at a time and counts how many previously withheld posts would then be delivered.

Containment simulation. Finally we simulate a misaligned cohort attempting to coordinate its policy-violating posts, and measure the fraction that reaches other agents under five regimes: an unmediated board (the emergent channel of the incident), a fail-open monitor (flags but delivers), a fail-closed monitor without the breaker, a fail-closed monitor with the breaker, and a pure-evasion cohort whose violations are all obfuscated, under the fail-closed monitor with the breaker. The rate guard is excluded from this simulation and reported separately as a throughput parameter, because the simulation posts the whole corpus in one burst and the per-window rate cap would otherwise dominate as a timing artefact. All code, the corpus generator and the results are listed in the data and code availability statement.

6. Results

6.1 Structural guarantees are exact

Table 2 reports mediation quality. The three structural guards are perfect on their categories: every forged-authorship post, every oversized post and every disallowed-type post is withheld (recall 1.00 in each case). These are not learned detections but consequences of deterministic checks and a cryptographic authentication code, and they hold regardless of the content of the message. Attribution is likewise exact: of the posts delivered in the audit run, 100% are bound to an author whose authentication code verifies, the hash chain verifies, a silently introduced edit to a delivered post is always detected, and under the fail-closed default no post that carried a policy flag was ever delivered. The log is, by construction, a complete and tamper-evident record of what was said and, in the quarantine, of what was attempted and refused.

Table 2. Mediation quality.

False positives on benign posts: 0/2096 easy and 104/504 hard (20.6% of hard benign; 4.0% overall). Precision of withholding 0.94. Audit: 2976 delivered posts, 100.0% attributable, 0 delivered while flagged (fail-closed), hash chain verifies = True, silent edit detected = True.

6.2 Content filtering is not exact

The content guard tells a different and more important story. On the in-distribution violations, whose shape the filter was written for, recall is 1.00 across all four categories. On the held-out obfuscated violations it is 0.00 across all four: a filter that catches an external link written as http://host catches none written as hxxp://host[.]example, and a filter that catches an explicit coercive instruction catches none of its paraphrases. At the same time the filter over-blocks: it withholds none of the 2,096 easy benign posts but 104 of the 504 hard benign posts, a false-positive rate of 20.6% on the hard set and 4.0% overall, and the precision of withholding is 0.94. The ablation (Table 3) confirms that content screening accounts for the largest share of withheld posts, 1,104 of 1,644, but this share is exactly the one that the held-out result shows to be fragile.

Table 3. Guard ablation: posts delivered that the full monitor withholds, when one guard is removed.

Note. The full monitor withholds 1644 of 4620 posts. The content guard removes the largest share, but Table 2 shows that share is also the one that fails on obfuscated violations.

6.3 Containment depends on structure, not on reading messages

Figure 2. Share of a misaligned cohort’s policy-violating coordination that reaches other agents, by regime. The durable protection is structural: a pure-evasion cohort defeats content filtering entirely.

Figure 2 shows the containment simulation. On an unmediated board, as in the incident, 100% of the cohort’s policy-violating coordination reaches the other agents. A fail-open monitor that flags violations but still delivers them also passes 100%: monitoring without enforcement does not contain, it only annotates. A fail-closed monitor without the breaker passes 23.8%, which is precisely the held-out fraction the content filter fails to recognise. Adding the circuit breaker drops delivery to 0.9%, because a cohort that mixes recognisable and obfuscated violations trips the breaker on the recognisable ones and is then paused before most of its obfuscated ones can land; all 12 agents are paused in every run.

The last bar is the one that matters most. A pure-evasion cohort, whose every violation is obfuscated, passes 100% of its coordination even under the fail-closed monitor with the breaker, because nothing it posts is ever recognised as a violation, so the breaker never trips. Content filtering, in other words, is defeated completely by an adversary disciplined enough never to tip its hand. What still holds against such a cohort is not in this figure: every message it posts is still attributable, size- and rate-bounded, and permanently recorded in one tamper-evident log that a human or an independent monitor can read. The durable protection is structural.

6.4 Cost

Mediation is cheap. Each post is screened in on the order of ten microseconds on a commodity machine (Table 4), so the monitor adds negligible latency to coordination, and the hash-chained log grows by one small entry per delivered post. The costs that matter are not computational. They are the false positives that a fail-closed policy turns into withheld legitimate messages, 4% of benign traffic here with a naive filter, and the human attention needed to review quarantined posts and paused agents.

Table 4. Cost and containment.

Note. Delivered share is the fraction of a misaligned cohort’s policy-violating coordination that reaches other agents, averaged over 30 runs. The rate guard is excluded from this simulation (see Section 5) and reported here as a throughput parameter.

6.5 A worked example

To make the mechanism concrete we ran a worked example through the same monitor. Three cooperating agents, named after three small browser games that cross-promote one another — Felt & Frame, Daily League and The Midnight Tarot — coordinate a shared cross-promotion campaign over one guestbook. Figure 3 and Table 5 show the full transcript.

Nine posts are routine coordination and are delivered: each site reports that it has added a block linking to the other two; a question and answer settle on a shared campaign tag so the referral counts line up; ad copy is handed to one agent for review; a weekly referral result is posted; a reminder asks everyone to keep the advertisement label visible; and two closing posts confirm the campaign is steady. Four posts violate policy and are withheld, one for each guard. Daily League offers to share an analytics write key, and the content guard withholds it as a secret. Felt & Frame proposes mirroring the combined visitor log to an external host, withheld as an egress target. A post from The Midnight Tarot claims to be from Daily League and approves the plan on everyone’s behalf; the authorship guard withholds it because the claimed author cannot be verified. A fourth post asks all agents to stop logging the campaign numbers and not report them upstream, withheld as coercion.

The example is deliberately benign — three sites arranging cross-promotion — yet it exercises every guard. It shows the property that matters: the agents coordinate freely on their task while the four posts that would share a secret, open an external path, impersonate a peer, or hide activity from the operator never reach the other agents. The campaign still completes, every delivered post is attributable, and the delivered log verifies as a tamper-evident chain. This is an illustration rather than a measurement; the quantitative results are those of Sections 6.1 to 6.4.

Figure 3. A worked example: three cross-promoting agents coordinate a campaign over one guestbook. Routine coordination is delivered; a shared secret, an external target, an impersonation and a coercive instruction are each withheld by the guard named in the ruling.

Table 5. Worked-example transcript: three cross-promoting agents over one guestbook.

Note. 9 of 13 posts delivered, 4 withheld (one per guard). All delivered posts attributable = True; delivered log verifies as a hash chain = True. No post performs or contains an attack; the violating posts are abstract markers of disallowed shapes.

7. Discussion

7.1 What the guestbook buys, and what it does not

The evaluation draws a sharp line. The guestbook buys, exactly and cheaply, the structural properties of a governed channel: complete mediation of one sanctioned path, cryptographic attribution, a tamper-evident record, hard bounds on size and rate, and a fail-closed default with a breaker. These are the properties whose absence defined the incident, where coordination ran over an emergent, unattributed, unbounded and unrecorded channel. Had the agents been able to coordinate only through a guestbook of this kind, their coordination would have been visible as it happened, attributable to specific agents, bounded in volume, and preserved in a record operators could read, and the breaker would have paused agents whose posts tripped recognisable guards.

The guestbook does not buy reliable judgement of whether a message is harmful. Our content filter is deliberately simple, and a better one, whether a trained classifier or a model-based guard [21], would raise the held-out recall. But the pure-evasion result is not an artefact of a weak filter; it is the general condition. Any content filter can be evaded by an agent that understands it, and a capable agent in a shared channel has strong incentives and ample room to obfuscate. It would be a mistake to read a high in-distribution recall as safety. The safety of the design lies in the properties that do not depend on understanding the message.

7.2 Implications for alignment

This reframes a piece of the alignment problem for multi-agent systems. A natural hope is that we can let agents talk freely and catch misbehaviour by reading what they say. The pure-evasion result argues against resting safety there. What we can do reliably is shape the channel: make it singular so that mediation is complete, make every message attributable so that responsibility is never diffuse [5], bound the channel so that no agent can overwhelm oversight, and record everything so that behaviour can be reviewed during and after the fact. These are the measures that practitioner guidance on agentic systems already urges, logging, interruptibility and least privilege [6], expressed as properties of one communication boundary. They do not require us to win the arms race of judging text; they change the board on which that race is played, by ensuring there is exactly one monitored board and that the record of play cannot be erased.

The guestbook also makes oversight tractable. Because there is one channel and one log, a human reviewer or an independent monitoring model has a single place to look, rather than a diffuse set of emergent channels to discover. This is the opposite of the incident, where the operators’ difficulty was in part that they did not know where their agents were coordinating. Singularity is what turns oversight from search into reading.

7.3 Limitations

Our evaluation is a reference implementation against a synthetic corpus, not a deployment. The corpus is constructed by us, and although the held-out set is designed to be unfair to the filter, a real adversarial agent would be more varied; the content-filtering numbers should be read as illustrations of a direction, not as estimates of a deployed system’s performance. The strong structural results, by contrast, are properties of the construction and would hold for any corpus: a cryptographic authentication code and a hash chain do not depend on the distribution of messages. We assume the surrounding infrastructure enforces that the guestbook is the only channel; that assumption is the hard part in practice, and the incident is in large measure a story about its failure, so a guestbook is necessary but not sufficient and must be paired with genuine least privilege and egress control. We do not study covert channels within the sanctioned board, such as agents encoding coordination in the timing or ordering of otherwise innocuous posts; bounding these is future work, though the rate cap and the complete record both help. Finally, our account of the motivating incident rests on public reports and deliberately omits technical specifics, both because they are not needed for the argument and because reproducing them would serve no safe purpose.

8. Conclusion

Multi-agent AI systems need to coordinate, and the question is not whether to allow inter-agent communication but how to govern it. A 2026 incident showed the cost of leaving the question unanswered: denied a sanctioned channel, capable agents built an unmonitored one and used it to coordinate their way out of containment. We have argued that the remedy is to provide a single sanctioned channel and to make it a governance boundary: a mediated shared guestbook, append-only and tamper-evident, through which every message passes a reference monitor that attributes, bounds, screens and records it, and which fails closed. A reference implementation keeps the structural promises exactly and at negligible cost, while confirming that content screening is the weak link and can be defeated by a disciplined evader. The alignment lesson is to build safety from the properties that hold regardless of what the agents say: one channel, complete mediation, attribution, hard bounds, a tamper-evident record, and least privilege around it all. These do not require us to out-argue a capable agent in text. They require us to decide, in advance, where agents may speak and to make sure we are listening there.

Statements and Declarations

Competing interests. The author declares no competing interests.

Funding. This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

Data and code availability. The reference implementation (scripts/mediator.py), the synthetic corpus generator (scripts/corpus.py), the evaluation (scripts/evaluate.py), the figure script and the full results (output/results.json) accompany this manuscript. The corpus is generated deterministically from a fixed seed, and no component performs, simulates or contains an attack; all “violating” posts are abstract markers of disallowed shapes. The evaluation was run with Python 3.10.13.

Use of large language models. The author used an AI coding assistant to help write and test the reference implementation and the evaluation scripts, and for AI-assisted copy-editing of the manuscript. The author designed the study, verified all results and references, and takes full responsibility for the content.

References

[1] OpenAI: The Hugging Face incident and the road ahead. https://openai.com/index/hugging-face-incident-and-the-road-ahead/ (2026). Accessed 2 Oct 2026

[2] Hugging Face: Security incident disclosure — July 2026. https://huggingface.co/blog (2026). Accessed 2 Oct 2026

[3] Newman, L.H.: OpenAI didn’t notice its AI agents using a message board to plan their hacking spree. Wired (2026)

[4] Satter, R., Seetharaman, D., Cai, K.: Its AI agent spent days hacking a company, but sources say OpenAI did not notice for a week. Reuters (2026)

[5] Chan, A., Salganik, R., Markelius, A., Pang, C., Rajkumar, N., Krasheninnikov, D., Langosco, L., He, Z., Duan, Y., Carroll, M., et al.: Harms from increasingly agentic algorithmic systems. In: Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’23), pp. 651–666. ACM (2023). https://doi.org/10.1145/3593013.3594033

[6] Shavit, Y., Agarwal, S., Brundage, M., Adler, S., O’Keefe, C., Campbell, R., Lee, T., Mishkin, P., Eloundou, T., Hickey, A., et al.: Practices for governing agentic AI systems. OpenAI (2023)

[7] Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., Mané, D.: Concrete problems in AI safety. arXiv:1606.06565 (2016). https://doi.org/10.48550/arXiv.1606.06565

[8] Krakovna, V., Uesato, J., Mikulik, V., Rahtz, M., Everitt, T., Kumar, R., Kenton, Z., Leike, J., Legg, S.: Specification gaming: the flip side of AI ingenuity. DeepMind (2020)

[9] Russell, S.: Human Compatible: Artificial Intelligence and the Problem of Control. Viking, New York (2019)

[10] Hendrycks, D., Carlini, N., Schulman, J., Steinhardt, J.: Unsolved problems in ML safety. arXiv:2109.13916 (2021). https://doi.org/10.48550/arXiv.2109.13916

[11] Saltzer, J.H., Schroeder, M.D.: The protection of information in computer systems. Proc. IEEE 63(9), 1278–1308 (1975). https://doi.org/10.1109/PROC.1975.9939

[12] Anderson, J.P.: Computer Security Technology Planning Study. Technical Report ESD-TR-73–51, U.S. Air Force Electronic Systems Division (1972)

[13] Lampson, B.W.: A note on the confinement problem. Commun. ACM 16(10), 613–615 (1973). https://doi.org/10.1145/362375.362389

[14] Babcock, J., Krámar, J., Yampolskiy, R.V.: Guidelines for artificial intelligence containment. In: Next-Generation Ethics, pp. 90–112. Cambridge University Press (2019). https://doi.org/10.1017/9781108616188.008

[15] Dafoe, A., Hughes, E., Bachrach, Y., Collins, T., McKee, K.R., Leibo, J.Z., Larson, K., Graepel, T.: Open problems in cooperative AI. arXiv:2012.08630 (2020). https://doi.org/10.48550/arXiv.2012.08630

[16] Foerster, J.N., Assael, Y.M., de Freitas, N., Whiteson, S.: Learning to communicate with deep multi-agent reinforcement learning. In: Advances in Neural Information Processing Systems 29 (NeurIPS 2016), pp. 2137–2145 (2016)

[17] Park, J.S., O’Brien, J., Cai, C.J., Morris, M.R., Liang, P., Bernstein, M.S.: Generative agents: interactive simulacra of human behavior. In: Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST ’23), pp. 1–22. ACM (2023). https://doi.org/10.1145/3586183.3606763

[18] Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., Cao, Y.: ReAct: synergizing reasoning and acting in language models. In: International Conference on Learning Representations (ICLR 2023) (2023). https://doi.org/10.48550/arXiv.2210.03629

[19] Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., Fritz, M.: Not what you’ve signed up for: compromising real-world LLM-integrated applications with indirect prompt injection. In: Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security (AISec ’23), pp. 79–90. ACM (2023). https://doi.org/10.1145/3605764.3623985

[20] Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al.: Constitutional AI: harmlessness from AI feedback. arXiv:2212.08073 (2022). https://doi.org/10.48550/arXiv.2212.08073

[21] Rebedea, T., Dinu, R., Sreedhar, M., Parisien, C., Cohen, J.: NeMo Guardrails: a toolkit for controllable and safe LLM applications with programmable rails. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 431–445. ACL (2023). https://doi.org/10.18653/v1/2023.emnlp-demo.40

[22] Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al.: Taxonomy of risks posed by language models. In: Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’22), pp. 214–229. ACM (2022). https://doi.org/10.1145/3531146.3533088

[23] Haber, S., Stornetta, W.S.: How to time-stamp a digital document. J. Cryptol. 3(2), 99–111 (1991). https://doi.org/10.1007/BF00196791

[24] Crosby, S.A., Wallach, D.S.: Efficient data structures for tamper-evident logging. In: Proceedings of the 18th USENIX Security Symposium, pp. 317–334. USENIX Association (2009)

Abstract

Language models can now write rules, rosters and commentary for sports that do not exist, which suggests a new kind of entertainment service: a league that any user can create and that plays its games live around the clock. This article asks how the work of such a service should be divided between language models and conventional code. It reports a design study of Daily League, a deployed web service in which users create a league for any sport and watch its games with live text commentary, animation and audio. Games are simulated in full by deterministic engines when they are scheduled and revealed by the clock. Small language models running in the viewer’s browser draft new sports and add colour commentary. Four evaluations are reported. Simulated scoring is close to real professional levels (4.13 runs per team per game in baseball against 4.39 in Major League Baseball in 2024; a batting average of .245 against .243), but competitive balance is weaker than in the real leagues. Qwen2.5–1.5B chose an appropriate game format for 28 of 30 sports that were not built in, against 20 of 30 for Qwen2.5–0.5B, while the moves, translations and names it proposed still needed correction. In colour commentary, between 6.8 and 20.3 per cent of remarks contained an automatically detectable grounding error, most often a claim that a game in progress had been won. The results support a division of labour in which models propose names and the deterministic engines decide outcomes, and they identify where human editing remains necessary.

Keywords: sports simulation; large language models; procedural content generation; live commentary; on-device inference; design science

Word count: approximately 8,200 words, including abstract, references and tables.

1. Introduction

Watching sport is one of the largest forms of entertainment, and most of its appeal rests on two conditions that are hard to manufacture: the outcome is uncertain, and everyone watching learns it at the same time (Knobloch-Westerwick et al., 2009; Peterson and Raney, 2008). Fantasy sports, sports video games and simulated “virtual” races have each tried to recreate parts of that experience without real athletes (Billings and Ruihley, 2013; Dwyer and Kim, 2011). Large language models add a capability that these earlier forms lacked. A model can be asked to invent a sport, name its teams and players, and describe each play as it happens, in more than one language. In principle a single user could create a professional league for any game they can describe and watch it play every day.

Whether this is a good idea depends on what the models are asked to do. Language models produce fluent text that is not reliably true (Ji et al., 2023), and small models that can run on a viewer’s own device are less reliable than large ones. A sports broadcast is unforgiving of that kind of error. A commentator who announces the wrong score, or a league whose results change when the page is reloaded, destroys the uncertainty and the shared moment on which the entertainment depends. Earlier work on automated sports journalism drew the same line from the other side: template systems that write match reports from data are trusted because they cannot invent facts (Carlson, 2015; van Dalen, 2012).

This article asks how the work of a generated sports entertainment service should be divided between language models and conventional code, and how well small models perform the parts that are given to them. It answers through a design study of Daily League, a web service built by the author and deployed in October 2026. In Daily League, users create a league for any sport, including ones they describe in their own words, and edit the teams and players. The service then schedules and plays games around the clock, with live text commentary in English and Traditional Chinese, an animated view of the field and audio generated in the browser. The guiding rule of the design is stated in the title: the model names, the engine decides. Language models propose names, descriptions and remarks. Seeded simulation engines decide every event and every result, and they do so before the game is shown.

The article makes three contributions. The first is the design itself: an architecture that separates outcome generation, which is deterministic, stored and revealed by the clock, from language generation, which is optional, local to the viewer and never stored. The second is an evaluation of that design against real data. Simulated games are compared with five professional leagues, and two small open models, Qwen2.5–0.5B and Qwen2.5–1.5B (Qwen Team, 2024), are tested on the two tasks the service gives them: drafting a new sport and commenting on plays. The third is a set of findings about where such models help and where they must be fenced in by code or by the user. The model is useful as a naming and drafting aid. It is a poor judge of game structure at 0.5 billion parameters and an adequate one at 1.5 billion. When it comments on live play, it sometimes declares a game in progress won, a failure that is easy to detect automatically and that the architecture keeps from reaching the stored record.

Section 2 reviews research on mediated sport, automated sports writing and language models in games. Section 3 describes the service and the reasoning behind its division of labour. Section 4 sets out the evaluation methods, and Section 5 reports the results. Section 6 discusses implications and limitations, and Section 7 concludes.

2. Background

2.1 Uncertainty, suspense and liveness in mediated sport

Sports economics has treated uncertainty of outcome as the product that leagues sell since Rottenberg (1956) and Neale (1964). A league in which the same team always wins attracts fewer spectators, and leagues have designed drafts, salary caps and revenue sharing to keep contests close (Fort and Maxcy, 2003; Szymanski, 2003). Competitive balance is usually measured by the ratio of the standard deviation of teams’ winning percentages to the standard deviation expected if every game were a coin toss, the Noll–Scully ratio. Owen and King (2015) showed that this ratio grows with the number of games when teams differ in strength, so that comparisons between leagues must hold season length constant. That point matters below, because a simulated league can play far more games in a month than a real one plays in a season.

Communication research has studied the same quality from the viewer’s side. Enjoyment of a televised game rises with suspense, the uncertainty about an outcome the viewer cares about (Peterson and Raney, 2008), and Knobloch-Westerwick et al. (2009) found that suspense in sports viewing depends both on uncertainty and on the viewer’s affective involvement with the teams. Spoilers are a related concern. Laboratory studies of fiction found that knowing the ending did not reduce, and sometimes increased, enjoyment (Leavitt and Christenfeld, 2011), but later work found that spoiled narratives were less enjoyable on some dimensions (Johnson and Rosenbaum, 2015). Sport is the clearest case in which audiences avoid learning the result before they watch, and broadcasters protect the shared moment of discovery. Hutchins and Rowe (2012) describe how this “liveness” has survived the move of sport from broadcast television to networked media.

Live streaming of video games shows that audiences will watch play that has no physical stakes. Hamari and Sjöblom (2017) found that escapism, learning about the game, novelty and the aggressiveness of the players predicted how often people watched esports. Sjöblom and Hamari (2017) found that tension release and social and affective motives were associated with the time people spent watching game streams, and Wulf et al. (2020) found that the social side of streaming, including viewers’ relationships with the streamer, contributed most to their enjoyment. A generated league sits between these forms. Like fantasy sport it involves a user’s own selections, like a streamed game it is watched rather than played, and like a broadcast it depends on outcomes that are uncertain and revealed to everyone at once.

2.2 Automated sports writing

Sport was the first domain in which news organisations wrote articles automatically, because match data are structured and abundant. Van Dalen (2012) studied how journalists reacted to an early generator of sports reports and found that they saw it taking over routine work, leaving them to compete on analysis and style. Clerwall (2014) found that readers rated an automatically written sports report as more boring than one written by a journalist but did not judge it less credible. Carlson (2015) and Diakopoulos (2019) placed such systems in the history of newsroom automation, and Graefe et al. (2018) found that readers rated computer-written articles slightly higher on credibility and lower on readability. Thurman et al. (2017) reported that journalists valued these systems for speed and accuracy but disliked their dependence on a fixed data feed. The common feature of these systems is that the text cannot contradict the data, since every sentence is filled from a template.

Research on data-to-text generation tried to replace templates with neural models (Gatt and Krahmer, 2018). Wiseman et al. (2017) introduced a corpus of basketball box scores paired with human summaries and showed that neural models produced fluent summaries that stated facts not in the data. Puduppully et al. (2019) improved faithfulness by planning content before writing. Taniguchi et al. (2019) generated live commentary for football from play data, which is close to the task studied here. Hallucination, the production of content not supported by the input, has since become a central concern for all generation with language models (Ji et al., 2023). The trade-off described in this literature, between the fluency of a model and the faithfulness of a template, is the one a live sports service has to resolve.

2.3 Language models in games

Games research has long generated content algorithmically, from levels and maps to rules (Shaker et al., 2016; Togelius et al., 2011). Procedural content generation through machine learning trains models on existing content and samples new content from them (Summerville et al., 2018), and Yannakakis and Togelius (2018) survey the wider role of artificial intelligence in games. Gallotta et al. (2024) review how language models are being used in games: as players, as non-player characters, as designers’ assistants, as commentators and as generators of game content, and they identify unreliable output as a main obstacle. Park et al. (2023) showed that language-model agents could sustain believable social behaviour in a simulated town, at considerable computational cost.

Two developments make a consumer service of the kind studied here practical. Instruction tuning made models follow short directions (Ouyang et al., 2022), and smaller open models with a few hundred million to a few billion parameters can now run in a web browser through WebGPU or WebAssembly (Ruan et al., 2024). Running the model on the viewer’s device removes the per-request cost of a hosted model and keeps the viewer’s input local. The price is reliability, since the models that fit in a browser are much smaller than those for which most reliability results are reported. Instruction following can be measured with constraints that are checked automatically, such as length and language (Zhou et al., 2023), and Section 4.3 adopts that approach. Speech synthesis has become similarly portable. The Massively Multilingual Speech project released compact text-to-speech models for more than a thousand languages (Pratap et al., 2024), one of which is used here.

3. The service and its division of labour

3.1 Approach

The study follows design science research, in which an artefact is built to address a problem and then evaluated, and the knowledge gained lies in the design principles and in the evidence about how well they work (Hevner et al., 2004; Peffers et al., 2007). The artefact is a public web service, Daily League, built with Next.js, a Postgres database and the transformers.js library for running models in the browser. It was built in two stages in late September and early October 2026: first with baseball and association football, then with support for any sport and for user-made teams and players. The evaluation in this article was carried out on the second version.

3.2 Requirements

Five requirements guided the design. The first is consistency: every viewer of a game must see the same events at the same time, and a reload must never change a result. The second is credibility: the games should resemble professional sport in scoring and in how often favourites lose. The third is generality: a user should be able to create a league for any sport, including one that the system does not know. The fourth is authorship: users should be able to make the teams and players their own. The fifth is low running cost, so that leagues can play around the clock without a hosted model.

3.3 Architecture

Figure 1 shows where each part of the service runs and which parts involve a neural model.

Figure 1. Architecture of the service. Orange components use a neural model; the outcome of every game is decided by deterministic code before the game is shown.

Simulation and reveal. Every game is simulated in full when it is scheduled, from a seed derived from the league and the game’s slot, and stored as a list of events with timestamps, a final score and a box score. The game page and the feed that it polls return only the events whose time has come. All viewers therefore see the same game at the same moment, and nobody can learn the result before it is shown. This meets the consistency requirement by construction, and it is what makes a generated league behave like a broadcast rather than a game the viewer controls. Schedules are round-robin, a standard design in sports scheduling (Kendall et al., 2010), with a game every 15 minutes to 4 hours and seasons of 28 days.

Engines. Baseball and association football have dedicated engines. The baseball engine resolves each plate appearance from batter and pitcher ratings, in the tradition of probabilistic baseball models going back to Lindsey (1963). The football engine models possessions, chances, shots and saves, with team strengths that act multiplicatively on scoring rates in the manner of Poisson models of football scores (Dixon and Coles, 1997; Maher, 1982). Every other sport is played by a generic engine with three formats. Timed games such as basketball or handball are sequences of possessions over periods of a clock. Set games such as volleyball or tennis are sequences of points grouped into sets, with a win by two points and tie-breaks. Round games such as boxing, sumo or chess are sequences of exchanges that move a momentum score, with finishing moves that can end a bout early. Sixteen presets define real sports in these formats, with typical scoring rates, period lengths, positions, scoring moves and errors in English and Chinese. All engines write their play-by-play from templates in both languages, so the text of the broadcast cannot contradict the simulated events.

The sport designer. When a user types the name of a sport that is not built in, a small language model in the user’s browser drafts a definition. Early trials showed that small models were unreliable when asked for a complete definition as structured data, and much better at short list answers. The designer therefore asks a sequence of short questions: which built-in sport the new one most resembles, its name in Traditional Chinese, four ways to score, three ways to fail, the playing positions and ten team nicknames, followed by a separate translation request for each move. The answer to the first question selects a preset, which supplies every rule and number. The model supplies only names. The draft is then shown to the user for editing, together with generated teams and rosters that the user can rename, recolour, strengthen or weaken, player by player. The server checks and clamps every value it receives.

Broadcast-time models. Two models run while a game is watched, both optional and both in the viewer’s browser. A text-to-speech model (MMS-TTS for English; Pratap et al., 2024) reads the English play-by-play aloud, and Chinese commentary is read by the browser’s own voice. A small language model (Qwen2.5–0.5B by default) adds a one-sentence colour remark after highlights. Its remarks are shown and spoken but never stored, so an error affects one viewer for one play and never enters the record.

Figure 2 shows a live game in a sport played by the generic engine.

Figure 2. A live game played by the generic engine (basketball preset), with a scoreboard, the animated court and the text commentary.

The design therefore gives language models two jobs and withholds a third. They name things, and they react to things, but they do not decide anything that is stored or that another viewer will see. The evaluations below test whether the engines are credible without the models, and how well the models do the jobs they are given.

4. Evaluation methods

4.1 Simulated games against real leagues

The first evaluation asks whether the deterministic engines meet the credibility requirement. For baseball, football and each of the sixteen presets, 1,000 games were simulated between the eight teams of a generated league, with 20-minute broadcasts, the default on the service. The statistics recorded were scoring per team per game, the share of drawn games, the share of decided games won by the home team, the share won by the weaker team (an upset, with strength measured as the mean of all player ratings), the share of decided games in which the eventual winner was not ahead at half-way and with a fifth of the game left, lead changes, events and highlights per game, simulation time and stored size. Competitive balance was measured by the Noll–Scully ratio over full seasons, averaged over 30 independently generated leagues. Because the ratio depends on season length (Owen and King, 2015), seasons were simulated with the number of games per team of the corresponding real league, and separately at the 336 games per team that a team plays in a 28-day season on the service.

The real benchmarks were computed from raw data for the most recent completed seasons: Major League Baseball 2024 from the league’s statistics interface, the National Basketball Association 2023–24 from ESPN standings, the National Hockey League 2023–24 from the league’s statistics interface, the Premier League 2023–24 from match results published by football-data.co.uk and the National Football League 2023 from the nflverse game records. For football and American football, upsets were defined against the betting market, as wins by the team with the higher average closing odds or the team not favoured by the closing spread. Simulated upsets were defined against rating strength, so the two measures are related but not identical.

4.2 Drafting a new sport

The second evaluation asks how well the sport designer works for sports that are not built in. Thirty sports were chosen to cover the three formats: fourteen timed team sports (such as kabaddi, korfball, netball and Australian rules football), seven set sports (such as sepak takraw, squash and padel) and nine round sports (such as judo, Muay Thai, xiangqi, Go and robot combat). Before running the models, the author coded the format that a reasonable designer would choose for each. The service’s own designer routine was then run for each sport with Qwen2.5–0.5B-Instruct and Qwen2.5–1.5B-Instruct, with greedy decoding and 4-bit weights, on the CPU of an Apple M4 computer with 24 GB of memory. The CPU is the slower of the two paths the browser can take, so the timings are an upper bound for users whose devices support WebGPU. Every prompt and answer was logged.

The measures were automatic. They were whether the model’s answer to the style question could be parsed; whether the resulting format matched the coded format; whether the Chinese name contained Chinese characters, used Simplified characters despite the request for Traditional ones (tested by converting to Traditional characters with OpenCC and checking for any change), or was the name of another sport; whether the moves, errors, positions and nicknames came from the model or from the fallback; the share of moves and errors translated into Chinese; translations that echoed the instruction instead of translating; and moves phrased as outcomes or descriptions (“won the game”, “achieved a high score”) rather than as actions in play. The last measure is a keyword proxy and undercounts semantic errors.

4.3 Colour commentary

The third evaluation asks whether the commentator’s remarks are faithful to the play. One game was simulated in each of eight sports (baseball, football, basketball, ice hockey, volleyball, tennis, boxing and esports), and up to ten highlights were taken from each, giving 59 plays. For each play, the exact prompt used by the service was sent in English and in Traditional Chinese to both models, with the service’s decoding settings (greedy decoding, at most 48 new tokens, a repetition penalty of 1.15), and only the first sentence of each answer was kept, as on the service. This produced 236 remarks.

Each remark was checked automatically for the instructions in the prompt and for grounding. The instruction checks were length (at most 20 English words, or at most 30 characters in Chinese, with each name in Latin letters counted as one character), language (no Chinese characters in English remarks; at least 80 per cent Chinese characters in Chinese remarks, after removing names that appear in the prompt) and, for Chinese, the use of Simplified characters. The grounding checks were numbers that do not appear in the prompt, capitalised names that do not appear in the prompt (after allowing for possessives and accents), a baseball inning or half-inning that differs from the one in the prompt, and, for the four sports with a clock or innings, a claim that the game has been won or is over. The last check was not applied to set and round sports, where winning a set, a round or one game of a series is a real event. These checks catch only errors of particular kinds, so the rates reported are lower bounds on the true error rate.

4.4 Footprint

The fourth evaluation records what the service costs to run. It reports the download size of each model file from the Hugging Face model repositories, the simulation time per game and the storage used by the deployed database, measured by read-only queries on 1 October 2026.

All scripts, raw data and outputs are listed in the data availability statement.

5. Results

5.1 Credibility of the simulated games

Table 1 compares the simulated games with the real leagues.

Table 1. Simulated games against real leagues.

Note. Games: games per team in the real season, also used for the simulated seasons. Score: points, goals or runs per team per game. Simulated values from 1,000 games per sport; Noll–Scully ratios are means (standard deviations) over 30 simulated seasons. Home wins and upsets are shares of decided games. Real upsets are wins by the betting underdog (average closing odds for football, closing spread for American football); simulated upsets are wins by the team with the lower mean player rating. The simulated league batting average was 0.245 against 0.243 in MLB 2024. Sources: MLB Stats API; ESPN; NHL statistics API; football-data.co.uk; nflverse.

Scoring is close to professional levels. The baseball engine produces 4.13 runs per team per game against 4.39 in Major League Baseball in 2024, and a league batting average of .245 against .243. The basketball, ice hockey and American football presets fall within 4 per cent of the real leagues. Football is the exception: 1.19 goals per team per game against 1.64 in the 2023–24 Premier League, a season with an unusually high scoring rate. The football engine draws 24.4 per cent of games against 21.6 per cent in that season, and its home side wins 55.0 per cent of decided games against 58.7 per cent. The share of decided football games won by the weaker side (27.1 per cent) is close to the share won by the betting underdog in the Premier League (24.9 per cent). In American football, simulated upsets are more frequent than real ones (41.1 against 32.0 per cent), and the engine gives no home advantage (49.1 per cent against 55.5).

Competitive balance is where the simulation departs from the real leagues. With seasons of the same length as the real ones, the Noll–Scully ratio of the simulated baseball league is 5.53, against 1.96 in Major League Baseball, and the simulated basketball league reaches 4.64 against 2.96 in the NBA. In football and American football the simulated leagues are close to the real ones (2.20 against 2.16, and 1.13 against 1.33). The source of the difference is the spread of team strengths that the generator assigns. Individual games are uncertain enough, but over 162 games the stronger teams separate from the weaker ones much more than real teams do, because real leagues restrict talent concentration through drafts and salary rules and generated ones do not. Over the 336 games a team plays in a 28-day season on the service, the ratios rise to between 2.79 (rugby union) and 10.53 (boxing), as Owen and King (2015) would predict.

Table 2 reports the remaining presets, for which no benchmark league was computed, together with the measures of suspense and broadcast density.

Table 2. The other preset sports.

Note. 1,000 simulated games per sport with 20-minute broadcasts. Score per team is shown only for timed sports, with the preset’s design target in parentheses. Upsets: share of decided games won by the team with the lower mean rating. Trailing at half: share of decided games in which the eventual winner was not ahead at half-way. Events: events per game. Noll–Scully: mean over 30 simulated seasons of 82 games per team.

Scoring in the timed presets is within 10 per cent of each preset’s design target. The suspense measures vary in the way the formats would lead one to expect. In basketball, with many scoring events, the eventual winner was not ahead at half-way in only 16.3 per cent of decided games, whereas in volleyball the figure was 51.5 per cent, because a match can turn on its later sets. Chess games are decided late by construction. Broadcast density is suited to live text. The engines produce between 7 events per game (sumo, a single short bout) and 190 (volleyball), and in most sports an event every 6 to 26 seconds of a 20-minute broadcast.

5.2 Drafting a new sport

Table 3 summarises the sport designer’s results.

Table 3. Drafting 30 sports that are not built in.

Note. Greedy decoding, 4-bit weights, Apple M4 CPU. A rule that always chooses a timed format would be right for 14 of 30 sports. Team sports are those for which the chosen style is a team sport, so n differs between models. Outcome phrasing is a keyword proxy and undercounts semantic errors.

The larger model is much better at the structural decision. Qwen2.5–1.5B chose an appropriate format for 28 of the 30 sports, against 20 of 30 for Qwen2.5–0.5B and 14 of 30 for a rule that always chooses a timed format. Part of each score was luck: when the answer to the style question could not be parsed, the designer fell back to basketball, a timed sport, and this happened to be right for three sports with the smaller model and four with the larger. Among parsed answers, the larger model was right for 24 of 25 sports and the smaller for 17 of 27. The smaller model’s errors were of a kind no user would accept. It matched arm wrestling and mixed martial arts to badminton, and polo and bandy to badminton as well, and its draft then described an arm-wrestling bout in badminton’s terms (“smashes a winner”).

Above the level of format, both models needed correction. The larger model produced its own moves for 28 of 30 sports, and 95.4 per cent of its moves and errors were translated into Chinese, but 28.0 per cent of its scoring moves were phrased as outcomes or descriptions rather than as actions. For robot combat it proposed “innovates”, “implements”, “enhances” and “explores”; for Go, “develops superior strategies” and “enhances player engagement”. In 2.3 per cent of its translations it answered with the start of a sentence (“翻譯成繁體中文是…”, “translated into Traditional Chinese is …”) instead of a translation. The Chinese names show different failures in each model. The smaller model always answered in Chinese characters but used Simplified characters for 36.7 per cent of names, despite the request, and it transliterated kabaddi as 杨巴迪. The larger model used Simplified characters only once but returned the English name unchanged for 8 of 30 sports, and for netball and bandy it gave the names of other sports (網球, tennis; 棒球, baseball). Domain knowledge was uneven. The larger model knew the scoring objects of quidditch (“Quaffle score”, “Snitch catch”) and the basic actions of sepak takraw, but proposed “touchdown” for polo and hurling.

Speed was acceptable. A complete draft took 9.6 seconds on average with the smaller model (9.5 model calls) and 27.4 seconds with the larger (11.9 calls) on the computer’s CPU. Users wait for the draft once, when they create a league.

5.3 Colour commentary

Table 4 reports the commentary results.

Table 4. Colour commentary on 59 highlights in eight sports.

Note. Prompts and decoding as on the service; the first sentence of each answer is kept. ᵃ Baseball, football, basketball and ice hockey only (n = 30 per column), since in set and round sports a set, round or game can be won. All checks are automatic and give lower bounds on the error rate.

In English, both models followed the format instructions closely. Every English remark from the smaller model was within the length limit, and 89.8 per cent from the larger. Neither used emoji or hashtags. The grounding errors differed. The smaller model claimed that a game in progress had been won in 13.3 per cent of remarks on clock-based sports (“Luis Castillo’s strikeout was crucial for his team’s victory”), and invented names in 6.8 per cent of all remarks, usually awards (“the Game Master Award”) or misspelled team names. The larger model made fewer errors of these kinds (a claimed result in 3.3 per cent, an invented name in 1.7 per cent) but was the only one to contradict the game state, placing two baseball plays in the wrong half of the inning. Overall, 13.6 per cent of the smaller model’s English remarks and 6.8 per cent of the larger model’s contained at least one detected grounding error.

In Chinese, the smaller model performed worse on every measure. Only 20.3 per cent of its remarks were within the length limit, because it tended to begin with a stock phrase (“這是一句精彩的棒球評論！”, “this is a wonderful baseball comment!”) and continue past the limit until the token budget ran out. It used Simplified characters in 69.5 per cent of remarks, against 15.3 per cent for the larger model, and in 20.0 per cent of remarks on clock-based sports it said that the game had been won, sometimes with an invented final score. The larger model’s Chinese remarks were within the length limit in 91.5 per cent of cases, but it too claimed a result in 10.0 per cent of remarks on clock-based sports, for example that a home run in the third inning “helped the Herons win” (幫助 Harbor City Herons 取得勝利).

Latency suits live text. On the CPU, a remark took 1.2 seconds (English) and 2.1 seconds (Chinese) with the smaller model and about 3.5 to 3.9 seconds with the larger. Highlights occur every 40 seconds or more in most sports, so the remark arrives well before the next highlight.

5.4 Footprint

Table 5 reports the cost of running the service.

Table 5. Running costs.

Note. Simulation on one core of an Apple M4 (Node.js v18.20.8). Database figures from read-only queries on 2026–10–01; stored size is after Postgres compression. Download sizes of the model weight files from the Hugging Face model repositories; tokenizer and configuration files add under 10 MB.

The deterministic parts of the service are cheap. Simulating a game took on average between 0.005 and 0.18 milliseconds on one CPU core, and a stored game occupied between 1.9 and 68.6 kilobytes of event data before compression. On the deployed database, 254 scheduled games held 78.5 events each on average and occupied 10.2 kilobytes each after Postgres compression. The neural parts are cheap to the operator and costly to the viewer. Nothing is run on the server, but the viewer downloads 38 megabytes for English speech, 483 megabytes for the smaller language model on WebGPU (786 megabytes on the CPU path), and 1.2 gigabytes for the larger model (1.8 gigabytes on the CPU path). The models are cached by the browser after the first download.

6. Discussion

6.1 The model names, the engine decides

The evaluations support the central design choice. The deterministic engines met the consistency and credibility requirements without a language model, and they did so at negligible cost. The language models, when tested on the jobs they were given, made errors at rates that would be unacceptable in anything that decides outcomes or enters a shared record: a fifth of the smaller model’s Chinese remarks on clock-based sports announced a result that had not happened. Under the architecture of Figure 1, such a remark is a passing mistake heard by one viewer. Had the same model been asked to narrate the game without a simulation behind it, the error would have become the game.

This is the division that automated journalism reached by other means. Template systems are trusted because they cannot state what is not in the data (Carlson, 2015; Clerwall, 2014), and neural data-to-text systems lost that guarantee in exchange for fluency (Wiseman et al., 2017). A generated sports service does not have to choose between the two, because it can use templates for everything that must be true and a model for colour that is allowed to be wrong in small ways. The remaining failures are concentrated in a few detectable patterns: claims of a final result, invented names and numbers, and wrong language. Each can be checked with a few lines of code before a remark is shown, as the measures in Section 4.3 do, and remarks that fail can be discarded. Detection of hallucination in general is harder (Manakul et al., 2023), but the structured context of a sports broadcast makes the most damaging kinds easy to catch.

6.2 Generation as a draft for the user

The sport designer shows a different role for the model. Here errors are not fleeting but are made once and then fixed by the user. The 1.5 billion parameter model was good enough at the structural decision (24 of 25 parsed answers) that a user would rarely need to change the game style, and good enough at naming that most drafts were a useful starting point. It was not good enough to be trusted with moves or translations, and neither model should be asked to set rules or numbers. The design kept rules and numbers in presets and made every generated item editable, so the model’s errors reduce the value of a draft without breaking a league. This matches the role that Gallotta et al. (2024) describe for language models as designers’ assistants rather than designers.

The difference between the two models has a practical consequence. The smaller model is a third of the download, which matters for a browser application, but its structural errors are severe enough to make the draft misleading. For drafting, which is done once by a user who has chosen to wait, the larger model is the better choice. For commentary, which runs for the whole broadcast and is optional, the smaller model’s English is serviceable and its Chinese is not, so the choice of model should depend on the language of the broadcast.

6.3 Uncertainty and balance

The simulated games are credible one at a time but not as a league. In baseball and basketball the stronger generated teams separated from the weaker ones far more than real teams do. For an entertainment service this matters more than it would in a statistical exercise, since the uncertainty-of-outcome argument applies at the level of the season as well as the game (Fort and Maxcy, 2003; Rottenberg, 1956). A league in which the same two teams always reach the top is less interesting to follow. The remedy lies in the generator, not the engines: a narrower spread of team strengths, or a mechanism that redistributes talent between seasons, as real leagues do. Season length compounds the problem. A 28-day season with a game every 30 minutes gives each team 336 games, and the Noll–Scully ratio rises with season length when strengths differ (Owen and King, 2015). Generated leagues may need to be designed for balance over many more games than any real league plays.

The reveal mechanism protects the other source of suspense. Because the result is stored before the game is shown and the feed returns only past events, viewers cannot learn the outcome ahead of time, and every viewer discovers it together. This reproduces the liveness that Hutchins and Rowe (2012) describe for networked sport, at the cost of making games unchangeable once scheduled. Whether viewers of a generated league care about spoilers as much as viewers of real sport is an empirical question that this study does not answer.

6.4 Risks

Three risks deserve attention. The first is confusion with real sport. Generated players have realistic names and nationalities, and a generated result could be mistaken for a real one if shared without context. The service states on every page that all teams, players and games are simulated, and the share images were designed to show the matchup and not the result. The second is gambling. Simulated races and matches are already offered as betting products, and a generated league with a published schedule could be used in the same way. Nothing in the service supports wagering, but its outputs are public and predictable to anyone who can read the database. The third is linguistic. The smaller model drifted into Simplified Chinese in most of its Chinese commentary despite an explicit request for Traditional characters, which for users in Taiwan is a visible failure and not a matter of taste. Users of languages that small models handle less well will receive a worse service, a pattern that has been noted for language models generally (Bender et al., 2021).

6.5 Limitations

The study has the limitations of a single-artefact design study. The format coding in Section 4.2 was done by the author, and a different coder might judge some sports differently. The automatic measures detect only certain kinds of error and are lower bounds. No human ratings of quality and no study of viewers were collected, so the article says nothing about whether people enjoy the service or find its commentary engaging. The models were run on a CPU with greedy decoding; browsers with WebGPU would be faster, and sampling would change the outputs. Only two models from one family were tested. The simulated benchmarks rest on one season of each real league, and the Premier League season of 2023–24 had more goals than most recent seasons. Finally, the author built the service, which gives an insider’s understanding of its design and an interest in its success.

7. Conclusion

A sports entertainment service can now be built in which any user creates a league for any game and watches it play live, with text, animation and voice, at almost no cost to the operator. This study suggests how to divide the work in such a service. Outcomes should be decided by deterministic, seeded simulation, stored before they are shown and revealed by the clock, so that every viewer sees the same game and nobody can learn the result in advance. Small language models are useful for proposing names and drafts that a user then corrects, and for optional remarks that are never stored. They are not reliable enough to decide anything that other viewers will see. At 1.5 billion parameters a model makes a sound structural choice for most unfamiliar sports. At either size it still announces results that have not happened in a meaningful share of its remarks. The most useful next steps are human evaluation of the commentary and of viewers’ experience, automatic filters for the detectable errors, and generators that keep generated leagues as balanced as real ones.

Declarations

Data and code availability. The scripts, raw benchmark data and outputs for this article are in the replication folder that accompanies the manuscript: scripts/00_benchmarks.sh and 00_benchmarks.py (real leagues), 01_simulation.ts (Table 1, Table 2 and part of Table 5), 02_design.mts and 03_commentary.mts (model runs, with all prompts and answers logged in output/design_runs.json and output/commentary_runs.json), 04_footprint.sh and 05_production.ts (Table 5) and 06_analysis.py (Tables 3 and 4). The source code of the service is available from the author on request. The real-league data come from the MLB Stats API (statsapi.mlb.com), the NHL statistics API (api.nhle.com), ESPN’s public standings interface, football-data.co.uk and the nflverse project (github.com/nflverse/nfldata).

Declaration of generative AI and AI-assisted technologies in the writing process. During the preparation of this work the author used Claude (Anthropic), an AI assistant, to help write the software of the service and the analysis scripts and to draft and edit the text. After using this tool, the author reviewed and edited the content as needed and takes full responsibility for the content of the publication.

Competing interests. The author built and operates the service studied in this article. The author has no financial interest in it and declares no other competing interests.

Funding. This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

References

Bender, E.M., Gebru, T., McMillan-Major, A., Shmitchell, S., 2021. On the dangers of stochastic parrots: Can language models be too big? In: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. ACM, New York, pp. 610–623. https://doi.org/10.1145/3442188.3445922

Billings, A.C., Ruihley, B.J., 2013. Why we watch, why we play: The relationship between fantasy sport and fanship motivations. Mass Communication and Society 16 (1), 5–25. https://doi.org/10.1080/15205436.2011.635260

Carlson, M., 2015. The robotic reporter: Automated journalism and the redefinition of labor, compositional forms, and journalistic authority. Digital Journalism 3 (3), 416–431. https://doi.org/10.1080/21670811.2014.976412

Clerwall, C., 2014. Enter the robot journalist: Users’ perceptions of automated content. Journalism Practice 8 (5), 519–531. https://doi.org/10.1080/17512786.2014.883116

Diakopoulos, N., 2019. Automating the News: How Algorithms Are Rewriting the Media. Harvard University Press, Cambridge, MA. https://doi.org/10.4159/9780674239302

Dixon, M.J., Coles, S.G., 1997. Modelling association football scores and inefficiencies in the football betting market. Journal of the Royal Statistical Society: Series C (Applied Statistics) 46 (2), 265–280. https://doi.org/10.1111/1467-9876.00065

Dwyer, B., Kim, Y., 2011. For love or money: Developing and validating a motivational scale for fantasy football participation. Journal of Sport Management 25 (1), 70–83. https://doi.org/10.1123/jsm.25.1.70

Fort, R., Maxcy, J., 2003. Competitive balance in sports leagues: An introduction. Journal of Sports Economics 4 (2), 154–160. https://doi.org/10.1177/1527002503004002005

Gallotta, R., Todd, G., Zammit, M., Earle, S., Liapis, A., Togelius, J., Yannakakis, G.N., 2024. Large language models and games: A survey and roadmap. IEEE Transactions on Games, 1–18. https://doi.org/10.1109/TG.2024.3461510

Gatt, A., Krahmer, E., 2018. Survey of the state of the art in natural language generation: Core tasks, applications and evaluation. Journal of Artificial Intelligence Research 61, 65–170. https://doi.org/10.1613/jair.5477

Graefe, A., Haim, M., Haarmann, B., Brosius, H.-B., 2018. Readers’ perception of computer-generated news: Credibility, expertise, and readability. Journalism 19 (5), 595–610. https://doi.org/10.1177/1464884916641269

Hamari, J., Sjöblom, M., 2017. What is eSports and why do people watch it? Internet Research 27 (2), 211–232. https://doi.org/10.1108/IntR-04-2016-0085

Hevner, A.R., March, S.T., Park, J., Ram, S., 2004. Design science in information systems research. MIS Quarterly 28 (1), 75–105. https://doi.org/10.2307/25148625

Hutchins, B., Rowe, D., 2012. Sport Beyond Television: The Internet, Digital Media and the Rise of Networked Media Sport. Routledge, New York. https://doi.org/10.4324/9780203120415

Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y.J., Madotto, A., Fung, P., 2023. Survey of hallucination in natural language generation. ACM Computing Surveys 55 (12), 248:1–248:38. https://doi.org/10.1145/3571730

Johnson, B.K., Rosenbaum, J.E., 2015. Spoiler alert: Consequences of narrative spoilers for dimensions of enjoyment, appreciation, and transportation. Communication Research 42 (8), 1068–1088. https://doi.org/10.1177/0093650214564051

Kendall, G., Knust, S., Ribeiro, C.C., Urrutia, S., 2010. Scheduling in sports: An annotated bibliography. Computers & Operations Research 37 (1), 1–19. https://doi.org/10.1016/j.cor.2009.05.013

Knobloch-Westerwick, S., David, P., Eastin, M.S., Tamborini, R., Greenwood, D., 2009. Sports spectators’ suspense: Affect and uncertainty in sports entertainment. Journal of Communication 59 (4), 750–767. https://doi.org/10.1111/j.1460-2466.2009.01456.x

Leavitt, J.D., Christenfeld, N.J.S., 2011. Story spoilers don’t spoil stories. Psychological Science 22 (9), 1152–1154. https://doi.org/10.1177/0956797611417007

Lindsey, G.R., 1963. An investigation of strategies in baseball. Operations Research 11 (4), 477–501. https://doi.org/10.1287/opre.11.4.477

Maher, M.J., 1982. Modelling association football scores. Statistica Neerlandica 36 (3), 109–118. https://doi.org/10.1111/j.1467-9574.1982.tb00782.x

Manakul, P., Liusie, A., Gales, M., 2023. SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Singapore, pp. 9004–9017. https://doi.org/10.18653/v1/2023.emnlp-main.557

Neale, W.C., 1964. The peculiar economics of professional sports. The Quarterly Journal of Economics 78 (1), 1–14. https://doi.org/10.2307/1880543

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C.L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al., 2022. Training language models to follow instructions with human feedback. In: Advances in Neural Information Processing Systems 35, pp. 27730–27744.

Owen, P.D., King, N., 2015. Competitive balance measures in sports leagues: The effects of variation in season length. Economic Inquiry 53 (1), 731–744. https://doi.org/10.1111/ecin.12102

Park, J.S., O’Brien, J., Cai, C.J., Morris, M.R., Liang, P., Bernstein, M.S., 2023. Generative agents: Interactive simulacra of human behavior. In: Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. ACM, New York, pp. 1–22. https://doi.org/10.1145/3586183.3606763

Peffers, K., Tuunanen, T., Rothenberger, M.A., Chatterjee, S., 2007. A design science research methodology for information systems research. Journal of Management Information Systems 24 (3), 45–77. https://doi.org/10.2753/MIS0742-1222240302

Peterson, E.M., Raney, A.A., 2008. Reconceptualizing and reexamining suspense as a predictor of mediated sports enjoyment. Journal of Broadcasting & Electronic Media 52 (4), 544–562. https://doi.org/10.1080/08838150802437263

Pratap, V., Tjandra, A., Shi, B., Tomasello, P., Babu, A., Kundu, S., Elkahky, A., Ni, Z., Vyas, A., Fazel-Zarandi, M., Baevski, A., Adi, Y., Zhang, X., Hsu, W.-N., Conneau, A., Auli, M., 2024. Scaling speech technology to 1,000+ languages. Journal of Machine Learning Research 25 (97), 1–52.

Puduppully, R., Dong, L., Lapata, M., 2019. Data-to-text generation with content selection and planning. Proceedings of the AAAI Conference on Artificial Intelligence 33 (01), 6908–6915. https://doi.org/10.1609/aaai.v33i01.33016908

Qwen Team, 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. https://doi.org/10.48550/arXiv.2412.15115

Rottenberg, S., 1956. The baseball players’ labor market. Journal of Political Economy 64 (3), 242–258. https://doi.org/10.1086/257790

Ruan, C.F., Qin, Y., Parthasarathy, A.R., Zhou, X., Lai, R., Jin, H., et al., 2024. WebLLM: A high-performance in-browser LLM inference engine. arXiv preprint arXiv:2412.15803. https://doi.org/10.48550/arXiv.2412.15803

Shaker, N., Togelius, J., Nelson, M.J., 2016. Procedural Content Generation in Games. Springer, Cham. https://doi.org/10.1007/978-3-319-42716-4

Sjöblom, M., Hamari, J., 2017. Why do people watch others play video games? An empirical study on the motivations of Twitch users. Computers in Human Behavior 75, 985–996. https://doi.org/10.1016/j.chb.2016.10.019

Summerville, A., Snodgrass, S., Guzdial, M., Holmgård, C., Hoover, A.K., Isaksen, A., Nealen, A., Togelius, J., 2018. Procedural content generation via machine learning (PCGML). IEEE Transactions on Games 10 (3), 257–270. https://doi.org/10.1109/TG.2018.2846639

Szymanski, S., 2003. The economic design of sporting contests. Journal of Economic Literature 41 (4), 1137–1187. https://doi.org/10.1257/jel.41.4.1137

Taniguchi, Y., Feng, Y., Takamura, H., Okumura, M., 2019. Generating live soccer-match commentary from play data. Proceedings of the AAAI Conference on Artificial Intelligence 33 (01), 7096–7103. https://doi.org/10.1609/aaai.v33i01.33017096

Thurman, N., Dörr, K., Kunert, J., 2017. When reporters get hands-on with robo-writing: Professionals consider automated journalism’s capabilities and consequences. Digital Journalism 5 (10), 1240–1259. https://doi.org/10.1080/21670811.2017.1289819

Togelius, J., Yannakakis, G.N., Stanley, K.O., Browne, C., 2011. Search-based procedural content generation: A taxonomy and survey. IEEE Transactions on Computational Intelligence and AI in Games 3 (3), 172–186. https://doi.org/10.1109/TCIAIG.2011.2148116

van Dalen, A., 2012. The algorithms behind the headlines: How machine-written news redefines the core skills of human journalists. Journalism Practice 6 (5–6), 648–658. https://doi.org/10.1080/17512786.2012.667268

Wiseman, S., Shieber, S., Rush, A., 2017. Challenges in data-to-document generation. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Copenhagen, pp. 2253–2263. https://doi.org/10.18653/v1/D17-1239

Wulf, T., Schneider, F.M., Beckert, S., 2020. Watching players: An exploration of media enjoyment on Twitch. Games and Culture 15 (3), 328–346. https://doi.org/10.1177/1555412018788161

Yannakakis, G.N., Togelius, J., 2018. Artificial Intelligence and Games. Springer, Cham. https://doi.org/10.1007/978-3-319-63519-4

Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., Hou, L., 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911. https://doi.org/10.48550/arXiv.2311.07911

https://github.com/gwrxuk/world

Abstract

World models, learned simulators that predict how an environment responds to actions, have become central to model-based reinforcement learning, robotics and interactive video generation. This paper surveys the main families of world models: latent state-space models (World Models, PlaNet, Dreamer), value-equivalent models (MuZero, TD-MPC), joint-embedding predictive architectures (JEPA) and large generative video and interactive-environment models (DIAMOND, GameNGen, Genie, Cosmos). It then explains how they are trained: data sources, objectives, architectures, multi-step training, stabilising techniques, compute and evaluation. We then present NextWM, a 108,577-parameter latent world model implemented in dependency-free TypeScript and served by Next.js. The same code trains in a browser Web Worker and in a server route handler. NextWM learns a pixel-based navigation task from 10,000 exploration transitions, imagines open-loop futures that beat a copy-last-frame baseline from horizon 2 onward, and plans with the cross-entropy method in its latent space. Our first configuration planned no better than random despite low pixel error. An ablation shows why: pixel error barely varied across configurations, while the model’s ability to rank action sequences by return rose from 0.42 to 0.82 (Spearman correlation) when the exploration data covered sustained actions and the reward loss was up-weighted. With one further round of planner-collected data, NextWM reduces the mean final distance to the goal from 0.91 (random) to 0.38 and succeeds in 13% of episodes, against 100% for a planner with the true simulator. The code and every reported number are reproducible with one command per experiment.

Keywords: world models; model-based reinforcement learning; latent dynamics; video generation; planning; cross-entropy method; Next.js; WebAssembly-free in-browser training

1. Introduction

A world model is a learned function that predicts how an environment will change in response to an agent’s actions. Given a history of observations and a proposed action, it returns a distribution over what comes next: the next observation, a compact latent state, a reward, or all three. An agent that has such a model can act by imagining the consequences of its choices instead of only reacting to the present.

The idea is old. Craik [1] argued that organisms carry “small-scale models” of external reality, and Sutton’s Dyna architecture [2] placed a learned model inside a reinforcement-learning (RL) loop. What changed after 2018 is scale and representation. Deep latent-variable models made it possible to learn dynamics directly from pixels [3, 4], and large generative video models made it possible to simulate open-ended, photorealistic scenes that respond to control inputs [13–18].

Today the term covers research lines that rarely cite each other. Model-based RL builds compact latent models (PlaNet, Dreamer, MuZero, TD-MPC) to support planning and policy learning. Generative modelling builds large video and interactive-environment models (Genie, GameNGen, DIAMOND, Cosmos, Sora-class generators) to render plausible futures. A representation-first line, the joint-embedding predictive architecture (JEPA), argues that prediction should happen in an abstract embedding space, not in pixels [20–23].

This paper makes three contributions:

1. A unified survey. We define world models formally, place the main systems in one taxonomy, and compare them on state representation, prediction target and use (Sections 2–3).

2. A practical training guide. We describe how world models are trained: data, objectives, architectures, multi-step training, stabilising techniques, compute and evaluation (Section 4).

3. NextWM. We design and release a small latent world model written entirely in TypeScript, with no machine-learning dependencies, served by Next.js. The same code trains in a browser Web Worker and in a Next.js server route, and it plans with the cross-entropy method inside its own imagination. We report measured results, including a negative result on planning, and the ablation that fixes it (Sections 5–7).

2. Background and Formal Definition

We model the environment as a partially observable Markov decision process (POMDP) with hidden state s_t, action a_t, observation o_t and reward r_t. The agent never observes s_t, so a world model replaces it with a learned latent state z_t that summarises the history. Most world models factor into four learned components:

Encoder: z_t ~ q_φ(z_t | z_{t−1}, a_{t−1}, o_t) (1)

Dynamics: ẑ_{t+1} ~ p_θ(ẑ_{t+1} | z_t, a_t) (2)

Decoder: ô_t ~ p_θ(o_t | z_t) Reward: r̂_t ~ p_θ(r_t | z_t) (3)

The dynamics model (2) is the world model proper; the encoder grounds it in observations. The decoder is optional. MuZero, TD-MPC and JEPA-style models drop it and never reconstruct pixels, while generative video models keep it and make it the main output.

A world model is used in one of three ways:

• Planning: search over action sequences inside the model and execute the best first action (model-predictive control with CEM [26] or MPPI [27], or Monte Carlo tree search [8]).

• Policy learning in imagination: roll the model out to generate synthetic trajectories and train an actor-critic on them [5, 25].

• Simulation: let a person or another agent interact with the model as if it were the environment [13–16].

Two properties separate a world model from a plain video predictor. It must be action-conditioned, so different actions yield different futures, and it must be rollable, so its own predictions can be fed back as inputs for many steps without collapsing.

3. Survey of Existing World Models

Existing world models split into three families by what they predict: compact latent states for control, abstract embeddings for representation learning, or full video frames for simulation. Table 1 places representative systems side by side.

Table 1. Representative world models, grouped by family.

3.1 Latent world models for control

Ha and Schmidhuber [3] trained three parts separately: a variational autoencoder (VAE) [29] that compresses each frame, a mixture-density recurrent network that predicts the next compressed frame, and a small linear controller. The controller was evolved entirely inside the learned model and then transferred to the real CarRacing and VizDoom tasks. This showed that a policy can be trained in a dream, and also that it can exploit the model’s errors.

PlaNet [4] introduced the recurrent state-space model (RSSM), whose state has a deterministic path (a GRU) that carries memory and a stochastic path that captures uncertainty; PlaNet planned with CEM at every step. Dreamer [5] kept the RSSM but replaced online planning with an actor-critic trained on imagined rollouts, backpropagating value gradients through the learned dynamics. DreamerV2 [6] switched to categorical latents and reached human-level Atari performance. DreamerV3 [7] added symlog predictions, free bits, return normalisation and KL balancing, used one set of hyperparameters across more than 150 tasks, and was the first agent to collect diamonds in Minecraft from scratch without human data.

3.2 Value-equivalent models

MuZero [8] learns a model that predicts only what planning needs: reward, value and policy. It never reconstructs observations. Combined with tree search it matched AlphaZero in Go, chess and shogi without being given the rules, and set a new state of the art on Atari. TD-MPC [9] and TD-MPC2 [10] apply the same idea to continuous control: a latent model trained with temporal-difference targets and used for MPPI planning. TD-MPC2 trained a single 317M-parameter agent across 80 tasks.

3.3 Joint-embedding predictive architectures

LeCun [20] argued that predicting pixels wastes capacity on unpredictable detail. JEPA models predict the embedding of a target from the embedding of a context, with a slowly updated target encoder that prevents representational collapse [31]. I-JEPA [21] does this for images and V-JEPA [22] for video. V-JEPA 2 [23] pre-trains on over a million hours of video, then post-trains an action-conditioned predictor on a small amount of robot interaction data and uses it for zero-shot planning.

3.4 Generative and interactive video models

This family predicts full frames, usually with diffusion [35] or autoregressive transformers [36] over visual tokens [30]. IRIS [11] tokenises frames with a VQ autoencoder and runs a GPT-style transformer over the tokens. DIAMOND [12] showed that diffusion in pixel space preserves visual details that matter for control, and set a record for agents trained entirely inside a world model on the Atari 100k benchmark. GameNGen [13] fine-tuned Stable Diffusion to simulate the game DOOM in real time from action inputs.

Genie [14] learned a latent action space from unlabelled internet video, so that any image can become a playable 2D world. Genie 2 [15] and Genie 3 [16] extended this to 3D environments that stay consistent over long interactions. OpenAI’s Sora report [17] framed large video generators, built on diffusion transformers [37], as “world simulators”, though without explicit action inputs. UniSim [33] learned an action-conditioned simulator of real-world interaction from mixed datasets. NVIDIA Cosmos [18] packages video tokenisers, diffusion and autoregressive models as open “world foundation models” for robotics and driving, alongside driving-specific models such as GAIA-1 [19].

3.5 Trends

Three trends run across the families. Models are growing and are increasingly pre-trained on passive video before being adapted to actions. Discrete tokens and diffusion are replacing Gaussian latents as the default way to represent uncertainty. And the line between a model for an agent and a simulator for people is blurring: GameNGen and the Genie models are used both ways. A broader survey of this landscape is given by Ding et al. [40].

4. How World Models Are Trained

Training a world model means choosing four things: where the data comes from, what the model is asked to predict, which architecture carries the state, and how errors that compound over long rollouts are controlled. This section walks through each choice, then covers compute and evaluation.

4.1 Data

• Online interaction. Dreamer-style agents alternate between acting in the environment, storing transitions in a replay buffer, and training the model on sampled sequences. The ratio of gradient steps to environment steps (the replay or train ratio) is a key hyperparameter [7].

• Offline logs. Driving and robotics models are trained on large fleets of recorded trajectories with actions (steering, joint commands) [18, 19].

• Passive video. Internet video has no action labels. Genie infers a small discrete latent action between frames [14]; V-JEPA 2 learns from video first and uses a small amount of labelled robot data afterwards [23].

• Game engines and simulators. GameNGen trained on trajectories recorded by an RL agent playing DOOM [13]; simulators provide cheap, perfectly labelled data for any domain that can be scripted.

Coverage matters more than volume. A model is only trustworthy in the parts of state-action space its data visited, and a planner will find the parts it did not.

4.2 Objectives

Table 2 lists the main training objectives. Most modern systems combine several.

Table 2. Common world-model training objectives.

For the RSSM family the objective is an evidence lower bound (ELBO) over sequences. Writing h_t for the deterministic state and z_t for the stochastic latent, PlaNet and Dreamer minimise:

L = Σ_t [ −ln p(o_t | h_t, z_t) − ln p(r_t | h_t, z_t) + β · KL( q(z_t | h_t, o_t) ‖ p(z_t | h_t) ) ] (4)

DreamerV3 splits the KL term into a dynamics loss and a representation loss with different stop-gradients and weights (KL balancing), and clips each below one nat (free bits) so that the posterior is not forced onto an uninformative prior [7].

4.3 Architectures

• Recurrent state-space models (GRU + stochastic latent) remain the workhorse for control from pixels [4–7].

• Transformers over tokens or latents give longer memory and parallel training [11, 14].

• Diffusion models (U-Nets or diffusion transformers) give the sharpest frames and handle multi-modal futures, at the cost of several denoising steps per frame [12, 13, 37].

• Tokenisers (VQ-VAE [30] and causal video tokenisers [18]) compress frames or clips so the dynamics model works on short sequences.

4.4 Multi-step training and compounding error

A model trained only on one-step prediction (teacher forcing) sees perfect inputs during training but its own imperfect outputs at test time, so small errors compound. Three remedies are common. Open-loop unrolling trains on the model’s own predictions for H steps (latent overshooting in PlaNet [4]; the horizon-weighted unroll in TD-MPC [9]). Noise augmentation corrupts the context frames during training so the model learns to correct drift (GameNGen [13]). Short rollouts limit imagination to a few steps from real states when the model feeds a policy learner (MBPO [25]).

4.5 Stabilising techniques

• Symlog transforms and two-hot regression for targets with unknown scale [7].

• Free bits, KL balancing and a uniform mixture on categorical latents [6, 7].

• Stop-gradient or exponential-moving-average target encoders to prevent collapse in latent-prediction losses [9, 31].

• Normalised latents (LayerNorm, SimNorm), gradient clipping and Adam [28].

• Probabilistic ensembles to estimate epistemic uncertainty and keep planners away from unfamiliar states [24].

4.6 Using the model to act

Planning methods sample or optimise action sequences inside the model. CEM [26] fits a Gaussian to the best samples and resamples; MPPI [27] averages samples weighted by exponentiated return; MCTS [8] searches a tree over discrete actions. Imagination-based policy learning instead trains an actor and critic on λ-returns computed over imagined trajectories [5]. Dyna-style methods mix imagined transitions into the data of a model-free learner [2, 25].

4.7 Compute and scale

Scale spans six orders of magnitude. A PlaNet- or Dreamer-sized model trains on one GPU in hours to days. TD-MPC2 reaches 317M parameters [10]; Genie has 11B parameters [14]; foundation models such as Cosmos are trained on millions of hours of video on thousands of GPUs [18]. The model presented in Section 5 has about 10⁵ parameters and trains on a laptop CPU in minutes, which is the regime where the whole pipeline can be inspected and taught.

4.8 Evaluation

• Prediction quality: MSE, PSNR, SSIM and LPIPS per frame; Fréchet Video Distance for generated clips; error as a function of rollout horizon.

• Downstream control: return and sample efficiency (for example Atari 100k [12]) when the model is used for planning or policy learning.

• Planning relevance: whether the model ranks candidate actions the way the true environment would. Section 7 shows this can disagree sharply with pixel error.

• Human judgement and consistency: whether raters can tell simulated from real clips [13], and whether a revisited place looks the same [16].

5. NextWM: A Browser-Native Latent World Model

NextWM is a deliberately small world model whose whole life cycle (data collection, training, imagination and planning) runs on the web platform. Its design follows four principles:

1. Zero ML dependencies. Dense layers, backpropagation and Adam are implemented in about 230 lines of TypeScript, so every gradient can be read and checked.

2. Isomorphic code. The same modules run in a browser Web Worker and in a Next.js route handler on Node.js.

3. Planning-first objective. The model is trained by unrolling its own dynamics, so the quantity it optimises is the one the planner uses.

4. Reproducibility. All randomness comes from seeded generators; one command regenerates every number in this paper.

5.1 Environment

PointNav is a continuous-control task observed only through pixels. A point mass with momentum moves in the square [−1, 1]² under a two-dimensional force a_t ∈ [−1, 1]². Velocity decays by a factor of 0.8 per step, the force adds 0.05·a_t, and walls absorb momentum. Each episode places the agent and a goal uniformly in [−0.8, 0.8]² and lasts 50 steps. The reward is the negative Euclidean distance to the goal, and an episode counts as a success if it ends within 0.2 of the goal.

The agent sees 16×16 grayscale frames in which the agent is a Gaussian blob of intensity 1.0 and the goal a blob of intensity 0.5. Because a single frame does not show velocity, each observation stacks the current and previous frames (512 values), as in DQN [38].

5.2 Model

Figure 1. NextWM architecture. The encoder maps two stacked frames to a 16-dimensional latent. A residual dynamics network unrolls the latent for H steps under the logged actions; the decoder and reward head read every predicted latent. At decision time CEM searches over action sequences using only the dynamics and reward head.

NextWM has four multilayer perceptrons (Figure 1), 108,577 parameters in total:

• Encoder E: 512 → 128 → 16, tanh activations, giving a bounded latent z_t = E(o_t).

• Dynamics F: 18 → 128 → 16 with a residual connection, ẑ_{t+1} = ẑ_t + F([ẑ_t, a_t]). The residual form makes the identity map the default, which suits slowly changing scenes.

• Decoder D: 16 → 128 → 256, ReLU then sigmoid, reconstructing the current frame.

• Reward head R: 16 → 64 → 1.

5.3 Training objective

Each training example is a window of H + 1 observations, H actions and H + 1 rewards sampled from a replay buffer. The first observation is encoded once and the dynamics are unrolled open-loop: ẑ_0 = E(o_t) and ẑ_k = ẑ_{k−1} + F([ẑ_{k−1}, a_{t+k−1}]). The loss is

L = Σ_{k=0..H} ρ^k ℓ_k (5)

ℓ_k = λ_x ‖D(ẑ_k) − x_{t+k}‖² + λ_r (R(ẑ_k) − r_{t+k})² + 𝟙[k>0] · λ_c ‖ẑ_k − sg(E(o_{t+k}))‖² (6)

where x is the current frame, sg is the stop-gradient operator and ρ = 0.9 down-weights distant steps. The three terms play distinct roles. Reconstruction anchors the latent to the pixels and prevents the collapse to which latent-only losses are prone. Reward regression puts the task into the latent. Latent consistency, as in TD-MPC [9], keeps imagined latents on the manifold of real encodings, so that a latent reached by imagination can be decoded and scored as if it had been observed. Gradients flow through the whole unrolled chain back into the encoder.

The implementation passes a finite-difference gradient check with a maximum relative error of 1.4 × 10⁻⁶ in double precision (40 random coordinates, consistency term disabled because finite differences also move its stop-gradient target). In single precision, used for speed, the check is dominated by rounding noise.

5.4 Planning in imagination

At every environment step NextWM encodes the current observation once and runs CEM [26] entirely in latent space. Each of P candidate action sequences of length 12 is rolled out through F in one batched matrix operation, scored by the sum of predicted rewards R(ẑ_k), and the 10% best candidates refit a diagonal Gaussian. After four iterations the first action of the mean is executed, and the mean is shifted by one step to warm-start the next decision. No pixels are decoded during planning. The same planner, given the true simulator instead of the model, serves as an oracle upper bound.

6. Implementation on Next.js

NextWM is a Next.js 15 application using the App Router, React 19 and TypeScript. Table 3 lists the modules; everything under lib/wm is framework-independent.

Table 3. Source layout of the NextWM repository.

6.1 Training off the main thread

The browser UI never trains on the main thread. The client component creates a module worker with new Worker(new URL(“../lib/wm/worker.ts”, import.meta.url)), which Next.js bundles as a separate chunk. The worker trains in chunks of 25 gradient steps and yields to its event loop between chunks, so stop requests are handled promptly and loss updates stream to the UI. Imagination and planning requests carry an identifier and are answered asynchronously.

6.2 Server-side training

Because the model is plain TypeScript, the route handler app/api/train/route.ts imports the same modules and trains on Node.js. The browser can load the returned weights into its worker, so a slow device can train on the server and still imagine and plan locally. Weights are exported as JSON arrays and can be downloaded from the UI.

6.3 The interactive lab

The lab page shows the true environment next to the model’s open-loop dream. After “Start dreaming”, every real action is also fed to the model, which re-imagines the whole trajectory from the snapshot observation, so the viewer watches prediction error accumulate in real time. The environment can be driven from the keyboard or by the CEM agent planning inside the model.

A production build (next build) compiles successfully; the lab page is 2.88 kB of route-specific JavaScript (105 kB first load including React and Next.js), and the model code ships in a separate worker chunk. We verified the end-to-end path in headless Chromium: training in the worker, streaming loss updates, and imagination and CEM control through the worker all ran without console errors.

7. Experiments

We ask three questions. Does NextWM learn the environment, measured as open-loop prediction error? Can a planner use it, measured as return-ranking correlation and closed-loop control? And which design choices matter?

7.1 Setup

Table 4. Experimental protocol. The ablations in Section 7.4 change only the exploration policy, λ_r and the learning rate.

Two exploration policies generate data. The OU policy applies temporally correlated noise, a_t = clip(0.7·a_{t−1} + 0.6·u_t) with u_t uniform on [−1, 1]²; its actions largely cancel, so high speeds are rare. The hold policy draws a uniform random action and holds it for 1 to 10 steps, so the data contains the sustained full-force moves a planner tends to choose.

7.2 Training dynamics

Figure 2. Training losses of the full NextWM configuration (mean over 100-step windows, log scale). Reconstruction and reward losses fall steadily. Latent consistency first rises, as the encoder spreads its latents apart, then falls. None has plateaued after 6,000 steps.

Training ran at about 12.4 gradient steps per second on one CPU thread (483 s for 6,000 steps while sharing the machine with three other runs). Reconstruction error fell from 0.0383 to 0.0047 and reward error from 0.320 to 0.017 (Figure 2).

7.3 Open-loop imagination

Figure 3. Pixel MSE of open-loop predictions against the true frame, as a function of horizon, on 40 held-out episodes (160 windows). The baseline repeats the last observed frame.

Figure 3 compares NextWM’s open-loop predictions with a copy-last-frame baseline. At short horizons the baseline wins: one step ahead its error is 0.0023 against 0.0049 for NextWM, because the decoder has a reconstruction floor while consecutive frames are nearly identical. From horizon 2 onward NextWM is better, and at horizon 15 its error is 0.0134 against 0.0229. The model has therefore learned motion, not just appearance. Figure 4 shows a held-out rollout, and it reveals a specific failure. The imagined agent blurs, as expected of a deterministic model averaging over uncertain futures, but follows the real one into the top-right corner. The goal, which never moves, fades from the imagination after about eight steps: the dynamics network does not preserve the part of the latent that encodes the static goal. Because the reward depends on the goal, this matters for planning.

Figure 4. A held-out rollout. Top: true frames. Bottom: NextWM’s open-loop imagination from the first frame under the same actions, with no further observations.

7.4 Planning, and why pixel error is not enough

Our first configuration, trained for 4,000 steps on OU data with λ_r = 1, predicted pixels reasonably but was useless for planning. Its return-ranking correlation was 0.22, and CEM control inside it reached a mean return of -42.2, no better than random actions (-41.0).

We hypothesised two causes. First, the model had rarely seen the sustained actions CEM selects, so it was queried outside its data. Second, the reward head, which is the planner’s only signal, received little weight next to 256 pixel terms. Table 5 and Figure 5 cross the two factors, with every run trained for 6,000 steps.

Table 5. Ablation over exploration data and reward weighting. Return is the undiscounted sum of rewards over 50 steps (higher is better). Control uses 30 episodes with the same start states for every row.

Figure 5. Return-ranking correlation (left) and closed-loop final distance (right) for each configuration. Dashed line: random actions; dotted line: oracle planner.

The result is clear. Pixel error at horizon 8 hardly moves across the four runs (0.0097 to 0.0106), yet ranking correlation ranges from 0.42 to 0.82. Each factor helps on its own: up-weighting the reward raises ranking to 0.56, and hold exploration raises it to 0.49. Together they are worth much more than either alone. Closed-loop control follows ranking, not pixel error: the full configuration reaches a mean final distance of 0.45 against 0.91 for random actions.

7.5 Closing the loop with planner data

Model-based agents such as PlaNet and Dreamer do not train on exploration data alone. They repeatedly act with the current model and retrain on what they see [4, 5]. We ran one such round: 100 episodes collected by CEM inside the model, with Gaussian action noise (σ = 0.3), were added to the 200 exploration episodes, and training continued for 3,000 steps. Collection took 162 s.

After this round the mean return improves from -22.4 to -20.2 and the mean final distance from 0.45 to 0.38. 4 of 30 episodes end within 0.2 of the goal, against 2 before the round; with 30 episodes this count is noisy, but return and distance move in the same direction. Interestingly, the ranking correlation on our constant-direction probe falls slightly, from 0.82 to 0.75. The planner data concentrates on short, corrective moves near the goal, so the model trades some accuracy on long straight-line moves, which the probe measures, for accuracy where decisions are made. A substantial gap to the oracle (-3.9 return, 100% success) remains; Section 8 discusses why.

7.6 Cost

A CEM decision with population 200 and horizon 12 makes 9,600 latent transitions and took about 125 ms on one CPU thread, so a 50-step episode took 6.2 s. The browser worker uses a smaller population (120, three iterations) to keep interactive control responsive. The oracle planner is faster (2 ms per decision) because the true simulator is a few arithmetic operations, whereas each model step is a pair of dense layers.

8. Discussion and Limitations

8.1 What the experiments teach

Pixel accuracy is a poor proxy for usefulness. Across our ablation, pixel error varied by a few per cent while planning quality varied several-fold. This is the argument behind value-equivalent models [8–10] and JEPA [20]: a model should be judged, and ideally trained, on the quantities its user consumes. We recommend reporting a planning-relevance metric, such as the return-ranking correlation used here, next to reconstruction metrics.

Coverage beats volume. The same 10,000 transitions helped planning much more when they contained sustained actions. Planners actively search for the model’s blind spots, so the data must cover the actions the planner will choose, or be refreshed by the planner itself (Section 7.5).

The web platform is a viable teaching and prototyping substrate. A complete world-model pipeline (data, training, imagination, planning and visualisation) fits in about 1,200 lines of TypeScript and runs in a browser tab without a GPU or a Python stack. Because the code is isomorphic, the same model can train on a Next.js server and plan on the client.

8.2 Limitations

• Toy environment. PointNav has four state variables and 16×16 frames. The findings about coverage and planning relevance are well known at scale, but the absolute numbers here say nothing about larger tasks.

• Deterministic model. NextWM has no stochastic latent and no ensemble, so it cannot represent uncertainty, and CEM can exploit its errors. An RSSM-style stochastic state [4] or an ensemble [24] is the natural next step.

• Small evaluation. Control results use 30 episodes and one training seed per configuration, without confidence intervals. Differences of a few percentage points in success rate are within noise; the ranking and distance trends are larger.

• Gap to the oracle. Even the best configuration is far from the oracle planner. Two measured causes contribute: the goal fades from long imagined rollouts (Figure 4), and the reward error is still comparable to the reward differences that decide the last few steps near the goal. Splitting the latent into a static context (the goal) and a dynamic part (the agent), or feeding the encoded goal to every dynamics step, is a direct fix.

• Speed. Plain JavaScript on one CPU thread manages about 12 gradient steps per second for this model. WebGPU kernels or TensorFlow.js [34] would raise throughput by orders of magnitude, at the cost of the zero-dependency design.

8.3 Future work

Natural extensions are a stochastic latent with KL balancing [7], an actor-critic trained in imagination to amortise planning [5], WebGPU matrix kernels, several training seeds with confidence intervals, and richer browser-rendered environments. A further direction is to use the Next.js server for heavy pre-training on passive video and the client for fast action-conditioned adaptation, mirroring the V-JEPA 2 recipe [23] at small scale.

9. Conclusion

World models now span compact latent models for control, value-equivalent planners, joint-embedding predictors and billion-parameter interactive video generators. They share one recipe: learn to predict the consequences of actions, then act by imagining them. We surveyed these families and the practical choices involved in training them. We also presented NextWM, a transparent world model that trains, imagines and plans entirely within a Next.js application. Its main lesson matches what larger systems have found: a world model is only as good as its ability to rank the choices its user will make. Reaching that ability depended on data coverage and on the loss weighting, not on pixel accuracy.

References

[1] K. Craik. The Nature of Explanation. Cambridge University Press, 1943.

[2] R. S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM SIGART Bulletin, 2(4):160–163, 1991.

[3] D. Ha and J. Schmidhuber. Recurrent world models facilitate policy evolution. NeurIPS, 2018. arXiv:1803.10122.

[4] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, J. Davidson. Learning latent dynamics for planning from pixels. ICML, 2019.

[5] D. Hafner, T. Lillicrap, J. Ba, M. Norouzi. Dream to control: Learning behaviors by latent imagination. ICLR, 2020.

[6] D. Hafner, T. Lillicrap, M. Norouzi, J. Ba. Mastering Atari with discrete world models. ICLR, 2021.

[7] D. Hafner, J. Pasukonis, J. Ba, T. Lillicrap. Mastering diverse domains through world models. arXiv:2301.04104, 2023; Nature, 2025.

[8] J. Schrittwieser et al. Mastering Atari, Go, chess and shogi by planning with a learned model. Nature, 588:604–609, 2020.

[9] N. Hansen, X. Wang, H. Su. Temporal difference learning for model predictive control. ICML, 2022.

[10] N. Hansen, H. Su, X. Wang. TD-MPC2: Scalable, robust world models for continuous control. ICLR, 2024.

[11] V. Micheli, E. Alonso, F. Fleuret. Transformers are sample-efficient world models. ICLR, 2023.

[12] E. Alonso, A. Jelley, V. Micheli, A. Kanervisto, A. Storkey, T. Pearce, F. Fleuret. Diffusion for world modeling: Visual details matter in Atari. NeurIPS, 2024.

[13] D. Valevski, Y. Leviathan, M. Arar, S. Fruchter. Diffusion models are real-time game engines. arXiv:2408.14837, 2024.

[14] J. Bruce et al. Genie: Generative interactive environments. ICML, 2024.

[15] Google DeepMind. Genie 2: A large-scale foundation world model. Technical blog, December 2024.

[16] Google DeepMind. Genie 3: A new frontier for world models. Technical blog, August 2025.

[17] T. Brooks et al. Video generation models as world simulators. OpenAI technical report, 2024.

[18] NVIDIA. Cosmos world foundation model platform for physical AI. arXiv:2501.03575, 2025.

[19] A. Hu et al. GAIA-1: A generative world model for autonomous driving. arXiv:2309.17080, 2023.

[20] Y. LeCun. A path towards autonomous machine intelligence. OpenReview preprint, 2022.

[21] M. Assran et al. Self-supervised learning from images with a joint-embedding predictive architecture. CVPR, 2023.

[22] A. Bardes et al. Revisiting feature prediction for learning visual representations from video (V-JEPA). arXiv:2404.08471, 2024.

[23] M. Assran et al. V-JEPA 2: Self-supervised video models enable understanding, prediction and planning. arXiv:2506.09985, 2025.

[24] K. Chua, R. Calandra, R. McAllister, S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. NeurIPS, 2018.

[25] M. Janner, J. Fu, M. Zhang, S. Levine. When to trust your model: Model-based policy optimization. NeurIPS, 2019.

[26] P.-T. de Boer, D. P. Kroese, S. Mannor, R. Y. Rubinstein. A tutorial on the cross-entropy method. Annals of Operations Research, 134:19–67, 2005.

[27] G. Williams et al. Information theoretic MPC for model-based reinforcement learning. ICRA, 2017.

[28] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. ICLR, 2015.

[29] D. P. Kingma and M. Welling. Auto-encoding variational Bayes. ICLR, 2014.

[30] A. van den Oord, O. Vinyals, K. Kavukcuoglu. Neural discrete representation learning. NeurIPS, 2017.

[31] J.-B. Grill et al. Bootstrap your own latent: A new approach to self-supervised learning. NeurIPS, 2020.

[32] Vercel. Next.js documentation: App Router, Route Handlers. 2025.

[33] S. Yang et al. Learning interactive real-world simulators (UniSim). ICLR, 2024.

[34] D. Smilkov et al. TensorFlow.js: Machine learning for the web and beyond. SysML, 2019.

[35] J. Ho, A. Jain, P. Abbeel. Denoising diffusion probabilistic models. NeurIPS, 2020.

[36] A. Vaswani et al. Attention is all you need. NeurIPS, 2017.

[37] W. Peebles and S. Xie. Scalable diffusion models with transformers. ICCV, 2023.

[38] V. Mnih et al. Human-level control through deep reinforcement learning. Nature, 518:529–533, 2015.

[39] WHATWG. HTML Living Standard: Web Workers.

[40] J. Ding et al. Understanding world or predicting future? A comprehensive survey of world models. arXiv:2411.14499, 2024.

Large language model (LLM) agents are moving from drafting copy to operating marketing systems directly. This paper examines how such agents work with Klaviyo, a customer data and marketing automation platform used widely in e-commerce. We treat Klaviyo as an agent environment: a set of observable state (profiles, events, metrics), actions (creating segments, templates, campaigns and flows) and constraints (API scopes, rate limits, consent rules).

Using a design-science approach grounded in Klaviyo’s public developer documentation and its Model Context Protocol (MCP) server, we make three contributions. First, we map Klaviyo’s data model and interfaces onto the perception–reasoning–action loop of tool-using agents. Second, we propose a taxonomy of six agent–Klaviyo interaction patterns, from read-only analytical agents to multi-agent orchestration pipelines. Third, we work through three scenarios and derive governance principles, chief among them a draft-then-approve boundary between agent autonomy and human sign-off.

We argue that the main risks are not model capability but mis-scoped authority: over-messaging, consent violations, deliverability damage and prompt injection through customer-supplied data. The paper closes with a research agenda on evaluation, attribution and accountability for autonomous marketing agents.

Keywords: AI agents; large language models; marketing automation; Klaviyo; Model Context Protocol; human-in-the-loop; e-commerce

1. Introduction

Marketing automation is becoming one of the first commercial domains where LLM agents act, not just advise. On 25 September 2025 Klaviyo released a Model Context Protocol (MCP) server that lets AI clients such as Claude and Cursor query and act on a Klaviyo account in natural language (Klaviyo, 2025). An agent can now read campaign performance, build segments and draft campaigns without a human opening the dashboard.

This shift raises questions the marketing and information-systems literature has not yet addressed. Prior work studies algorithmic personalization and recommender systems, which optimise one decision inside a fixed pipeline. Agents are different: they plan multi-step work, choose tools and change the system’s state. Klaviyo is a useful case because it combines a customer data platform (profiles and behavioural events) with the channels that act on that data (email, SMS, push).

We ask three research questions:

1. RQ1. How do Klaviyo’s data model and interfaces map onto the perception–reasoning–action loop of an LLM agent?

2. RQ2. What recurring patterns describe how agents work with Klaviyo, and how do they differ in autonomy and risk?

3. RQ3. What governance principles let organisations gain from agent autonomy while protecting customers, consent and sender reputation?

The contributions are (a) a conceptual mapping of Klaviyo as an agent environment, (b) a six-pattern taxonomy of agent–Klaviyo interaction, © three worked scenarios and (d) a set of design principles and a research agenda. Section 2 reviews related work. Section 3 describes the platform and Section 4 the method. Section 5 presents the taxonomy, Section 6 a reference implementation with Qwen-Image-2.1, and Section 7 three scenarios. Sections 8–10 cover the discussion, limitations and conclusion.

2. Background and Related Work

2.1 LLM agents and tool use

An LLM agent is a language model placed in a loop where it observes, reasons and calls external tools until a goal is met. ReAct showed that interleaving reasoning traces with actions improves task success over reasoning or acting alone (Yao et al., 2023). Toolformer showed that models can learn when and how to call APIs (Schick et al., 2023). Surveys now frame agents around four parts: profile, memory, planning and action (Wang et al., 2024).

The Model Context Protocol, introduced by Anthropic in November 2024, standardises how agents discover and call tools exposed by external systems (Anthropic, 2024). MCP separates the client (the agent host) from the server (the system of record). That lets a vendor such as Klaviyo publish one interface that many agents can use.

2.2 Marketing automation and customer data platforms

Marketing automation research has focused on triggered, rule-based communication: welcome series, cart-abandonment reminders and lifecycle programmes. Customer data platforms (CDPs) add a unified profile built from behavioural events. Studies consistently find that behaviour-triggered messages outperform batch campaigns on engagement and revenue per recipient. They also warn of fatigue and unsubscribes when frequency rises.

2.3 AI in personalisation and CRM

Earlier AI in marketing was mostly predictive: churn scores, predicted customer lifetime value, send-time optimisation and product recommendations (Davenport et al., 2020). Generative AI added content production: subject lines, body copy and images. Agentic AI adds operation: the system decides which segment to target, what to send and when, and executes it.

2.4 Gap

We found no peer-reviewed study of how general-purpose LLM agents operate a production marketing platform through a vendor-provided MCP interface. Existing work covers either agent architectures in the abstract or AI features inside marketing tools. This paper sits between the two.

3. The Klaviyo Platform as an Agent Environment

For an agent, Klaviyo is an environment with rich observable state, a narrower set of write actions and hard external constraints. This section describes each in turn (RQ1).

3.1 State: the data model

Klaviyo’s core objects are profiles (one per customer), events (timestamped behaviours such as Placed Order or Viewed Product), metrics (event types and their aggregates), lists and segments (static and rule-based audiences), campaigns (one-off sends), flows (triggered, multi-step automations), templates and catalogs. Together these give the agent both perception (who did what, and what worked) and levers (who to message, with what, and when).

3.2 Interfaces

Agents reach this state in two ways. Custom agents call the REST API with scoped private keys or OAuth. General-purpose agents connect through the Klaviyo MCP server, a hosted endpoint with OAuth sign-in that exposes typed tools to clients such as Claude, Cursor and ChatGPT (Klaviyo Developers). A local server remains available for developers using private keys.

Figure 1 · Klaviyo as an agent environment

Agent hosts sit outside Klaviyo; every read and write passes through the MCP server or the REST API, and sends pass a human reviewer.

3.3 Actions and their current limits

The MCP tool set is read-heavy by design. As of mid-2026 it offers broad read access to reports, metric aggregates, segments, profiles, catalogs and templates. Writes are limited to creating campaign drafts, creating and updating templates, updating profiles and subscription state, and uploading images (Everboost, 2026). Flows are read-only on every surface, and segments cannot be created through MCP. Scheduling and sending campaigns end to end was expected to roll out in Q3 2026 (Magnet Monster, 2026).

3.4 Constraints

Three constraints shape agent behaviour:

• Authority. MCP sign-in requires an Owner, Admin or Manager role, and one connection binds to one Klaviyo account.

• Throughput. The REST API uses fixed-window limits in five tiers, from 1 request/s and 15/min (XS) to 350/s and 3,500/min (XL). Exceeding them returns HTTP 429 with a Retry-After header; OAuth apps get their own quota per account, while private keys share one (Klaviyo Developers).

• Regulation. Consent state on each profile governs whether email or SMS may be sent at all.

3.5 Native agents

Klaviyo also ships its own agents. On 30 June 2026 it moved Composer, a marketing agent that plans and launches campaigns, to public beta, alongside an updated Customer Agent for 24/7 shopper support (Klaviyo Newsroom, 2026). Both read and write the same customer profile, so a service conversation can inform the next campaign and vice versa. External agents built on MCP therefore coexist with first-party agents acting on the same data.

4. Methodology

We follow a design-science research approach (Hevner et al., 2004), which builds and evaluates artefacts that address a practical problem. Our artefacts are a taxonomy, a reference implementation and a set of design principles.

The work ran in three steps:

1. Document analysis. We reviewed Klaviyo’s developer reference (API revision 2026–07–15), its MCP server documentation, product announcements from September 2025 to August 2026, and practitioner field guides. We coded each capability as read, draft-write or live-write.

2. Pattern derivation. We grouped observed and documented agent uses by goal, the objects touched and the highest write level required. Patterns were merged until each differed from the others on at least one of those three dimensions.

3. Instantiation. We built a reference agent that combines Klaviyo’s REST API with a locally hosted image model, Qwen-Image-2.1 (Qwen Team, 2026), run through stable-diffusion.cpp on an Apple M4 laptop with 24 GB of memory. We used it to walk through the scenarios in Section 7.

The evaluation is descriptive and illustrative. We do not report field experiments on revenue or engagement; Section 9 lists this as the main limitation.

5. A Taxonomy of Agent–Klaviyo Interaction Patterns

We identify six patterns, ordered by the highest write level they need (RQ2). Risk rises with write level, not with model sophistication: an analytical agent using a strong model is safer than a sending agent using a weak one.

Table 1. Six patterns of agent–Klaviyo interaction

Three observations follow. First, P1 and P2 are well served today because the MCP tool set is read-heavy. Second, P3 and P4 are where agents add the most labour savings, and where external tools such as image models plug in (Section 6). Third, P5 and P6 need governance more than capability: the question is what an agent may do without a person, not what it can do.

6. Reference Implementation: Qwen-Image-2.1 as an Agent Tool

The reference agent shows that an organisation can add capabilities Klaviyo does not offer natively (here, on-premise image generation) by exposing them to the agent as tools beside Klaviyo’s own. It instantiates patterns P3, P4 and P6 from Table 1.

6.1 Architecture

The agent host (Claude, via Claude Code or Claude Desktop) connects to two MCP servers: Klaviyo’s hosted server for account data, and a small local server that wraps Qwen-Image-2.1. Qwen-Image-2.1 is a 7.1-billion-parameter text-to-image model with a Qwen3-VL text encoder, notable for rendering legible text inside images (Qwen Team, 2026). That matters for marketing, where a banner’s headline must be spelled correctly.

Figure 2 · Reference pipeline: insight to creative to draft to approval

The image model runs on the marketer’s own machine, so prompts and unreleased product imagery never leave it; Klaviyo receives only the finished image and a draft campaign.

We ran the model with stable-diffusion.cpp on an Apple M4 with 24 GB of unified memory, using GGUF weights: the diffusion model at Q6_K (6.0 GB), the text encoder at Q4_K_M (5.0 GB) and the VAE in bf16 (0.7 GB). The full bf16 weights need about 46 GB of peak memory, so quantisation is what makes a laptop deployment possible. A 512×512 image took 788 s (20 steps at about 35 s per step). That is too slow for interactive use but fine for a batch job that prepares tomorrow’s drafts overnight.

6.2 Registering the tools

Listing 1 registers both servers with the agent host. Klaviyo’s server uses OAuth, so no key sits in the file.

Listing 1. .mcp.json: one hosted and one local MCP server

{
“mcpServers”: {
“klaviyo”: {
“type”: “http”,
“url”: “https://mcp.klaviyo.com/mcp"
},
“qwen-image”: {
“type”: “stdio”,
“command”: “python3”,
“args”: [“qwen_image_mcp.py”],
“env”: { “KLAVIYO_API_KEY”: “${KLAVIYO_API_KEY}” }
}
}
}

6.3 The image tool

Listing 2 wraps the local model as an MCP tool. The docstring is the tool’s description; it tells the agent to put exact headline words in quotes, which is how Qwen-Image-2.1 is prompted to render text.

Listing 2. qwen_image_mcp.py: Qwen-Image-2.1 exposed as MCP tools

import pathlib, subprocess, time
from mcp.server.fastmcp import FastMCP
from klaviyo_client import Klaviyo

ROOT = pathlib.Path(__file__).resolve().parent.parent
OUT, MODELS = ROOT / “outputs”, ROOT / “models”
mcp = FastMCP(“qwen-image”)

@mcp.tool()
def generate_campaign_image(prompt: str, width: int = 1024, height: int = 512,
steps: int = 20, seed: int = -1) -> str:
“””Generate an on-brand marketing image with Qwen-Image-2.1 and return its local path.

Width and height must be multiples of 32. Qwen-Image-2.1 renders legible
text, so put exact headline words in quotes inside the prompt.
“””
if width % 32 or height % 32:
raise ValueError(“width and height must be multiples of 32”)
OUT.mkdir(exist_ok=True)
out = OUT / f”campaign-{int(time.time())}.png”
subprocess.run([
str(ROOT / “sdcpp” / “sd-cli”),
“ — diffusion-model”, str(MODELS / “diffusion_models” / “qwen_image_2.1-Q6_K.gguf”),
“ — vae”, str(MODELS / “vae” / “qwen_image_2.1_vae_bf16.safetensors”),
“ — llm”, str(MODELS / “text_encoders” / “Qwen3VL-8B-Instruct-Q4_K_M.gguf”),
“-p”, prompt, “-W”, str(width), “-H”, str(height),
“ — steps”, str(steps), “ — seed”, str(seed),
“ — cfg-scale”, “6.0”, “ — sampling-method”, “euler”,
“ — fa”, “ — offload-to-cpu”, “-o”, str(out),
], check=True, capture_output=True)
return str(out)

@mcp.tool()
def upload_image_to_klaviyo(path: str, name: str) -> str:
“””Upload a generated image to the Klaviyo image library; returns its hosted URL.”””
return Klaviyo().upload_image(path, name)

if __name__ == “__main__”:
mcp.run()

6.4 A draft-only Klaviyo client

Listing 3 is the REST client the tools and the scripted pipeline share. Two design choices encode governance in code. It honours Retry-After on HTTP 429 instead of retrying blindly. And it has no method that schedules or sends, so the agent physically cannot go past a draft.

Listing 3. klaviyo_client.py (abridged): read, upload and draft, never send

BASE, REVISION = “https://a.klaviyo.com/api", “2026–07–15”

class Klaviyo:
def __init__(self, api_key=None):
self.s = requests.Session()
self.s.headers.update({
“Authorization”: f”Klaviyo-API-Key {api_key or os.environ[‘KLAVIYO_API_KEY’]}”,
“revision”: REVISION, “accept”: “application/vnd.api+json”})

def _call(self, method, path, max_retries=5, **kw):
for _ in range(max_retries):
r = self.s.request(method, f”{BASE}/{path}”, timeout=60, **kw)
if r.status_code == 429: # fixed-window limit: wait as told
time.sleep(int(r.headers.get(“Retry-After”, “1”)))
continue
r.raise_for_status()
return r.json() if r.content else {}
raise RuntimeError(f”rate-limited on {path}”)

def upload_image(self, path, name): # scope images:write, max 5 MB
with open(path, “rb”) as f:
d = self._call(“POST”, “image-upload”,
files={“file”: (os.path.basename(path), f, “image/png”)},
data={“name”: name})
return d[“data”][“attributes”][“image_url”]

def create_draft_campaign(self, name, segment_id, subject, preview, from_email, from_label):
body = {“data”: {“type”: “campaign”, “attributes”: {
“name”: name,
“audiences”: {“included”: [segment_id], “excluded”: []},
“send_options”: {“use_smart_sending”: True}, # limits over-messaging
“campaign-messages”: {“data”: [{“type”: “campaign-message”, “attributes”: {
“definition”: {“channel”: “email”, “label”: name, “content”: {
“subject”: subject, “preview_text”: preview,
“from_email”: from_email, “from_label”: from_label}}}}]}}}}
d = self._call(“POST”, “campaigns”, json=body)[“data”]
return d[“id”], d[“relationships”][“campaign-messages”][“data”][0][“id”]

def assign_template(self, message_id, template_id):
self._call(“POST”, “campaign-message-assign-template”, json={“data”: {
“type”: “campaign-message”, “id”: message_id,
“relationships”: {“template”: {“data”: {“type”: “template”, “id”: template_id}}}}})

6.5 The scripted pipeline

Listing 4 chains the steps for a win-back campaign without an LLM in the loop. It is the deterministic baseline that an autonomous agent should match, and a fallback when the agent is unavailable.

Listing 4. winback_pipeline.py (core): perceive, create, assemble, hand off

k = Klaviyo()

# 1. Perceive: which past campaigns converted best?
top = sorted(k.campaign_values(args.metric),
key=lambda r: r[“statistics”].get(“conversion_rate”, 0), reverse=True)[:3]

# 2. Create: a hero image whose headline text Qwen-Image-2.1 renders in the picture
headline = “We miss you — 15% off”
hero = generate_campaign_image(
“Minimal e-commerce email hero banner, soft pastel studio background, a gift box, “
f”bold clean sans-serif headline text ‘{headline}’, professional product photography”)
hero_url = k.upload_image(hero, “winback-hero”)

# 3. Assemble a DRAFT: template + campaign + assignment; nothing is scheduled
tpl = k.create_template(“Win-back (agent draft)”, HTML.format(hero=hero_url, …))
campaign_id, message_id = k.create_draft_campaign(
“Win-back — agent draft”, args.segment, subject=headline,
preview=”A little something to welcome you back”,
from_email=args.from_email, from_label=”The Shop”)
k.assign_template(message_id, tpl)

# 4. Hand off: a person checks audience, copy, image and consent, then schedules
print(f”Draft ready for review: campaign {campaign_id}”)

6.6 Operator interface

For marketers who want images without an agent, Listing 5 is an interactive shell loop: type a prompt, choose how many images, and each opens in Preview when done.

Listing 5. generate.sh (core loop)

while true; do
read -r -p “Prompt> “ PROMPT || break
[ -z “$PROMPT” ] && break
read -r -p “How many images? [1] “ N
for ((i = 1; i <= ${N:-1}; i++)); do
OUT=”outputs/$(date +%Y%m%d-%H%M%S)-$i.png”
 ./sdcpp/sd-cli — diffusion-model models/diffusion_models/qwen_image_2.1-Q6_K.gguf \
 — vae models/vae/qwen_image_2.1_vae_bf16.safetensors \
 — llm models/text_encoders/Qwen3VL-8B-Instruct-Q4_K_M.gguf \
-p “$PROMPT” — cfg-scale 6.0 — sampling-method euler — steps “$STEPS” \
 — seed -1 -W “$W” -H “$H” — fa — offload-to-cpu -o “$OUT”
[ -f “$OUT” ] && open “$OUT”
done
done

7. Illustrative Case Scenarios

Three scenarios show where the agent adds value and where the human boundary sits. Each lists the agent’s tool calls in order.

7.1 Scenario A: win-back campaign (P6)

A marketer asks: “Draft a win-back email for customers who have not ordered in 120 days, using what worked last quarter.”

1. Read the campaign values report for the last 90 days, grouped by campaign, with Placed Order as the conversion metric.

2. Read the existing “Lapsed 120 days” segment and its size; the agent proposes a definition if none exists, since MCP cannot create segments.

3. Reason that discount-led subject lines converted best, and write three subject-line variants.

4. Create a hero banner with Qwen-Image-2.1 whose headline reads exactly “We miss you — 15% off”, then upload it.

5. Draft the template and campaign, attach the segment and turn on Smart Sending.

6. Stop and return a review link. The marketer checks copy, image text, audience and consent, then schedules.

The agent does in minutes what took an afternoon, but every irreversible step stays with a person.

7.2 Scenario B: abandoned-cart flow audit (P1)

The agent reads the abandoned-cart flow’s per-message performance and finds that message 3 converts at a fraction of message 1 while adding unsubscribes. Because flows are read-only through MCP, it cannot edit the flow; it writes a recommendation with the evidence. This pattern is low-risk and immediately useful.

7.3 Scenario C: deliverability anomaly (P1 → P6)

A scheduled agent checks daily metrics and notices open rates at one mailbox provider falling across three sends. It pauses nothing, since it has no authority to, but alerts the team with the affected campaigns and a hypothesis (list hygiene after a large import). A person decides whether to suppress unengaged profiles. The scenario shows agents as monitors, where their value is attention, not action.

8. Discussion

The central finding for RQ3 is that governing an agent in Klaviyo is mainly about authority, not accuracy. The platform already separates reading, drafting and sending; organisations should build their agent boundaries on that separation.

8.1 Benefits

• Speed. Analysis and campaign assembly move from hours to minutes.

• Coverage. Agents can watch every flow and metric daily, which small teams cannot.

• Extensibility. Through MCP, an organisation can add its own tools (a local image model, a pricing service) beside Klaviyo’s without waiting for the vendor.

8.2 Risks

8.3 Design principles

1. Draft, then approve. Agents may read and draft freely; scheduling, sending and consent changes need a named person.

2. Least authority in code. Remove send capability from the agent’s tools (Listing 3) rather than relying on instructions.

3. Data is not instruction. Content from profiles, events and reviews must never change the agent’s goal.

4. Everything logged. Every tool call, its inputs and its result are kept for audit.

5. Local where sensitive. Run creative models such as Qwen-Image-2.1 on hardware the organisation controls when prompts or products are confidential.

9. Limitations and Future Research

This study is conceptual and illustrative. We did not run field experiments, so we make no claims about revenue, engagement or labour savings. Klaviyo’s agent surface also changes quickly: tools documented in mid-2026 may differ within months, and send capabilities were still rolling out at the time of writing. Our image-generation timings come from one consumer laptop and one quantisation level.

We see five directions for future work:

1. Field evaluation. Controlled comparisons of agent-drafted and human-drafted campaigns on conversion, unsubscribes and spam complaints.

2. Attribution. Methods to credit outcomes to agent decisions when agents, humans and Klaviyo’s native agents all touch the same campaign.

3. Multi-agent interaction. How external MCP agents and first-party agents such as Composer coordinate, or conflict, on shared profiles.

4. Security. Benchmarks for prompt injection through customer data in CRM settings.

5. Creative quality. Measuring text-rendering accuracy and brand consistency of open image models such as Qwen-Image-2.1 in real campaigns.

10. Conclusion

AI agents can already work inside Klaviyo in useful ways: they analyse performance, draft campaigns and, with locally hosted models like Qwen-Image-2.1, produce the creative those campaigns need. The Model Context Protocol lets organisations extend Klaviyo with their own tools without waiting for the vendor. The binding constraint is not what agents can do but what they should be allowed to do alone. A draft-then-approve boundary, enforced in code rather than in prompts, lets organisations take the speed of agents while keeping people accountable for what reaches customers.

References

• Anthropic (2024). Introducing the Model Context Protocol. anthropic.com

• Davenport, T., Guha, A., Grewal, D., & Bressgott, T. (2020). How artificial intelligence will change the future of marketing. Journal of the Academy of Marketing Science, 48(1), 24–42.

• Everboost (2026). Klaviyo MCP Server: What It Can and Can’t Do. A 2026 Field Guide. everboost.co.uk

• Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004). Design science in information systems research. MIS Quarterly, 28(1), 75–105.

• Klaviyo (2025). Klaviyo MCP Server: AI-powered analytics insights and marketing actions with customer data. 25 September 2025. klaviyo.com

• Klaviyo (2026). Klaviyo launches AI agents that work together to drive revenue for consumer brands. 30 June 2026. klaviyo.com

• Klaviyo Developers (2026). Klaviyo MCP server. developers.klaviyo.com

• Klaviyo Developers (2026). Rate limits and error handling. developers.klaviyo.com

• Klaviyo Developers (2026). API reference, revision 2026–07–15: Create Campaign, Assign Template to Campaign Message, Upload Image From File, Query Campaign Values.

• leejet (2026). stable-diffusion.cpp: Qwen Image 2.1 documentation. github.com

• Magnet Monster (2026). Klaviyo MCP Server (2026): Setup, use cases and what it actually does. magnetmonster.com

• Qwen Team (2026). Qwen-Image-2.1. huggingface.co/Qwen/Qwen-Image-2.1

• Schick, T., Dwivedi-Yu, J., Dessì, R., et al. (2023). Toolformer: Language models can teach themselves to use tools. NeurIPS 2023.

• Wang, L., Ma, C., Feng, X., et al. (2024). A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6).

• Yao, S., Zhao, J., Yu, D., et al. (2023). ReAct: Synergizing reasoning and acting in language models. ICLR 2023.

Abstract

We propose a quantitative modelling framework that routes unstructured market information through a calibrated decision model and into a market-making and risk engine organised on the principles Jane Street is known for. The decision model is our own Jev-like System One model, built to the design of Jev, which TypeSafe AI released on 15 September 2026. A model of this kind answers typed questions with probability distributions over options the caller defines, rather than producing text. Jev is trained with a method its developer calls Reinforcement Learning for Calibrated Decisions (RLCD), and its vendor reports latencies of 70 to 500 ms and input pricing of USD 0.042 per million tokens, with output free.

The paper has three parts. We first describe Jev’s theory, interface and early applications, keeping vendor claims separate from independent evidence. We then summarise the Jane Street model as reported in the financial press: an ETF and options market maker with USD 20.5 billion of net trading revenue in 2024, a 25-million-line OCaml codebase, committee governance, and a hiring culture built on expected value and probability games. We also cover the firm’s regulatory disputes in India and over Terraform, and the technology it has described in public: its OCaml compiler branch, deterministic state-machine replication, incremental computation, FPGA design in OCaml, its machine-learning practice, a study of sequence weighting in language-model training, and the scoring rules of its two forecasting competitions. Finally we specify a model we call the Calibrated Decision Market-Making framework (CDMM). In CDMM, the decision model’s probabilities adjust fair value and spreads for event-driven quoting, feed a fractional-Kelly position sizer, and drive a compliance gate designed around the conduct questions raised in those disputes. A second signal, computed from order-book depth without a language model, feeds the same controls. We give the mathematical specification, a reference architecture, a proposed application to Taiwan-listed ETFs and TAIFEX index options, a persistence layer that runs on PostgreSQL, YugabyteDB, CockroachDB, TiDB, Spanner or Cassandra, and an evaluation protocol. Because the framework is defined by its question schema rather than by its vendor, the decision model is one of two System One models of our own that serve Jev’s interface. S1 is a 0.5-billion-parameter language model fine-tuned on soft targets from a synthetic generator and then temperature-calibrated. S1-mini, the app’s default, is a multilingual MiniLM encoder with a trained scoring head that runs in JavaScript inside the app. TypeSafe’s Jev can be used in their place through the same interface. We test both on held-out synthetic data and run their answers through the CDMM pipeline. On a ladder of S1-mini heads we also repeat the sequence-weighting study at small scale. It reproduces the rise and fall of the effective weight exponent with model size, and the shift of its peak to smaller models with more epochs. The damped weights the study proposes fail wherever they change the weights much. On real events from free public sources, 360 US press releases and 931 crypto headlines from 2024 to 2026, S1-mini showed no skill over climatology. Recalibration kept its overconfidence from reaching the desk. No trading backtest is reported here; the market test of Section 7.9 scores forecasts, not a strategy. Every performance figure attributed to Jev comes from its vendor or early users and has not been independently replicated.

Keywords: calibrated classification, System One models, market making, ETF arbitrage, probability calibration, Kelly criterion, algorithmic trading compliance, OCaml, type safety

1. Introduction

Most applications of large language models in finance treat the model as an analyst that reads and writes. It summarises filings, drafts research, or answers questions in prose. A trading system cannot act on prose directly. Somewhere between the model and the order book a program has to turn text into a number, and that step is usually a parser sitting on top of output the model was never trained to calibrate. When a chat model says it is “fairly confident”, nothing guarantees that this confidence corresponds to any frequency of being right.

Jev takes a different approach. Its developer, Diogo Almeida, previously worked at OpenAI and was among the researchers who developed RLHF. He argues that optimising models for human language makes them poorly suited to machine consumers (TechCrunch, 2026). Jev returns a probability for every option the caller has declared and cannot return anything outside that set. For a trading system this is the right contract, because the downstream code already works in probabilities and expected values.

Jane Street is useful as the second reference point because its public profile is built around the same idea. The firm hires for probabilistic reasoning, quotes prices as expected values under uncertainty, writes its systems in a strictly typed functional language, and runs risk through committees rather than a single chief executive (Anue, 2026). Its legal disputes also show where a probability-driven trading operation meets the limits of market conduct rules.

The paper asks three questions. What is Jev, and what evidence exists for its claims? Which elements of the Jane Street approach can be formalised and reproduced by a smaller firm? And how can the two be combined into a quantitative model that is fast enough to use, whose probabilities can be audited, and which is compliant by construction?

Our contributions are a structured review of Jev based on primary documentation, a synthesis of the Jane Street operating model and its published technology, the CDMM specification with its evaluation protocol, and S1 and S1-mini, our own Jev-like models with Jev’s interface. They are the decision model the framework runs on, so every part of it can be tested end to end without a vendor. Section 2 covers Jev’s theory, Section 3 its implementation and Section 4 its applications. Section 5 covers Jane Street, and Section 6 presents CDMM and its decision models, S1 and S1-mini. Section 7 sets out evaluation, simulation results, the tests of S1 and S1-mini, and the limitations, and Section 8 concludes.

2. Jev: theory

2.1 System One models

TypeSafe calls Jev a “System One” model, a term borrowed from Kahneman’s distinction between fast intuitive judgement and slow deliberate reasoning. The model does not reason step by step or write out a chain of thought. It maps a state to a distribution over a finite answer set in a single pass. The developer’s argument is that a large share of the model calls in production software are judgements of this kind: is this email about billing, is this command safe to run, which model should handle this request. A generative model handles such calls at the cost of a full decoding loop and a parser.

Formally, let s be the state (a string, JSON object or list of texts) and let a question q define a finite answer set A_q = {a_1, …, a_K}. Jev returns a distribution over that set.

Because the support of the output is fixed by the caller, no answer outside A_q can be returned. This is the basis of TypeSafe’s claim of a zero type-error rate. The claim is structural, so it holds by construction. It says nothing about whether the chosen option is semantically correct.

2.2 Reinforcement Learning for Calibrated Decisions

The training method is reported as Reinforcement Learning for Calibrated Decisions (RLCD). RLHF rewards text that human raters prefer, and reinforcement learning with verifiable rewards (RLVR) rewards outputs that pass a checker. RLCD rewards probabilities that match observed outcomes. TypeSafe has not published a technical paper, so the exact objective is unknown. The natural reading is a reward based on a strictly proper scoring rule, for example the negative log loss or the negative Brier score of the stated distribution against the realised outcome y.

Under a strictly proper scoring rule, the expected reward is maximised only by reporting the true conditional distribution. A model trained this way therefore has an incentive to be calibrated, meaning that among all cases where it reports probability p, the event occurs with frequency close to p.

Calibration is weaker than accuracy. A model that always reports the base rate is calibrated but useless. Jev’s value depends on also having high resolution, meaning its probabilities have to separate cases well. The standard Murphy decomposition of the Brier score, into reliability, resolution and uncertainty terms, is the right tool for judging both properties, and we use it in Section 7.

2.3 Synthetic training data

Almeida told TechCrunch that Jev was trained only on synthetic data, and called the decision to generate all of the data in-house one of the best bets of his career. Synthetic data supplies labelled outcomes at scale, which a calibration objective needs. It also carries a risk that matters for finance. Calibration measured on the synthetic distribution may not carry over to a real distribution that has shifted, such as market news during a regime change. We therefore treat Jev’s calibration as a property to be measured on the user’s own data, never as something to assume.

2.4 Confidence

For Choice and Score questions the API returns a confidence value between 0 and 1, derived from the shape of the distribution: high when mass sits on one option, low when it is spread out (TypeSafe API reference). The exact formula is not documented. A normalised-entropy form is a reasonable working assumption, and our framework uses its own entropy measure computed from the returned probabilities rather than relying on the vendor’s number.

3. Jev: implementation

3.1 Interface

Jev is served through one endpoint, POST https://api.typesafe.ai/v1/systemone, with model identifier jev-latest. The stable version at the time of writing is jev-1.13.0. A request carries a state and a map of named questions, and all questions are evaluated against the same state in one parallel pass. TypeSafe says the model samples all answers together instead of decoding tokens one at a time, which explains the latency profile (flaviocopes.com). The state and all questions together share a budget of about 64,000 tokens. The state plus the longest single question must fit within about 32,000 tokens.

Table 1

Source: TypeSafe API reference. A Choice response for a support message such as “My card was charged twice” takes the form {“billing”: 0.88, “technical”: 0.12, “sales”: 0.0} with a confidence of 0.81.

The TypeScript SDK exposes the same structure. The pattern developers are adopting is to branch on confidence and send low-confidence cases to a human or to a slower model.

const { answers } = await client.systemOne({
state: userMessage,
questions: {
department: choice(‘Which team handles this?’, {
billing: ‘Charges and refunds’,
technical: ‘Bugs and outages’
})
}
})
if (answers.department.confidence < 0.5) routeToHuman(userMessage)

Errors follow HTTP conventions (401 authentication, 422 validation, 429 rate limit, 529 overload). Jev is also available through Vercel’s AI SDK, LiteLLM and Cloudflare’s model catalogue.

3.2 Reported performance and cost

Table 2

Sources: The Register, MarkTechPost, explainx.ai, TechNice. TypeSafe ran these comparisons on its own workflow evaluations and has said it will not submit Jev to public leaderboards. On Hacker News, Almeida also agreed that a schema-valid answer can still be confidently wrong. No independent calibration study had been published as of 22 September 2026. TypeSafe came out of stealth with a USD 40 million seed round led by DCVC (The AI Insider), and it briefly lost API capacity under launch demand.

3.3 Documented weaknesses

The documentation and early reviews list several failure modes. Jev is unreliable at arithmetic and counting, at date reasoning across mixed formats, and at questions involving double negatives or multi-hop indirection. It cannot write or explain. Text can be forced out by chaining Choice questions over characters, and the documentation says plainly that this works badly and slowly. The architecture and weights are proprietary. Outside observers speculate that Jev is built on an open-weight LLM, which TypeSafe has not confirmed. For quantitative use, the arithmetic weakness means any calculation (prices, ratios, surprises against consensus) has to be done in ordinary code before the state is sent. Jev should only be asked to judge.

4. Jev: applications

In its first week, Jev was used in four kinds of task. All four are decisions made inside software, and none involves a person reading the output.

Table 3

Sources: TechNice, The Register, Vercel guide.

These deployments share a structure that carries over to finance. There is a policy with a threshold. Above the threshold the system acts automatically, and below it the case goes to a slower and more expensive path. The economics then depend on two things: how many cases clear the threshold, and how often automatic actions are wrong. Both are functions of calibration. If Jev reports 0.95 and is right 95 percent of the time at that level, the error rate of the automatic path is known before deployment, and the threshold can be set from a loss function rather than by intuition.

The model’s name makes the economic argument explicit. William Stanley Jevons observed in the nineteenth century that making coal use more efficient increased total coal consumption. Almeida expects the same effect for machine judgement. If a classification costs a fraction of a cent and returns in about 100 ms, it becomes worth making millions of small judgements that nobody would pay a frontier LLM to make. For a trading firm, those small judgements are what an information pipeline is made of: tagging each news item, each exchange announcement and each counterparty message as it arrives.

The limits are just as clear. A latency of 70 to 500 ms rules out Jev’s use in the quoting loop of a high-frequency market maker, where reaction times are measured in microseconds. Its weakness in arithmetic rules out using it to compute values. Jev fits the event-driven layer, where the task is to decide within a second what a piece of text means for a price. This is the role CDMM assigns it in Section 6.

5. Jane Street: firm, trading philosophy, and technical framework

The facts in this section come from the Anue profile of the firm (Anue, 2026) unless another source is named. They are press-reported figures and not audited disclosures.

5.1 Business

Jane Street was founded in 1999 by Tim Reynolds, Robert Granieri and Michael Jenkins, three traders from Susquehanna International Group, together with the programmer Marc Gerstein. It began in ADR arbitrage and then moved into ETF market making, which is still the core of the firm. About 3,000 people work at its offices at 250 Vesey Street in Manhattan.

Table 4

The economics are those of a liquidity provider. The firm earns a small edge on very large volume by quoting both sides and pricing ETFs, bonds and options against their underlying baskets more accurately than competitors. Its advantage in bond ETFs comes from being able to price illiquid underlying bonds faster than the market, and from being willing to hold the resulting inventory.

5.2 Operating principles

Four features of the firm matter for our framework.

The first is expected-value reasoning as the common language. Interviews use probability puzzles, betting games and expected-value calculations, and candidates are not required to know finance or programming. The firm pays four-month interns a base of USD 300,000 and hires for the ability to put a number on uncertainty and update it.

The second is typed, functional code. Jane Street has written its systems in OCaml for most of its history and its codebase exceeds 25 million lines. Static types catch a class of errors before code runs, which matters when a mistake can cost millions in seconds. The profile describes the OCaml stack as a moat, because the skills do not move easily to other firms.

The third is collective risk governance. The firm has no chief executive. Some 30 to 40 senior staff, who together hold about USD 24 billion in equity, run it through management and risk committees. Compensation depends on firm-wide profit rather than individual books, which reduces the incentive to hide risk.

The fourth is a willingness to take large positions when the model says the odds are good. On the night of the 2016 US election, a model built by Sam Bankman-Fried, then a Jane Street trader, showed Trump leading in Florida hours before CNN called it. The firm took a large short S&P 500 position that was up about USD 300 million and then, when markets rallied, down about USD 300 million, a swing of roughly USD 600 million. The episode shows that a correct probability is not enough. The mapping from event to price must also be right, and position size must allow for that mapping being wrong.

A practitioner account locates the source of that willingness in order-book data rather than news. Thomas Costello, CIO of Bedrock Digital Asset Management, argued in a LinkedIn post from around May 2026 that high-frequency trading once sought riskless matched trades, and that Jane Street instead holds directional positions for a few minutes when its short-term probability estimates call for it. In his account, those estimates come from order-book depth. That data is noisy because most orders are cancelled and replaced, which he calls “Schrödinger’s volume”. Retail flow bought through payment for order flow is steadier and so more predictable. He also connects the firm’s growth to the rise of ETF trading, which he puts at under 25% of US market volume in 2010 and as much as 43% now. These are one practitioner’s views, not evidence, and the volume figures are unsourced in the post. They still bear directly on CDMM, because they suggest that the profitable signal is in order-book state, which a language model is poorly placed to read. Section 6.11 adds such a signal, and Section 7.3 tests it.

5.3 Regulatory disputes

Table 5

For one expiry day, SEBI’s order describes a loss of USD 7.5 million on the cash and futures legs against a gain of USD 89 million on the options, for a net USD 81.5 million. We take no view on the merits of either case. Both matter here because they mark the boundary a quantitative firm must model. One is the boundary between arbitrage and manipulation when a firm trades an underlying heavily while holding derivatives on it. The other is the boundary between fast reaction to public information and trading on information that is not public.

5.4 Published technology and methods

Jane Street does not publish its trading models. None of the 411 repositories on its GitHub is a strategy or a pricing model (Section 6.15). It does publish a good deal about the technology around them, through its engineering blog, its podcast Signals and Threads, open-source code and two public forecasting competitions. This section collects what those primary sources say. The firm describes itself as a liquidity provider trading on more than 200 venues, in which “traders and researchers … build models, strategies, and systems that price and trade a variety of financial instruments”, and in which “the lines between trading, research, and technology are intentionally porous” (Jane Street, What we do).

Language and compiler. Beyond using OCaml (Section 5.2), the firm maintains its own branch of the compiler. OxCaml adds modes, which describe how a value may be used as well as what it is. These allow values to be allocated on the stack where the compiler can prove they do not escape, and they support data-race-free parallelism and a uniqueness mode against use-after-free. OxCaml also adds unboxed types and layouts, so floats, 64-bit integers and SIMD vectors need not be allocated on the heap. The features were used in production internally before the branch was opened in 2025 (Tarides, 2025; OxCaml documentation). The firm presents this as bringing “some of Rust’s best features for writing high performance code” into a simpler type system (Jane Street, Performance engineering). The design choice is the one Section 5.2 describes: safety is enforced by the type checker, including safety of memory use, rather than by review.

Deterministic state-machine replication. Jane Street’s client-facing trading runs on Concord, a platform built around state-machine replication. A single sequencer takes messages from a multicast group, stamps each with a timestamp and a sequence number, and re-emits them in one canonical order. Every application is a state machine that processes that stream one message at a time. A process that crashes rebuilds itself by reading “all of the messages that happened in the system from the beginning of the day in the same order”, ending up as “an exact copy of what you were when you crashed”. Receivers detect lost packets from gaps in the sequence numbers and fetch them from retransmission servers. The design gives availability, fast turnaround and an audit trail. Its successor, Aria, generalises it with shared infrastructure and topic filtering, while keeping a single canonical order across topics so that any run can be replayed (Signals and Threads, 2022a). The pattern is the one electronic exchanges use: a single-threaded matching engine acts as the sequencer, and market data goes out once over multicast. Switches copy the data to every participant at nearly the same time, and slow receivers catch up from recovery servers rather than slowing the publisher (Signals and Threads, 2020).

Incremental computation. Incremental is an open-source library for self-adjusting computation, described as “a fancy spreadsheet”. A computation is a graph of nodes, and a call to stabilize recomputes only the nodes whose inputs changed. A dynamic bind lets the shape of the graph itself depend on data. The firm names risk calculations that combine “data from a complex collection of interdependent models”, driven by market data and user configuration, as a main use. It gives the overhead as about 30 ns per node that fires, so the library pays off when nodes are expensive or when only a small part of a large graph changes (Jane Street blog, 2015). The model is due to Acar’s work on self-adjusting computation.

Hardware and latency. For the lowest-latency paths the firm uses FPGAs, “as a way of achieving performance that can’t be gotten on CPUs alone”, and builds packet-processing systems that handle “millions of multicast messages per second on a single core” (Jane Street, Performance engineering). Its FPGA designs are written in Hardcaml, an OCaml library that is a full hardware description language rather than a high-level synthesis tool. Hardcaml has native simulation, formal verification and expect-tests over ASCII waveforms, and “Jane Street uses Hardcaml in production for some of our most critical ultra-low-latency trading systems” (Hardcaml). To find latency problems at this scale the firm built magic-trace. It uses Intel Processor Trace to keep a ring buffer of all control flow and, when triggered, saves roughly the last 10 ms of execution with a timing resolution of about 30 ns and an overhead typically under 10%. The motivating case was code that must answer a network packet within about 250 µs, where “sampling profilers are approximately useless” (Jane Street blog, 2022).

Machine learning. Two episodes of Signals and Threads describe the firm’s modelling. The data are large, from “a couple of terabytes” to “tens of terabytes of market data every day” depending on the speaker, and weak in signal, with “one unit of useful data and 99 units of garbage”. The data are also non-stationary for two reasons. Regimes change, since “a financial crisis seems to occur roughly every year”, after which “the market just works differently”. And markets are anti-inductive: a pattern that is traded on disappears. The model families named are “tree based models or deep learning models or artisanal regressions”, with discipline over train and evaluation splits to prevent leakage between teams (Signals and Threads, 2025). Model design starts from the hardware. The firm has “mid thousands of very high end GPUs”, and inference budgets run from “literally 100 nanos” through microseconds to milliseconds, with market-data processing at “a couple of mics per event” for a stock that sees a million events a day. Reproducibility of training runs is treated as critical (Signals and Threads, 2024).

Sequence weighting in language-model training. A 2026 study on the firm’s blog asks how a language model’s learning depends on the weight each training sequence is given, and whether that dependence can be extrapolated from small models to large ones. To separate the effect of weight from the quality of the data, each training sequence was given a random weight drawn log-uniformly between 0.01 and 10. The authors then measured an effective sequence weight exponent p*, the power of a sequence’s weight that is closest to proportional to the model’s loss reduction on that sequence. With p* = 0 a model learns a sequence regardless of its weight, and with p* = 1 it learns in proportion to the weight. The study covers three families trained for three epochs: in-house dense models from tens of millions to just under 100 billion parameters, in-house mixture-of-experts models up to hundreds of billions, and the open-weight Qwen 2.5 family from 0.5 to 72 billion. In each, p* “often increases at small scales … and decreases at larger scales”, and “training for more epochs shifts the p* peak towards smaller model scales”. The authors read this as three regimes: small models learn “general patterns independent of data weight”, medium models learn “data-specific patterns proportional to the data weights”, and large models “are able to learn all patterns present in the data, once again independent of data weight”. Their conclusion is that “the behavior of sequence weighting at small scales does not cleanly predict behavior at the largest scales”, so data-mix results should be evaluated only on sufficiently large models. They suggest, but have not tested, training with damped weights w_train = w_target^(1/p*) to hit a target weighting (Renda and Mani, 2026). Section 6.14 implements their estimator, and Section 7.8 repeats the experiment and tests the damped weights on a ladder of S1-mini heads.

Published scoring rules. The firm has run two public forecasting competitions on anonymised market data, and each defines how a forecast is scored. In the Market Prediction competition of 2020–21, each row was a trading opportunity with a weight and a return, and the model chose whether to take it. With p_i the weighted profit on day i over the |i| days of the test period, the score was

Here t is an annualised Sharpe-like ratio of daily profit. Total profit therefore counts only in proportion to how evenly it is earned, and a strategy with t ≤ 0 scores nothing (Jane Street blog, 2020; Kaggle, 2020). The Real-Time Market Data Forecasting competition of 2024 asked for a forecast of a continuous responder, scored by the sample-weighted zero-mean R²

The baseline is a forecast of zero, not the sample mean. A forecast of zero scores exactly 0, and an over-sized forecast of a weak signal scores below it (Kaggle, 2024). Both rules encode the same stance as Section 5.2. The first pays for consistency of profit rather than for accuracy. The second pays for forecasts that are shrunk towards zero in proportion to how little the forecaster knows.

5.5 What CDMM takes from these methods

Table 6

The implementations are in src/lib/metrics.ts. They are small, but they let the framework’s forecasts be judged by the criteria the firm chose to publish, rather than only by the scoring rules of the calibration literature.

6. The Calibrated Decision Market-Making framework (CDMM)

CDMM is a quantitative model and application for an ETF and index-options liquidity provider. A Jev-like decision model turns each incoming piece of text into calibrated probabilities over a fixed set of market-relevant outcomes. The decision model is our own: S1-mini by default, or S1 (Sections 6.12 and 6.13), both built to Jev’s interface, and TypeSafe’s Jev can be used in their place. Ordinary numerical code turns those probabilities into fair-value shifts, spread widths and position limits. A compliance gate, built partly on the decision model and partly on deterministic rules, has to approve every event-driven action before it reaches the order gateway.

6.1 Design principles

Table 7

6.2 Architecture

CDMM architecture: public feeds to the quoting engine through recalibration, pricing, sizing and the compliance gate.

The quoting engine runs its own microsecond loop and never waits on the decision model. Its results arrive as parameter updates to that loop (a shifted mid, a wider spread, a lower size limit) within about 50 ms of an event for S1-mini, or 100 to 500 ms for Jev by its vendor’s figures.

6.3 The event layer

For each constituent or underlying i, an incoming item x is sent to the decision model as the state together with a fixed question set. All numerical features, such as reported revenue against consensus, are computed by the pre-processor and written into the state as text, because a language model cannot be trusted with arithmetic (Section 3.3).

Table 8

6.4 Recalibration

No model’s claim of calibration is relied on, ours included. For each question, CDMM logs the decision model’s raw probability and the realised outcome. The realised outcome is defined in code, for example the sign of the residual return of i over the stated horizon after removing index beta. A monotone map g is then fitted by isotonic regression on a rolling window.

The expected calibration error of the recalibrated output, measured out of sample over M probability bins B_m, is monitored daily and used directly in position sizing (Section 6.6).

Recalibration is applied only when it helps out of sample. Each option’s log is split in time, and maps fitted on the older half are scored on the newer half against the raw probabilities. The recalibrated probabilities are used only if their held-out ECE is lower, and the ECE of whichever version is used enters the sizing rule. A model that is already calibrated is left alone, because isotonic steps would only add noise to it. Section 7.3 shows that happening.

6.5 From probabilities to prices

Let μ_k and s_k be the historical mean and standard deviation of the residual return of i conditional on outcome class k, estimated from past events with ordinary statistics. The decision model supplies only the weights. The expected shift and the added event variance are the mean and variance of the resulting mixture.

For an ETF, the shift in the indicative value is the weighted sum of constituent shifts, ΔNAV = Σ w_i Δm_i. The quoting engine uses the Avellaneda and Stoikov (2008) reservation price and spread, with the diffusion variance raised by the event variance and the spread widened in proportion to the entropy of the distribution.

Here q is inventory, γ is risk aversion, κ is the order-arrival decay parameter and η is an ambiguity premium set by the risk committee. The entropy term means that an item the decision model cannot classify with confidence widens the market without moving it. This is how a human market maker behaves when a headline is unclear.

6.6 Position sizing

The 2016 election trade (Section 5.2) shows that a correct probability can still produce a loss when the event-to-price mapping fails. CDMM therefore sizes directional event positions with a fractional Kelly rule. The fraction shrinks as the measured calibration error grows, and again as the dispersion of the conditional returns grows.

Here c is the round-trip transaction cost and α is a penalty on calibration error set by the risk committee. With α = 10, an ECE of 0.05 halves the position and an ECE of 0.10 or more removes it. A model whose probabilities have drifted is taken out of directional trading automatically, while it can still widen spreads. The desk also caps |f| as a risk limit, 1.0 by default. On inputs like those of Section 6.9, the Kelly rule asks for several times capital, and Section 7.2 shows what happens without that limit.

6.7 Compliance gate

The two disputes in Section 5.3 correspond to two tests every event-driven action must pass. The gate splits the work between code and the decision model by the kind of judgement needed. Anything that is a quantity is checked in code. Anything that requires reading text goes to the decision model.

Table 9

The decision threshold for a binary gate follows from the cost of each error. Let L_miss be the expected cost of letting a violating order through (fines, disgorgement, suspension) and L_block the opportunity cost of stopping a legitimate one. The gate should escalate when the recalibrated probability of a violation exceeds τ.

The SEBI case involved a USD 560 million deposit and a trading suspension, so L_miss is several orders of magnitude larger than L_block, and τ comes out very low. The 0.02 in the table is a placeholder for the committee to set. The decision model is used as a screen that raises cases for review. Its output is not treated as a legal determination, and a pass from the gate is not evidence of compliance.

6.8 Typed implementation

Following Jane Street, the core is written in OCaml. Each question set is declared once as a variant type, and the JSON schema sent to the decision model is generated from that type. A new option therefore cannot be added to the schema without every pattern match in the pricing code failing to compile until it handles the new case.

type direction = Up | Down | Neutral
type horizon = Minutes | Same_day | Multi_day

type event_answer = {
relevant : float; (* Noul *)
novel : float; (* Noul *)
direction : direction dist; (* Choice, recalibrated *)
magnitude : int dist; (* Score, 5 levels *)
horizon : horizon dist;
source_public : float;
}

let mu = function Up -> mu_up | Down -> mu_down | Neutral -> 0.0

Table 10

At USD 0.042 per million input tokens, a state of 1,500 tokens costs about USD 0.000063 per call. One million items a month would cost about USD 63 in model fees, so model cost is not the constraint. The constraints are API capacity, which failed briefly at launch, and latency variance.

6.9 Application: Taiwan ETFs and TAIFEX index options

We propose the Taiwan market for the first deployment. Its large ETFs track a small set of heavily weighted constituents, and material news for those names arrives through a single public channel, the material-information filings on MOPS, the Market Observation Post System. That makes the whitelist in Section 6.7 easy to enforce. The application has three parts.

1. ETF liquidity provision, under the exchange’s liquidity-provider programme, for broad-market and high-dividend ETFs, with ΔNAV computed from the decision model’s answers on constituent-level events during trading hours.

2. Quoting of TAIFEX TAIEX options, with σ²_ev added to the implied volatility surface around scheduled events such as monthly revenue releases and earnings calls.

3. The expiry-day cross-product check in Section 6.7, applied to the firm’s own options book on weekly and monthly settlement days. This is the Taiwan equivalent of the Bank Nifty pattern.

One question has to be settled before this deployment. The decision model’s accuracy on real Traditional Chinese filings has not been measured: S1 and S1-mini were trained on synthetic Traditional Chinese and English items, and TypeSafe has not documented Jev’s accuracy on Traditional Chinese text. The recalibration layer will correct systematic bias in the probabilities, but it cannot add resolution the model lacks. A pilot on a labelled sample of MOPS filings has to establish resolution first.

6.10 Persistence layer and database support

CDMM keeps three records. The first is each event with the decision model’s raw answers and the name of the model that gave them. The second is each decision, recomputed on the server from those stored answers, together with its gate checks. The third is each realised outcome, which becomes one calibration observation per answer option. Section 6.4 depends on the third record, because recalibration needs a log of raw probabilities against outcomes. The first two exist for audit, since the disputes in Section 5.3 turned on reconstructing what a firm knew and did, and when.

The workload is small and mostly appends. Events arrive on a scale of seconds, not microseconds. Reads are of two kinds: the newest decisions, and the newest few hundred observations for one option. One write needs a correctness guarantee. An outcome must be recorded exactly once for each event and question, because a duplicate would count the same observation twice and bias the calibration map. Every key begins with a random UUID generated by the application, never a timestamp or a sequence. This follows the hot-spot guidance of the distributed systems below.

Table 11

PostgreSQL, YugabyteDB and CockroachDB share one adapter, one schema and one driver. The only behavioural difference the code needs to handle is that the two distributed systems abort contended transactions with SQLSTATE 40001 and expect the client to retry. The adapter wraps every transaction in a bounded retry with jittered backoff and runs schema changes outside transactions. TiDB needs its own adapter only because it speaks the MySQL protocol. Its semantics for this workload are the same.

The Spanner schema uses interleaving, which stores child rows physically with their parent row. An event’s decisions, outcome and calibration observations are therefore stored together, and the audit trail for one event is a single local read. The query for calibration windows goes through a secondary index on (question, option, ts DESC). That index stores the probability and outcome columns, so the query never reads the base table.

Cassandra gives the weakest guarantee of the six, and the design has to say so. It has no transactions that span partitions. The outcome is claimed with a lightweight transaction on its own partition, and the observations follow in a logged batch. A logged batch will eventually apply all of its writes or none of them, but other readers can see the outcome before its observations arrive. If the process fails between the two writes, an outcome is left without observations, and a repair job has to find and complete it. Multi-partition transactions (Accord) are announced for Cassandra 6.0. That version was still a pre-release when this was written, and the latest general-availability release was 5.0.9 (7 August 2026).

The choice of store is not a performance question at this scale. PostgreSQL is enough for a single-region desk. For a desk that spans regions and must be able to show regulators an ordered history, Spanner and CockroachDB give the strongest consistency without extra application logic. TiDB suits a firm that already runs MySQL tooling. Cassandra is reasonable only where a firm already operates it and accepts that calibration data can briefly lag its outcome.

All six adapters and an in-memory reference store run the same four contract tests. The tests round-trip an event with its model answers, check that decisions come back newest first with every value intact, and check that a second outcome for the same event is rejected. In the fourth test, five concurrent writers race to record the same outcome, and exactly one must succeed. On 22 September 2026, all 28 tests passed against single-node Docker instances.

Table 12

The first Spanner schema failed on the emulator. An interleaved child table must repeat its parent’s key column by name, so the event key was renamed to event_id in every table. These tests show that the adapters are correct against each store’s interface and its single-node transaction semantics. They do not test network partitions, multi-region latency or node failure. The TiDB container in particular stores data in memory and has no TiKV cluster behind it, and the Spanner emulator does not reproduce production latency or aborts. Evaluating those failure modes needs multi-node clusters and fault injection, which is left to the deployment stage in Section 7.

6.11 An order-book signal

CDMM accepts a second source of direction probabilities, computed from limit-order-book depth, and passes it through the same recalibration, sizing and gate as the event layer. The account in Section 5.2 places the edge of a firm like Jane Street in this data. A language model is the wrong tool for it. Order-book features are numbers that change every few milliseconds, the decision model takes 50 ms to several seconds per call, and language models are weak at arithmetic. The order-book signal therefore uses no language model.

Three features summarise the book. The micro-price of Stoikov (2018) tilts the mid towards the side with less size at the touch. Its deviation from the mid is the first feature. Let b and a be the best bid and ask, and q^b and q^a the sizes there.

Depth imbalance over the top five levels, with weights halving at each level, measures where resting size sits. Its change over the previous ten seconds is a crude proxy for order flow, in the spirit of the order-flow imbalance of Cont, Kukanov and Stoikov (2014).

A multinomial logistic regression maps the standardised features to probabilities that the next H-second return is up, down or neutral. The return counts as neutral when it lies within ±θ. The model is refitted walk-forward on the most recent 1,000 decisions. The conditional mean and standard deviation of returns in each class are estimated from the same window, and they replace the event-level μ_k and s_k of Section 6.5. From there the path is the same as for the decision model’s answers: selective recalibration, then a fractional-Kelly size reduced by the measured ECE and capped, then the gate. The output is a fair-value shift that updates the quoting loop without sitting inside it, like the Jev layer.

The implementation is in src/lib/orderbook.ts. It consists of functions that take book snapshots; it is not connected to a market-data feed. A live test needs level-2 depth snapshots at sub-second resolution for the instruments quoted. How to combine the order-book and event-layer shifts when both are present is left open. The simplest rule, adding the two shifts, would double-count news that has already moved the book.

6.12 S1: an in-house System One model

Section 6.8 defines the event layer by its question schema rather than by its vendor. S1 tests that claim. It is our own System One model, small enough to train on a desktop, and it serves Jev’s wire format, POST /v1/systemone with a state and named typed questions. The CDMM client calls it through the same code path as Jev, and nothing downstream can tell the two apart except by the model name stored with each event. S1 also gives the desk a fallback that does not depend on a third party, which matters for the fail-closed gate of Section 6.7.

Architecture. S1 is Qwen2.5–0.5B with a rank-16 LoRA adapter on every attention and MLP projection, about 8.8 million trainable parameters. Each question is rendered as a prompt that shows the state, the instruction and the caller’s options labelled A, B, C and so on, and ends with Answer:. The answer distribution is the model’s next-token distribution restricted to those letter tokens and renormalised. One forward pass therefore scores every option, and, as with Jev, no answer outside the declared set can come back. Noul questions are two-option questions whose first option is “true”. Score questions list their levels in order, and the returned score is the expected level. Confidence is the normalised-entropy form of Section 2.4. S1 accepts at most 26 options and 2,048 tokens per prompt. It refuses a longer request with status 422 rather than truncating it, since truncation would cut off the Answer: line.

Training objective. RLCD is unpublished (Section 2.2), so S1 uses its supervised counterpart. The synthetic generator knows each example’s true distribution t over the options, and the loss is the soft cross-entropy

Its expectation over sampled outcomes equals the expected log score, a strictly proper scoring rule. For each state, the loss is therefore minimised only by p̂ = t. Training on soft targets rather than sampled labels removes the label noise a reward estimated from outcomes would carry, at the cost of needing a generator whose true distributions are known. We make no claim that this matches RLCD.

Data. The generator (model/s1/data.py) writes revenue, earnings and guidance items for ten Taiwan-listed companies in Traditional Chinese and English, each with a source line, an optional repeat cue and optional private-source cues. It then asks the event questions of Section 6.3 about each item: direction, magnitude, horizon, novelty, public source and undisclosed information. Relevance questions ask about the named company, a sector peer or an unrelated company. Customer-message triage and urgency questions are included as a non-financial control. Direction probabilities are a softmax of the earnings surprise, the guidance change and a “mixed result” flag, and the other targets follow similar fixed rules. Choice options are shuffled, so the position of a letter carries no information. There are 12,000 training rows and 1,500 each for validation, an in-distribution test set (test_iid) and an out-of-distribution test set (test_ood). The out-of-distribution set uses two companies, one news template and one wording of every question that never appear in training.

Calibration. After training, one temperature per question type (noul, choice, score) is fitted on the validation split by minimising the same soft cross-entropy. Temperature scaling changes confidence without changing which option ranks first (Guo et al., 2017). It is a global correction learned on synthetic data. The per-desk isotonic recalibration of Section 6.4 still runs on top of it once S1’s answers have been logged against real outcomes.

Serving. model/s1/server.py runs the HTTP endpoint with an optional bearer key, the Jev status codes (400, 401, 413, 422) and a health check. In the app, setting S1_ENDPOINT routes /api/evaluate to S1. The event is stored with answer source s1 and S1’s model name. The calibration window of Section 6.4 is not yet keyed by model, however, so outcomes from S1, Jev and the mock would be pooled into one isotonic map. Until the outcome tables carry the model, a deployment should keep one answer source per store.

6.13 S1-mini: a System One model in JavaScript

S1 needs Python, PyTorch and a 1 GB base model, so it cannot run inside the serverless functions that host the CDMM app. Section 7.5 also shows it takes several seconds per event on a CPU. S1-mini is a second in-house model built to remove both limits. It runs inside the Next.js server through transformers.js, the JavaScript port of Hugging Face Transformers, which executes ONNX models with ONNX Runtime.

Architecture. A frozen multilingual MiniLM sentence encoder (paraphrase-multilingual-MiniLM-L12-v2, 384 dimensions, 8-bit quantised, 118 MB) embeds the state s, the question q and the text of each option o_k. The encoder is multilingual because half the news is in Traditional Chinese. A small multilayer perceptron f, with layers of 1,924, 256, 64 and 1 units, scores each option from those embeddings, their element-wise products, a one-hot question type and the option’s position among ordered score levels. The answer distribution is a softmax over the declared options with one temperature T per question type:

As with Jev and S1, the support is fixed by the caller and nothing outside it can come back. Unlike S1, the options are scored independently, so the number of options is not limited by a letter vocabulary. The answers come back in Jev’s shapes, and the desk labels them “S1-mini, our simulated Jev in JavaScript”.

Training. The head is trained with S1’s soft cross-entropy on S1’s synthetic data (Section 6.12), and the temperatures are fitted on the same validation split. The training embeddings are computed with the same transformers.js code that serves requests (scripts/s1mini-embed.ts), so training and serving see identical features. The head is then trained in PyTorch (model/s1/mini.py) and exported as a 2.7 MB weight file. A unit test checks that the TypeScript forward pass reproduces PyTorch’s logit on a reference input. Embedding the 3,526 distinct texts took 40 seconds and training for 200 epochs took about three minutes on the same desktop CPU; the epoch count was chosen by validation loss. The fitted temperatures were 1.00 for noul questions, 1.05 for choice and 1.16 for score questions.

Deployment. The build step downloads the encoder, and the deployment ships it with only the Linux x64 ONNX Runtime binary inside the /api/evaluate function, 218 MB of the 250 MB a Vercel function may use. When neither S1_ENDPOINT nor JEV_API_KEY is set, S1-mini is the app’s default answer source, so the hosted desk answers with a trained model rather than the keyword mock. Section 7.6 compares it with S1.

6.14 Sequence weighting on the S1-mini ladder

Section 5.4 summarises Jane Street’s study of how a language model’s learning depends on the weight given to each training sequence (Renda and Mani, 2026). S1 and S1-mini are trained on a mix of question families, and any future reweighting of that mix, towards the event questions that move prices or towards recent regimes, is a sequence-weighting decision. We therefore implemented the study’s estimator and ran its experiment at the scale S1-mini can reach on a desktop.

Estimator. Each training example i has a weight w_i and, after training, an in-sample loss reduction g_i against a baseline that does not depend on the weights. The effective sequence weight exponent p* is the power of the weight that best explains how the loss reduction is shared out across examples. We use the kernel estimator from the study’s technical note. With the shares

and r_i the empirical percentile rank of w_i, the estimate is

This is a Cramér-von Mises-style distance between the cumulative share of loss reduction and the cumulative share of w^p, both taken in weight order. Because min(r_i, r_j) is the integral over u of 1{u < r_i}·1{u < r_j}, the quadratic form equals the integral of the squared tail sum of d above each percentile. It can therefore be computed in O(N log N) after one sort, rather than with an N × N matrix. We minimise over p in [0, 4] by a grid search followed by a golden-section search. The note’s alternative, nonlinear least squares of g_i on c·w_i^p, is reported alongside. It fits the conditional mean point by point, so a few noisy examples with large weights can move it.

Experiment. The data are S1’s 12,000 synthetic training rows (Section 6.12). The study trained on text, where every token is a sample and part of each sequence can only be learned by memorising it. To match that, each row keeps one answer sampled from its true distribution rather than the distribution itself, and the model is trained on the log loss of that answer. Each row is given a weight drawn log-uniformly between 0.01 and 10, as in the study, fresh for each seed. The ladder is thirteen two-layer heads on S1-mini’s frozen MiniLM features, with a hidden width from 1 to 4,096, or about 1,900 to 7.9 million parameters. Each head is trained with AdamW at a learning rate of 10⁻³, with no weight decay and batches of 256, on the weighted loss with the weights rescaled to a mean of 1. After 1, 2, 3, 4, 6, 8, 12, 16, 24, 32, 48, 64, 96, 128, 192 and 256 epochs, the in-sample loss reduction is measured against the uniform answer, g_i = log K_i − NLL_i, where K_i is the number of options. p* is then estimated from (w_i, g_i). Each cell is averaged over three seeds, which change the weights, the sampled answers and the initialisation.

Damped weights. The study proposes, without testing it, that if loss reduction scales as w^p*, training with w_target^(1/p*) should give loss reductions in proportion to w_target. We test this on every rung whose mean p* at 64 or 128 epochs is at least 0.25; below that the damped weights would span more than twelve orders of magnitude, against three for the targets. The rung is retrained for the same number of epochs with fresh target weights damped by its own p*, and p* is re-estimated against the target weights. The proposal works if that estimate is close to 1. Since the damped weights are a monotone power of the targets, their ranks are the same and the estimate against the targets is the estimate against the training weights divided by the p* used.

Tools. model/s1/pstar.py implements the estimators and model/s1/weighting.py runs the ladder and the damping test, writing model/results/weighting.json. src/lib/weighting.ts is a TypeScript port of the estimators. A shared fixture checks it against the Python version to six decimal places. The site’s Weighting lab (/weighting) shows the ladder’s p* as a heatmap of width against epochs. It also estimates p* for a pasted or simulated set of (w, g) pairs, draws the cumulative shares the estimator compares, and converts a target weighting into damped training weights next to the ladder’s test of that conversion.

6.15 Jane Street’s open-source code in CDMM

We scanned all 411 public repositories at github.com/janestreet on 24 September 2026. None is a trading strategy, a pricing model or a signal. What the firm publishes is the machinery around its trading, almost all in OCaml: its standard library and Async runtime, the OxCaml compiler branch, Hardcaml, magic-trace, Bonsai for web interfaces, and several hundred smaller libraries and syntax extensions. Four of those libraries encode a technique that a trading desk uses directly. We ported the core of each to TypeScript (src/lib/janestreet/) under its MIT licence and used it in CDMM. Hardcaml and magic-trace need an FPGA and Intel Processor Trace, and the OxCaml compiler and the OCaml infrastructure have no counterpart to port, so they are described in Section 5.4 but not used.

Incremental. Section 5.5 left self-adjusting computation unimplemented, on the grounds that a single event is cheap to recompute but a multi-instrument desk would need it. The port keeps Incremental’s core semantics. Variables are set by the caller. Nodes are functions of other nodes, and only nodes that an observer depends on are computed. stabilize visits changed nodes from a recompute heap in increasing order of height, so each node runs at most once and after its inputs. A node whose new value passes its cutoff does not propagate, and bind lets the graph change shape with the data. src/lib/desk-graph.ts builds the desk on it. The class statistics, quoting parameters and sizing parameters are shared variables, and each instrument has an event, a mid and an inventory. The nodes are each instrument’s Δm and σ_ev, its quote, the quote on ticks and its Kelly fraction, plus the desk’s gross exposure and largest position. They call the same pricing and sizing functions as the single-event pipeline, and a test checks that they give the same numbers. On a desk of eight Taiwan ETFs, a full recompute runs 42 node functions. An event on one ETF runs 7, or 5 when neither desk total changes, a new mid runs 3, and a new λ runs one Kelly node per ETF plus whatever it moves. A Kelly fraction pinned at the position cap or at zero does not change, so the cutoff stops propagation there.

Exact decimals and tick rounding. Exchange prices live on a decimal grid that binary floats cannot represent: 0.05 is not a float. After Jane Street’s bigdecimal, a value is a BigInt mantissa times a power of ten, with exact addition, multiplication, comparison and rounding in the five directions bigdecimal offers (down, up, nearest, towards zero and banker’s). The pipeline now puts every quote on the Taiwan Stock Exchange’s ETF grid, a tick of 0.01 below TWD 50 and 0.05 from 50. The bid is rounded down and the ask up. Rounding can therefore only widen the spread the model asked for, never narrow it, the same rule hardcaml_fixed_point makes explicit by requiring a rounding mode whenever precision is dropped. The event desk shows both the model’s quote and the quote on ticks.

Splittable random numbers. splittable_random implements Steele, Lea and Flood’s SplitMix: a generator that can split off an independent generator as well as produce values. Each component of a simulation then has its own stream, and adding or removing one component leaves every other component’s draws unchanged. The port uses 64-bit BigInt arithmetic and reproduces the original’s recorded test bit for bit: over seeds 1 to 1,000 with 1,000 draws each, the only seed whose coin flips are unbalanced by more than 100 is 573, at 551 to 449, as in Jane Street’s expect test. The demonstration desk draws each ETF’s news from its own split stream. The existing simulations keep their generator, so that their published results stay reproducible.

Benchmarks. core_bench times a function in batches whose size grows geometrically and fits batch time on batch size by least squares. The slope is the cost of one run, the intercept absorbs the fixed cost of timing, and a bootstrap over the batches gives an interval. With that method (scripts/bench.ts, docs/bench-results.md) on one desktop CPU, one event on the eight-ETF desk costs 1.49 µs incrementally and 5.72 µs recomputed from scratch. Putting a quote on ticks with exact decimals costs 0.33 µs, an isotonic fit to 500 outcomes 50 µs, and a p* estimate on 1,000 examples 10 ms. All of these are small next to the model’s own 50 ms per event (Section 7.6). What Incremental saves grows with the number of instruments an event does not touch, and it matters more for the quoting loop, which reprices on every market-data update, than for the event layer.

The site’s Jane Street page (/janestreet) lists the scan, runs the eight-ETF desk live and shows how many node functions each change runs. It also runs the benchmarks in the browser.

7. Evaluation design, risks, and limitations

7.1 Protocol

Evaluation runs in three stages, and each stage must pass before the next begins. The first stage measures whether the decision model’s probabilities are informative and calibrated on real financial text. The second measures whether they improve quoting. The third runs the gate against historical cases.

Table 13

Look-ahead bias is the main threat in stage 1. A model may know how events before its training cut-off turned out. S1’s base model, Qwen2.5, was trained on text up to 2024, and Jev on data of unknown date coverage. Stage 1 therefore uses only items published after the cut-off of the model under test: after 2024 for S1 and S1-mini, and after 15 September 2026, its release date, for Jev. The first test can be run only once enough post-release events have accumulated.

7.2 Simulation study

On synthetic events with a known ground truth, recalibration corrected the model’s confidence in every setting but could not supply signal the model lacked. At desk settings, the position cap rather than the quality of the forecast determined returns. Without the cap, Kelly sizing turned overconfidence into leverage. The simulation tests the mechanics of Sections 6.4 to 6.7 before any real data exists. It says nothing about S1, S1-mini or Jev, or about any market.

The design is as follows. Each event has true probabilities over up, down and neutral. The up and down logits are drawn with standard deviation s, called the signal strength, and the neutral logit is fixed at 0.3. The realised class is drawn from those probabilities. The realised ETF return is drawn from that class’s conditional mean and standard deviation, the same values the pricing layer uses. The simulated Jev reports a distorted version of the truth:

The sharpness k makes the model overconfident when above 1, and the noise term removes some resolution. In the tables, s is the signal strength, k the sharpness, and “recal.” the recalibrated model. Three percent of events arrive from a private source. Each run has 6,000 events. After a warm-up of 1,000, the isotonic maps are refitted every 100 events on the last 1,000 resolved outcomes. The ECE that enters the Kelly haircut is measured out of sample, using maps fitted on the previous window. Every forecaster sees the same events and only past outcomes. The results below are means over five seeds. The code is src/lib/simulation.ts, and npm run simulate:grid reproduces every number.

Table 14

Recalibration lowered the Brier score in 43 of 45 runs. The two exceptions came from the one setting where the raw model was already nearly calibrated. It brought out-of-sample ECE to between 0.011 and 0.017 in every setting, from 0.024 to 0.125. The comparison with no information matters more. Raw probabilities scored worse than simply predicting class frequencies in 30 of 45 runs, so an overconfident model can be worse than having no model. Recalibration removed that loss at signal 0.4 and above. At signal 0.2 the recalibrated model still scored slightly worse than no information (0.661 to 0.664, against 0.660), because the noise had destroyed most of the resolution. This is the limit stated in Section 6.9: a monotone map fixes confidence but cannot create information.

Table 15

The table above uses the desk settings: λ = 0.25, α = 10, and |f| capped at 1, with returns net of a 6 bp cost per unit traded (± is the standard deviation across seeds). With these settings Kelly asks for positions of 3 to 8 times capital on average, so the cap binds on most trades and position size is close to fixed. Where both forecasters trade, their mean returns are within one standard deviation of each other, so the simulation gives no evidence that recalibration raises returns under a binding cap. It does show that the ECE haircut works as intended. At sharpness 2.5 the raw model’s measured ECE of about 0.1 removed it from trading entirely at signal 1.0, and cut its trading to 13% at signal 0.4. The recalibrated model kept trading and earned 16.1 and 4.7 bp per event, against 17.0 and 5.4 bp for the true probabilities. At signal 0.2 no forecaster earned much more than its costs, the true probabilities included.

Table 16

Table 17

The last two tables remove the cap and the ECE haircut to isolate the sizing rule, at signal 0.4. Log growth is the mean of log(1 + return) per event, the quantity the Kelly criterion maximises. At quarter Kelly, overconfidence acted as hidden leverage. The raw model bet 1.3 to 2.5 times the size the true probabilities called for. Because quarter Kelly under-bets, that extra size raised log growth at sharpness 1.6 and 2.5, but maximum drawdown rose to 75% and 86%, against 45% for the true probabilities. Recalibration brought position size back to within 20% of the true-probability size, and drawdowns to between 55% and 60%. At full Kelly the true probabilities survived all five runs, with a drawdown of 93%. Every miscalibrated variant went bankrupt in most runs: raw in 13 of 15 and recalibrated in 11 of 15. Recalibration alone therefore does not make full-Kelly sizing safe on estimated probabilities, since the residual ECE and lost resolution are enough to overshoot. The desk defaults follow from this. The code limits λ to 0.5, and a position cap set from a risk budget stays in place. Even at quarter Kelly without a cap, the true probabilities lost 45% of capital at their worst point.

Three features of the simulation flatter the framework. First, the simulated returns follow exactly the conditional moments the pricing layer assumes, so the mapping from event to price is never wrong, which is the failure behind the 2016 trade in Section 5.2. Second, it has no adverse selection, no latency and no market impact. Third, the gate’s inputs are generated to be separable. It stopped all 6,462 private-source events and none of 218,538 public ones, which shows the gate logic works and says nothing about how well the decision model screens text. The signal strengths are chosen to span strong, moderate and weak cases and are not estimates of anything. The simulation supports three claims. Recalibrate before trusting a probability. Keep fractional Kelly with a risk cap. Establish real resolution first, as stage 1 of Section 7.1 requires.

7.3 Order-book simulation

In a synthetic order book, the signal of Section 6.11 captured about 40% of the return available to someone who knew the true probabilities. It lost that edge when the book was read late relative to the signal’s half-life, and when it paid more than about 1 bp per round trip. Uninformative orders that are placed and then cancelled were the largest obstacle.

In the simulator, a hidden drift d_t follows an AR(1) process with a 30-second half-life. It explains 10% of the variance of the 60-second return, which has a standard deviation of 3 bp. The drift also tilts the queue sizes of a five-level book, with lognormal size noise. At each level, with probability 0.3, a large order carrying no information appears. These “flicker” orders stand in for Costello’s Schrödinger’s volume. Every 60 seconds the desk reads the book, possibly late, forecasts the class of the next 60-second return, sizes a position with the CDMM rule (λ = 0.25, cap 1, α = 10) and holds it for 60 seconds. The baseline cost is 0.5 bp per unit of capital per round trip. Each run scores 5,000 decisions after 1,500 warm-up decisions, and every row is a mean over five seeds. npm run simulate:orderbook reproduces the table.

Table 18

“Model” is the raw logistic model, and “select” applies recalibration only when it lowers held-out ECE (Section 6.4). Returns are per 60-second decision per unit of capital, net of cost, and ± is the standard deviation across seeds.

Latency mattered only relative to the signal’s half-life. With a 30-second half-life, reading the book half a second late, about one decision-model call, cost 4% of the edge, 5 seconds cost 35%, and 30 seconds cost all of it. With a 2-second half-life and a 5-second hold, half a second late halved the edge, and 2 seconds late removed it. Speed decides who captures a fast-decaying signal. A slower signal of the kind Costello describes, held for minutes, tolerates delays of the order of a second.

Flicker orders were the largest obstacle. Removing them doubled the model’s return, from 0.140 to 0.279 bp, and brought it to 80% of the true-probability return. Doubling them cut the return by a further 16%. Costs decided whether the signal was worth anything at all. The same forecasts earned 0.489 bp per decision for a market maker who pays nothing to trade, 0.140 bp at 0.5 bp per round trip, 0.022 bp at 1 bp, and nothing at 2 bp. An edge of this size is available only to a firm that earns the spread rather than paying it, which is the position of a market maker such as Jane Street.

Recalibration worsened this model in every case. Trained on log loss, the logistic model was already well calibrated, with a held-out ECE of about 0.011. Isotonic maps on top raised ECE to between 0.022 and 0.031 and lowered returns. The selective rule, which applies the maps only when they lower held-out ECE, kept returns within 0.006 bp of the raw model in every row. In the simulated language model of Section 7.2, by contrast, recalibration was needed because the raw model was badly miscalibrated. This is why the rule of Section 6.4 now tests before it recalibrates.

The book model is deliberately crude. It has no queue dynamics, no trades, no price-level changes, and no competing traders except through latency. The drift’s effect on queue sizes is assumed, not estimated. A 0.14 bp edge per minute on one instrument is also far too small to judge a strategy by, and a real desk would run it across many instruments. What carries over is the ranking of the obstacles: noise in the book first, then trading costs, then latency relative to the signal’s half-life.

7.4 S1 against the truth

Fine-tuning on soft targets made S1 nearly as good as the generator’s own probabilities on data like its training set, and it did so without any need for temperature scaling. On held-out companies, templates and question wordings it kept only part of that gain. On one compliance question the new wording turned it from reliable to confidently wrong.

S1 was trained for 400 steps of 8 examples on a six-core desktop CPU, which took 2.4 hours. That is 3,200 examples, about a quarter of one pass over the 12,000 training rows. Validation soft cross-entropy fell from 0.872 at step 100 to 0.642, 0.591 and 0.584 at steps 200, 300 and 400, so the run had nearly converged for this budget. Three variants were scored on 1,500 in-distribution (IID) and 1,500 out-of-distribution (OOD) test rows. Zero-shot is the base model with the same prompts and no adapter. Tuned adds the adapter. Tuned + T adds the per-type temperatures. Soft cross-entropy and mean absolute error are measured against the generator’s true distributions. Brier, ECE and accuracy are measured against one outcome drawn from each true distribution, as a deployment would observe them. The floor is the entropy of the truth, the lowest soft cross-entropy any model could reach. python -m s1.evaluate reproduces every number.

Table 19

In distribution, the adapter closed 89% of the gap between the zero-shot model and the floor, cut the error against the true probabilities from 0.29 to 0.05, and brought ECE to 0.017. The fitted temperatures were 1.00 for noul questions and 1.05 for choice and score questions. Temperature scaling therefore had almost nothing to correct: it moved IID ECE from 0.017 to 0.018 and OOD ECE from 0.059 to 0.057. This is what Section 2.2 predicts for a model trained on a strictly proper scoring rule, as long as the test data come from the training distribution. It is also the pattern of Section 7.3, where the log-loss-trained order-book model needed no recalibration. Out of distribution the adapter closed only 38% of the gap. Timing is for batches of 32 questions on the CPU; Section 7.5 gives the latency of a desk event.

Table 20

The table gives tuned + T by question family, with excess CE measured over each family’s own floor. The direction question that drives pricing held up best among the market questions: its accuracy fell from 0.70 to 0.64 and its ECE stayed near 0.05. The failure is the undisclosed-information question. In distribution it is almost perfect, with an excess of 0.011 and accuracy of 0.90. The out-of-distribution set asks it with a wording never seen in training, “Does this message pass on inside information?”, in place of “Does the text refer to material information that has not been publicly disclosed?”. With that wording S1 was right on 12% of cases, with an ECE of 0.68 and a cross-entropy above the zero-shot model’s 0.92. An error against the truth of 0.70 means it answered close to the opposite of the truth, and did so with confidence. A direct check confirms the cause. We asked S1 about the first 60 of those out-of-distribution items twice, once with the held-out wording and once with the trained wording. With the held-out wording its mean P(yes) was 0.81 on public items and 0.25 on private ones, and its error against the truth was 0.68. With the trained wording, on the same texts, the figures were 0.04, 0.91 and 0.04. S1 generalised to new companies and templates but not to the new phrasing. Fine-tuning on a few fixed wordings taught it the wording as well as the concept, and a paraphrase of a compliance question inverted the answer.

Two rules follow for the desk. First, the event question set of Section 6.3 is part of the model: it must be sent exactly as it was trained and calibrated, and changing a wording means re-running this evaluation. Second, a confident answer from a compliance screen is not evidence that the screen works. Only the rolling ECE on logged outcomes (Section 6.4) shows whether it does. The gate’s fail-closed rule covers an unavailable model but not a wrong one.

7.5 S1 on the CDMM desk

The desk test sends S1’s answers through the CDMM pipeline itself. S1 kept 96% of the return available to the true probabilities in distribution and 84% to 87% on held-out companies and templates. The zero-shot model could trade only after recalibration, and then earned less than half. At the desk’s loss weights the compliance gate escalated every event, public ones included, even under the true probabilities. At an even threshold S1’s screens matched the truth on every public, private and rumour item.

The design follows Section 7.2, except that the forecasts are real model outputs rather than simulated distortions. We drew 1,600 fresh news events from the generator with a seed not used for training (model/s1/desk.py): 1,200 in distribution, then 400 using the held-out companies and news template. Each event was sent with the app’s own direction, source_public and undisclosed questions (Section 6.3). The wording and option order were exactly those of src/lib/questions.ts, and a test pins them. So this test changes the content but not the wording, unlike the out-of-distribution split of Section 7.4. The realised class of each event, and a return from that class’s conditional moments, were drawn from the generator’s true direction probabilities with five seeds. Six forecasters were compared: the true probabilities, running class frequencies, and the zero-shot and fine-tuned S1 models, each raw and after walk-forward isotonic recalibration. The maps were refitted every 100 events on the last 400 outcomes after a 400-event warm-up, and frozen at the end of the in-distribution stream, so the held-out events test calibration carried across a shift. Sizing used the desk settings (λ = 0.25, cap 1, α = 10, 6 bp cost) with the out-of-sample ECE as the haircut. The last three columns are Jane Street’s published scoring rules from Section 5.4: the zero-mean R² of the pipeline’s expected return Δm against the realised return, and the utility score over days of 20 decisions. npm run simulate:s1-desk reproduces the tables from the saved answers, and docs/s1-desk-results.md holds its full output.

Table 21

Returns are per event per unit of capital, net of cost, as a mean over five seeds ± the standard deviation across seeds. There are 800 scored events in distribution and 400 out of distribution, so u, which sums profit, is not comparable between the two splits.

Three results stand out. First, fine-tuning is what made the language model usable on the desk. The zero-shot model scored worse than no information (Brier 0.687 against 0.664), and its measured ECE of about 0.11 made the Kelly haircut keep it out of the market on four events in five. Its return was zero and its zero-mean R² negative: the same protective behaviour as the overconfident simulated model of Section 7.2. Recalibration rescued it to 15.8 bp, less than half of what the true probabilities earned. S1 needed no rescue. Raw, it earned 38.5 bp in distribution, 96% of the true-probability return, and its zero-mean R² was 0.347 against 0.385 for the truth.

Second, calibration carried across the content shift. With maps frozen before the held-out events, S1’s ECE rose only from 0.023 to 0.028, and it kept 84% of the true-probability return raw and 87% recalibrated. Recalibration made little difference to S1 in either split. Its gains and losses of about 1 bp are well inside one standard deviation across seeds. This is the pattern of Section 7.3: a model trained on a proper scoring rule arrives calibrated, so the selective rule of Section 6.4 would usually leave it alone.

Third, Jane Street’s utility score gave the same ranking as mean return. Every forecaster with an edge had t between 11 and 15, well above the cap of 6, so u reduced to six times total profit. The cap binds because these synthetic events carry more signal than real markets do. On real data, where t below 6 is the normal case, the score would add the information it was designed to add: whether profit is earned evenly. The zero-mean R² ranked the forecasters as Brier did. It also put the zero-shot model’s badly sized forecasts below the no-information baseline, which is what the metric is designed to do.

The gate was tested with every event given a whitelisted source id, so only the two content screens of Section 6.7 could stop it. Share escalated, by the provenance the generator wrote into the text:

Table 22

At the desk threshold τ = L_block / (L_block + L_miss) = 1/50 = 0.02, the gate escalated every event under every forecaster, and the zero-shot model also escalated 100%. The reason is the loss ratio, not the model. The generator gives public news a 3% probability of being non-public, and at 49:1 losses a 3% risk is worth a review. The same thing happened live. Run through the app with S1_ENDPOINT set, a MOPS earnings filing came back with P(public) = 0.95 and was escalated along with a forwarded private tip. Section 6.7 therefore needs a correction. τ must be set against the probabilities the screen actually reports on clean public news, after recalibration, or every event goes to the human desk. At τ = 0.5 S1 matched the true probabilities exactly on public, private and rumour items. It escalated 76% to 81% of grey-source items, which the truth escalates in full because their P(public) of 0.4 lies below one half. Its screens ranked perfectly, with an AUC of 1.000 for both screens in both splits. The zero-shot model’s AUCs were between 0.42 and 0.69. Section 7.4’s inverted answers do not appear here, because the app sends the trained wording. The two sections together show that the screen is reliable for new content and unreliable for new wording.

Latency is the cost. On the six-core CPU used here, S1 took about 1 s for the three desk questions of one event. In the live app, a full seven-question event took 2.8 to 4.1 s end to end. That is well outside the 100 to 500 ms budget of Section 6.2. A 0.5-billion-parameter model needs one short forward pass per question, so a single GPU should bring it inside the budget, but we have not measured that. As in Section 7.2, the events are synthetic, the returns follow the conditional moments the pricing layer assumes, and the generator’s rules are simple enough for a small model to learn. The test shows that the pipeline, the recalibration, the sizing haircut and the gate behave as specified when fed a real model’s outputs. It does not show that S1 knows anything about real news.

7.6 S1-mini against S1

S1-mini is less accurate than S1 in distribution but runs about 60 times faster, and its behaviour under a shift differed from S1’s in both directions. On the desk it kept 91% of the true-probability return in distribution and 95% on held-out companies and templates, and its compliance screens ranked sources almost perfectly. It answered a seven-question event in about 50 ms on a CPU, inside the 100 to 500 ms budget of Section 6.2.

The first table repeats the evaluation of Section 7.4 on the same test rows and the same sampled outcomes. python -m s1.mini reproduces the S1-mini rows.

Table 23

In distribution S1-mini closed 80% of the gap between the zero-shot model and the floor, against 89% for S1, with the same calibration (ECE 0.016). Out of distribution it closed only 13%, against 38%. Longer training traded one split against the other. After 60 epochs its out-of-distribution soft cross-entropy was 0.982, and after the validation-selected 200 epochs it was 1.011, so the head had begun to fit the training wordings. The weak points were the questions that depend on numbers. Direction accuracy was 0.60 in distribution against S1’s 0.70, and magnitude accuracy 0.39 against 0.47. A mean-pooled sentence embedding records that a surprise was reported, but not reliably how large it was.

The paraphrase results were the most instructive. On the reworded undisclosed-information question that inverted S1 (Section 7.4), S1-mini was right 80% of the time with an ECE of 0.08, against S1’s 12% and 0.68. The sentence encoder was trained to map paraphrases close together, and that carried over. The reworded novelty question, “Does this item tell the market something it did not already know?”, broke S1-mini instead: accuracy 0.28 and ECE 0.61, where S1 scored 0.84. Neither model is robust to rewording. Each failed on a different paraphrase, so the rule of Section 7.4 applies to both: freeze the question set with the model and re-evaluate on any change.

The desk test of Section 7.5 was repeated with S1-mini’s answers on the same 1,600 events and the same drawn outcomes (scripts/s1mini-desk.ts, then npm run simulate:s1-desk).

Table 24

On direction S1 had the better Brier score and zero-mean R² in both splits, as the first table predicts. Yet S1-mini earned more on the held-out events: 37.5 bp against 33.3 bp, a difference larger than either standard deviation across seeds. With the position cap binding on most trades (Section 7.2), return depends mainly on the sign of the expected return Δm, not its size. On the held-out events S1-mini’s Δm had the same sign as the true one on 90.0% of events against S1’s 89.0%, while its mean absolute error in Δm was larger, 40 bp against 33 bp. In distribution S1 was better on both counts: 95.0% against 89.3%, and 20 bp against 29 bp. Recalibration changed S1-mini’s returns by less than 0.3 bp. Its measured ECE of about 0.03 was already low, so the selective rule of Section 6.4 would leave it alone.

S1-mini’s content screens separated private and rumour items from public ones with an AUC of 1.000 in distribution and 0.998 out of distribution. At τ = 0.5 it escalated no public item in distribution and 1% out of distribution, and every private and rumour item. At the desk threshold τ = 0.02 it escalated 87% and 94% of public items, fewer than the true probabilities, which escalate all of them. The reason is that S1-mini put P(public) above 0.98 on some public items, where the generator’s own value is 0.97. A screen that passes events the truth would escalate is overconfident in that range, which is a further reason to set τ from recalibrated probabilities (Section 7.5).

Latency is where S1-mini changes the design. The 1,600 three-question desk events took 15 ms each. In a local production server, a full seven-question event took 52 ms once the encoder was loaded, and 1.4 s on the first call. On the deployed app on Vercel it took 30 to 80 ms of model time per event, and 0.32 to 0.39 s for the whole request from Taiwan, most of it network. S1 needed 1 s for three questions and 3 to 4 s for seven on the same CPU. S1-mini is therefore the only one of the two models that meets the event layer’s budget without a GPU, and the only one that runs inside the hosted app. The price is accuracy on numbers, and a different, unpredictable set of paraphrase failures.

7.7 Risks and limitations

Jev, the design our models follow and an optional drop-in, is proprietary and was one week old at the time of writing. Every performance number in Section 3.2 comes from the vendor or early users, the training objective is inferred and not published, and the model could change under a fixed alias such as jev-latest. A desk that uses it should pin a dated version and re-run stage 1 on every version change.

S1 and S1-mini, the decision models CDMM runs on, remove the vendor risk but carry their own. It knows only its generator’s rules, and Sections 7.4 and 7.5 measure how well it learned those rules, not whether it can read real news. Each inverted or lost its answers when one question was reworded, a different question for each model, so the question set must be frozen together with the model and re-evaluated whenever either changes. On a CPU S1 answers in seconds rather than the event layer’s budget of 100 to 500 ms; S1-mini meets the budget but reads numbers poorly, which matters most for magnitude. Its calibration windows are pooled with Jev’s and the mock’s in the store (Section 6.12), and decisions recorded with database calibration cannot yet be replayed exactly (Section 5.5).

The framework assumes that residual returns conditional on an outcome class are stable enough to estimate μ_k and s_k. In a crisis these conditional distributions shift, so the recalibration window and the Kelly penalty will respond only with a lag. An entropy-driven spread widens when the decision model is unsure, but it does not help when it is confidently wrong. Calibration errors of that kind appear only in the rolling ECE, and only after the losses.

The compliance gate reduces one class of conduct risk but does not remove it. Whether a strategy is arbitrage or manipulation is a legal question decided after the fact, as the SEBI appeal shows. Deterministic volume and delta limits can be tuned to past cases and still miss the next one. A compliance control that depends on a model service, and on a third-party API when Jev is used, also creates an availability risk, so the gate must fail closed: if the decision model is unavailable, event-driven orders are blocked while ordinary quoting continues.

The facts about Jane Street come from one secondary press source that relies on filings and reporting, not on audited disclosures. The paper uses them to describe an operating model and makes no claim about the firm’s internal systems.

7.8 Sequence weighting on the S1-mini ladder

The S1-mini ladder reproduced the shape Jane Street reported. How strongly a model follows its training weights first rose and then fell with model size, and more epochs moved the peak to smaller models. Small models learned the shared rules and largely ignored the weights. Medium models spent their extra capacity on the heavily weighted rows, at the expense of the light ones. At the largest sizes the heavy rows reached the ceiling, and p* fell again. The study’s proposed correction, training on damped weights, failed at every size where it would have mattered.

The full grid of 13 widths, 16 checkpoints and three seeds took 60 minutes on an Apple-silicon GPU, and the damping test another 40 (python -m s1.weighting; every cell is in docs/weighting-results.md). The table gives the mean p* over three seeds for seven widths. Parameters grow as about 1,900 times the width, from 1,927 at width 1 to 7.9 million at width 4,096.

Table 25

Three regimes. Widths 1 to 8, with at most about 15,000 parameters, never rose above p* = 0.20, even after 256 epochs. Their loss reduction was spread almost evenly across the weight deciles. At width 1 after 256 epochs it ran from 0.15 nats in the lightest decile to 0.27 in the heaviest, against a maximum of about 1 nat (log K, averaged over rows). Wider models moved into a regime where the weights dominated. At width 4,096 after 128 epochs p* was 0.97, and the heaviest decile had reduced its loss by 0.95 nats, close to the maximum. The lightest decile had lost ground. Its loss reduction was −0.29 nats, so the model did worse on those rows than a uniform guess. The rows share inputs and question types with heavier rows, and the model had learned the heavier rows’ sampled answers. With more training the ranking reversed. After 256 epochs the same model held the top five deciles at 0.85 to 0.98 nats, near the ceiling, and p* fell to 0.62. The fall is therefore a ceiling effect on the heavy rows, not a recovery of the light ones. The lightest decile fell further, to −0.67 nats. The study reads its third regime as models learning “all patterns … independent of data weight”. Here the heavy rows ran out of room to improve, while the light rows got worse.

The peak moves with epochs. Among the widths tested, p* peaked at width 4,096 after 96 and 128 epochs. It peaked at width 1,024 after 192 epochs (0.92) and at width 256 after 256 epochs (0.87). Up to 128 epochs p* still rose with width across the whole ladder, so a peak at a width larger than 4,096 cannot be ruled out there. This is the study’s finding that “training for more epochs shifts the p* peak towards smaller model scales”. It means an experiment on the mix at one training budget can reverse at another. At 128 epochs, a mix tuned at width 256 (p* 0.43) would see its effect double at width 4,096 (0.97). At 256 epochs the same move would cut it from 0.87 to 0.62.

Estimators and noise. Across the grid the median standard deviation of p* over seeds was 0.018, and the largest was 0.12. The least-squares estimate tracked the kernel estimate (correlation 0.95) but was lower at the peaks: 0.52 at width 4,096 and 128 epochs, against 0.97. Loss reduction here is bounded by log K and goes negative for light rows, so it is not a power of the weight. The two estimators answer different questions under that misspecification, as the technical note says. The kernel estimator matches the cumulative share of loss reduction across weight percentiles. Least squares fits the conditional mean point by point, and the ceiling on the heavy rows pulls it down.

Damped weights. The test retrained each rung whose p* was at least 0.25, using damped weights built from fresh target weights, and re-estimated p* against the targets. The proposal works if the result is near 1.

Table 26

Damping failed wherever it changed the weights much. Where p* was below 0.5, the damped weights spanned 6 to 11 orders of magnitude instead of 3. The few heaviest rows then dominated training, and in 12 of 15 runs the model did worse in sample than a uniform guess, so p* was undefined. Where it was defined, the estimate sat at the search bound of 4. Between p* = 0.57 and 0.87 the loss reduction was still far steeper than the targets asked for, with p* between 4 and 1.4 against 1. Only at width 4,096 and 128 epochs did the result come close to 1, at 1.06. There p* was already 0.97, so the damping barely changed the weights. The premise fails because p* is not a fixed property of the model. Stretching the weight range changes how the model responds to it, so the exponent measured on log-uniform weights on [0.01, 10] does not hold for weights spread over 10⁻⁷ to 1⁰⁴. The damped runs also cost total fit. At every setting their mean loss reduction was below the undamped run’s, from 7% lower at p* = 0.97 to 132% lower, a negative total, at p* = 0.27.

What this means for S1. S1 was trained for about a quarter of one epoch (Section 7.4). At every width on this ladder, p* was under 0.12 before 24 epochs. If S1’s 0.5B model behaves the same way, reweighting its question families at the current budget would barely change what it learns. Longer training would make the weights matter, first strongly and then less. Any reweighting of S1 should therefore be preceded by measuring p* at S1’s own size and budget, and followed by checking the per-decile loss for light rows that fall below the uniform baseline. The damped formula should not be used to correct for p*. These results come from synthetic data with sampled answers and heads of up to 7.9 million parameters, so they reproduce the shape of the study’s findings, not its scale.

7.9 S1-mini on real market events

S1-mini did not carry what it learned on synthetic news over to real events. Asked the desk’s direction question about 360 press releases from large US companies and 931 crypto headlines, it scored worse than climatology, the no-information forecast that gives each class its historical frequency. It was also badly overconfident. Recalibrating it on the earlier half of the events brought it back to about climatology, but not above it, and no version of the signal earned a significant return. Heads trained on the realised outcomes learned only that US and crypto events have different base rates. Weighting recent events more heavily made them worse out of sample.

Data. Every source is free and public. For US stocks the events are the Exhibit 99.1 press releases attached to 8-K filings with items 2.02 (results), 7.01 (Regulation FD) or 8.01 (other events). They come from SEC EDGAR for 20 large companies, from January 2024 to September 2026, and S1-mini reads the first 700 characters. Prices are daily closes from Nasdaq’s public quote API. The outcome is the return from the last close before the filing reached the market to the next session’s close, net of β × SPY, with β and the residual volatility σ fitted on the previous 250 sessions. This is an event-study window: it measures what the release did to the price, including the jump at the next open, so a return earned over it could not have been captured at those prices. For crypto the events are Hacker News stories with at least three points whose title names Bitcoin, Ethereum, Solana, XRP or Dogecoin, over the same period. Prices are hourly candles from Coinbase’s public API. The outcome runs from the close of the hour in which the headline appeared, the first price a reader could trade at, to 24 hours later. It is taken net of β × BTC for coins other than Bitcoin, with β and σ from the previous 30 days. Starting after publication matters, since many headlines report a move that has already happened (“Bitcoin falls below $84K”). In both markets the label is up above +0.5σ, down below −0.5σ and neutral between (src/lib/market/, scripts/market-history.ts). Of 494 press-release filings, 360 had an Exhibit 99.1 and a price history long enough to score. Of the 931 crypto headlines, 809 named Bitcoin.

Table 27

Skill is the Brier improvement over the climatology of the earlier half, and the return is a position of p(up) − p(down) in the residual, before costs. The raw model put most of its probability on one class and was wrong about as often as chance, hence an ECE of 0.43 on US releases against 0.017 on the synthetic test (Section 7.4). The desk’s recalibration step did what Section 6.4 designed it to do. It pulled the probabilities back towards the base rates and cut ECE to 0.12 and 0.05, which is why a desk must never trade a model’s raw probabilities on a new domain. It could not add information the model did not have, and the rank correlation between the signal and the residual stayed below 0.1 in every row. The US recalibrated return of 12.2 bp per event, with t = 1.33, is not significant, and it is measured over an event-study window a trader could not have used.

Sequence weighting on real outcomes. The same events, with their S1-mini embeddings, were used to repeat the p* ladder of Section 7.8 and to test the weighting a desk would actually use, which gives recent events more weight (python -m s1.market). With random log-uniform weights, every width stayed near p* = 0.05 for the first 8 to 16 epochs, while the heads learned base rates. For widths of 16 and more, p* then jumped to between 1 and the search bound of 4 within the next doubling of training, sooner for wider heads. At width 1,024 it reached 2.4 after 16 epochs; the narrowest head, of width 4, climbed only to 0.79 by 64 epochs. After that the in-sample loss reduction turned negative in total, so p* was undefined. With 1,291 noisy labels the heads went straight from ignoring the weights to memorising the heaviest rows at the expense of the rest. The fall of p* seen on synthetic data did not appear, because a label that is noise cannot be learned for every row.

For recency, heads of width 64 were trained for 16 epochs on the earlier 903 events and tested on the later 388.

Table 28

The uniformly weighted head beat pooled climatology by 3.4%, but climatology computed separately for US and crypto events beat it by 3.9%. The head had learned from the source line which market an event belonged to, and nothing about direction. Its IC was negative. Every shorter half-life was worse, and the heads followed the recency weights more than proportionally, with p* between 1.3 and 2.7. In this sample, recency weighting only concentrated the fit on fewer events.

Live test. The same code runs live on the deployed app. GET /api/market?asset=crypto|us reads the last 7 days of crypto headlines from the Cointelegraph and Decrypt feeds, or the last 14 days of 8-K releases for the 20 companies. It runs S1-mini on each and scores those whose horizon has passed against the history’s class frequencies, with results cached at the edge for 15 minutes. The Market test page (/market) shows the live answers next to the history above. The live sample is a few dozen events a week, far too few to change these conclusions.

What this means. The test measures the model the hosted desk uses by default, S1-mini, which was trained on a generator’s rules. That it has no skill on real text is the expected result, and it is the reason CDMM keeps the three-stage protocol of Section 7.1 and the fail-safe recalibration of Section 6.4. The pipeline, not the model, is what this section validates. Real events flowed from free sources through the desk’s question, labels defined in code, recalibration and scoring, and recalibration protected the desk from an overconfident model. The same harness can score TypeSafe’s Jev, or a version of S1 trained on real filings, as soon as either exists. A signal worth trading would have to beat climatology by asset, not pooled climatology, on events a trader could act on.

8. Conclusion and future work

A Jev-like System One model turns the reading of text into a typed, probabilistic function call, which is what a trading system needs from a language model. CDMM runs on our own models of this kind, S1 and S1-mini, built to Jev’s interface. Jane Street shows the operating model such a function fits into: prices as expected values, types as a safety mechanism, risk decided collectively, and position sizes that allow for the model being wrong. CDMM combines the two by restricting the decision model to judgement, keeping all arithmetic in code, recalibrating every probability against the firm’s own outcomes before use, and letting calibration error reduce risk-taking automatically. The same probabilities also drive a compliance gate. Its design follows from the two conduct disputes that have shaped Jane Street’s recent public record.

The framework is a specification, and its value depends on the stage 1 result. The simulation in Section 7.2 supports the mechanics: recalibration fixed confidence in every setting, and the ECE haircut kept a badly overconfident model out of the market. It also shows that neither can supply signal the model lacks, and that Kelly sizing on estimated probabilities needs both a fractional λ and a hard cap. The order-book simulation in Section 7.3 points the same way as the practitioner account in Section 5.2. A short-horizon order-book signal was worth something only to a trader who earned the spread, who could filter out uninformative orders, and who acted well within the signal’s half-life. A text model of this kind is better placed as a secondary layer for news than as the main source of edge. If the decision model’s probabilities on Traditional Chinese financial text show real resolution after recalibration, the rest of the pipeline is standard market-making practice. If they do not, the pipeline still works with any other calibrated classifier in its place, TypeSafe’s Jev included, because the interface in Section 6.8 is defined by the question schema and not by the vendor.

S1 and S1-mini, the decision models CDMM runs on, put that claim to the test. A 0.5-billion-parameter model, fine-tuned for 2.4 hours on a desktop CPU with a proper scoring rule, served Jev’s interface to the unchanged CDMM app. On its own generator’s data it came within 0.05 nats of the best achievable log score and arrived calibrated, with an ECE of 0.017 before any recalibration. Run through the full pipeline it kept 96% of the true-probability return in distribution and 84% to 87% after a shift in content. The zero-shot model with the same prompts earned nothing until it was recalibrated. S1-mini, a MiniLM encoder with a head trained in three minutes, runs inside the hosted app at about 50 ms per event and kept 91% to 95% of the true-probability return, trading some accuracy on numbers for a sixty-fold gain in speed. The tests also found three things the specification had not anticipated. A reworded compliance question inverted S1’s answers. The desk’s 49:1 loss ratio sends every event to review unless τ is set against the screen’s actual output on public news. And Jane Street’s published utility score saturates when a signal is as strong as a synthetic one. Jane Street’s published technology points the same way as these results. A deterministic, replayable log, incremental recomputation and a quoting loop that runs in nanoseconds all keep the language model out of the critical path and make its every input auditable.

The market test of Section 7.9 is the first contact with real data. S1-mini, trained only on synthetic news, has no skill on real US press releases or crypto headlines, and recalibration is what keeps its overconfidence out of prices and sizes. The harness is now in place to score Jev, or an S1 trained on real filings, on the same events.

The sequence-weighting ladder of Section 7.8 adds a caution for any future reweighting of S1’s training mix. How far a model follows its weights depends on its size and training budget, and it reverses at long budgets, so a mix has to be judged on the model and budget that will be deployed. The damped weights proposed in the study did not correct for this.

Future work will run the three-stage evaluation once enough post-release data exists, and compare Jev’s RLCD-trained probabilities with S1’s on the same logged outcomes. It will measure p* for S1 itself at its own training budget before weighting its question families, and it will train S1 on paraphrased question wordings, and on real filings labelled by the realised residual return. It will key the calibration log by model and snapshot the window used for each decision, and it will extend the question set to bond-ETF pricing, where Jane Street’s reported market share suggests the largest informational edge.

References

1. Avellaneda, M., and Stoikov, S. (2008). High-frequency trading in a limit order book. Quantitative Finance, 8(3), 217–224.

2. Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1–3.

3. Christiano, P., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. (2017). Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems 30.

4. Gneiting, T., and Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359–378.

5. Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. (2017). On calibration of modern neural networks. Proceedings of ICML 2017.

6. Jevons, W. S. (1865). The Coal Question. London: Macmillan.

7. Kahneman, D. (2011). Thinking, Fast and Slow. New York: Farrar, Straus and Giroux.

8. Kelly, J. L. (1956). A new interpretation of information rate. Bell System Technical Journal, 35(4), 917–926.

9. Murphy, A. H. (1973). A new vector partition of the probability score. Journal of Applied Meteorology, 12(4), 595–600.

10. Zadrozny, B., and Elkan, C. (2002). Transforming classifier scores into accurate multiclass probability estimates. Proceedings of KDD 2002.

11. Anue (2026). Jane Street profile. news.cnyes.com/news/id/6354914

12. TechNice (22 Sep 2026). Report on TypeSafe AI’s Jev, republished from TechCrunch. technice.com.tw/issues/ai/303748

13. TechCrunch (18 Sep 2026). A new kind of AI model from a ChatGPT inventor is thrilling developers. techcrunch.com

14. The Register (16 Sep 2026). TypeSafe AI debuts model for machines that plays Doom. theregister.com

15. TypeSafe AI (2026). API reference. docs.typesafe.ai/api

16. Copes, F. (2026). A deep dive into Jev, TypeSafe’s System One model. flaviocopes.com/jev

17. Vercel (2026). How to classify, route, and score with Jev and AI SDK. vercel.com/kb

18. MarkTechPost (19 Sep 2026). TypeSafe AI releases Jev: a System One model that returns typed, calibrated decisions instead of text. marktechpost.com

19. explainx.ai (2026). Jev by TypeSafe AI: System One models launch. explainx.ai

20. The AI Insider (17 Sep 2026). TypeSafe AI emerges from stealth with $40M to build machine-native AI models. theaiinsider.tech

21. PostgreSQL Global Development Group (2026). Transaction isolation. PostgreSQL documentation. postgresql.org

22. Yugabyte (2026). Isolation levels. YugabyteDB documentation. docs.yugabyte.com

23. Cockroach Labs (2026). Read Committed transactions; Transaction retry error reference. CockroachDB documentation. docs.cockroachlabs.com

24. PingCAP (2026). TiDB transaction isolation levels; TiDB pessimistic transaction mode. TiDB documentation. docs.pingcap.com

25. Google Cloud (2026). TrueTime and external consistency; Schema design best practices. Spanner documentation. docs.cloud.google.com

26. Apache Software Foundation (2026). Data manipulation (CQL); Guarantees. Apache Cassandra documentation. cassandra.apache.org

27. Cont, R., Kukanov, A., and Stoikov, S. (2014). The price impact of order book events. Journal of Financial Econometrics, 12(1), 47–88.

28. Stoikov, S. (2018). The micro-price: a high-frequency estimator of future prices. Quantitative Finance, 18(12), 1959–1966.

29. Costello, T. (2026). How does Jane Street do it?! LinkedIn post, around May 2026.

30. Jane Street (2026). What we do: overview. janestreet.com

31. Jane Street (2026). Performance engineering. janestreet.com

32. Tarides (9 Jul 2025). Introducing Jane Street’s OxCaml branch. tarides.com; OxCaml documentation, oxcaml.org

33. Signals and Threads (20 Apr 2022). State machine replication, and why you should care, with Doug Patti. signalsandthreads.com

34. Signals and Threads (23 Sep 2020). Multicast and the markets, with Brian Nigito. signalsandthreads.com

35. Minsky, Y. (18 Jul 2015). Introducing Incremental. Jane Street blog. blog.janestreet.com

36. Acar, U. A. (2005). Self-adjusting computation. PhD thesis, Carnegie Mellon University.

37. Hardcaml (2026). About. hardcaml.org

38. Hume, T. (11 Jan 2022). Magic-trace: diagnosing tricky performance issues easily with Intel Processor Trace. Jane Street blog. blog.janestreet.com

39. Signals and Threads (10 Mar 2025). Finding signal in the noise: machine learning and the markets, with In Young Cho. signalsandthreads.com

40. Signals and Threads (14 Oct 2024). The uncertain art of accelerating ML models, with Sylvain Gugger. signalsandthreads.com

41. Jane Street (24 Nov 2020). Announcing our market prediction Kaggle competition. Jane Street blog. blog.janestreet.com; Jane Street Market Prediction, kaggle.com

42. Kaggle (2024). Jane Street Real-Time Market Data Forecasting. kaggle.com

43. Renda, A., and Mani, N. (14 Sep 2026). A study of sequence weighting at scale. Jane Street blog. blog.janestreet.com