At Netflix, data and machine learning practitioners run millions of workloads each day across orchestration systems and compute engines such as Titus, Spark, Trino, and Druid. On peak days, roughly 100,000 workloads fail. When workloads fail, the failure itself is only the beginning. Someone still needs to find the useful signal in several logs, decide whether the cause sits in their code or the platform, and determine whether to retry, reconfigure, or escalate. Even a small share of difficult failures can consume engineering time, waste compute, and interrupt critical data pipelines.

Our first answer to that problem was Pensive, an auto-diagnosis and remediation system in Netflix’s data platform. Pensive collected logs from failed workloads, used regular expressions and hand-authored Java rules to extract and classify errors, then passed retry or remediation guidance to products such as Maestro. It made ~ 50% of failures self-service and, in some cases, automatic.

Rules work well for known patterns, but their coverage grows only when someone recognizes a new failure mode, understands the logs, writes a rule, and keeps it current as engines and log formats change. Over one three-month period, Pensive processed roughly seven million job-failure classifications, yet about half fell into the unknown bucket. We needed a way to expand coverage without giving up the deterministic parts of Pensive that already worked.

To solve this ambiguous and complex problem at Netflix scale, we invested in Pensive AI to add AI-assisted error extractions and focused diagnosis and remediation guidance to our data and ML practitioners. The goal was not simply to place a model in front of a large collection of logs. We needed bounded extraction, predictable evaluation inputs, responsive product behavior, and quality checks that could follow the system from local development through production. Those requirements shaped every layer of the design, which we will discuss in this blog. Pensive AI wraps AI around deterministic processes, which gives broader coverage, low latency, controlled AI costs, and easier maintenance across the software development lifecycle. Building on this foundation, we plan to explore more agentic debugging experiences, as Part 6 describes.

Early results are encouraging. AI error extraction now covers 100% of failures, including the failures that legacy Pensive classified as unknown. Failed step attempts where no one used Pensive AI averaged 321% more manual debug-log exploration actions than attempts where someone did (an observational comparison, not a causal claim).

The two paths complement each other. Error extraction improves an existing capability and is always on. It lets the user of a failed Maestro workload quickly check the parts of the logs that may point to an immediate root cause. It has low latency and low cost, and it can help in a large share of cases. On-demand debugging enables a new capability. It looks at symptoms across the logs and adds the LLM’s own knowledge and reasoning for a deeper analysis, at slightly higher latency and cost. Extraction gives a fast first look at every failure, and on-demand debugging triggered by the user goes deeper when that first look is not enough.

Part 1: Finding the useful errors

Legacy Pensive rules are precise, but each one captures a particular log shape. A simplified rule looks like this:

- name: SparkContainerOOM
logSource: GENIE_STDERR
restartable: false
regex: "Container.*is running beyond physical memory limits.*Killing container"
summary: Spark AM container runs out of memory

With Pensive AI, we wanted to find useful errors that had no corresponding rule and adapt more easily when familiar errors changed shape after a platform upgrade. Data Portal (our UI for data pipelines) surfaces these extracted errors when available and links back to the source logs, giving practitioners a focused starting point for triage. That raises a question: how can we ask a model to identify an error without asking it to repeat a long stack trace token by token?

Producing output tokens is much slower than reading input tokens. A direct transcription approach would make long errors expensive and slow. Instead, Pensive numbers the lines of each log and gives the model a sequence of {lineNumber, text} records. The model identifies the first and last lines of the error span, and Pensive reconstructs the selected text from its own log copy, leading to lower cost and better latency. This way, the model always returns two integers (start and end line number), whether the selected logs span a few lines or hundreds of lines. The displayed content comes directly from the original log, reducing hallucinations. The model can still choose the wrong range, which is why extraction quality remains an integral measurement for our AI evals, as we will discuss later.

For orchestrated workflows, “Pensive AI Error Extraction” runs automatically whenever a step fails. A step attempt is one execution of a step within a Maestro workflow. Maestro already calls Pensive for a regex-derived error and restartability classification, so AI extraction runs in the same request and returns its results alongside the established fields. The overall Pensive response must return within Maestro’s hard 90-second post-processing timeout for a failed step attempt.

That deadline shaped the extraction path. Pensive extracts errors from the logs in parallel to reduce latency, and the per-log calls share one deadline set well inside Maestro’s timeout limit. A per-log extraction call that misses the deadline is excluded from the result, allowing the overall request to return on time.

The data contract between Pensive AI and the various consumers needed the same care. The legacy data model contained one flattened error string; AI extraction instead returns a list of errors, one for each log source, with deep links back to the originating logs. Changing the old field in place would have broken existing clients. We introduced the new data model beside the legacy non-AI error extraction data model and continued producing both. Data Portal, our user interface for monitoring and observing data pipelines, preferred AI errors when available and fell back to the legacy value when they were not during the rollout. That allowed a gradual rollout and ensured that historical failures still display an error, since we chose not to backfill the old data due to log retention issues.

The migration is still in progress today. The Data Portal UI surface used for failed Maestro steps reads the new AI extracted error field today, while the query editor (for editing and running SQL), Workbench notebooks (managed compute platform and UI to edit and run jupyter based notebooks) and some slack clients calling Pensive still use the legacy error extraction path. Each client needs a compatible schema change before it can consume the new data. Keeping both fields populated lets teams adopt the new behavior at their own pace without forcing a coordinated cutover as we continue evolving Pensive AI.

Part 2: Full failure diagnosis and the product experience

Beyond error extraction, we built Pensive AI Debugging, a tool that analyzes failed step attempts and provides root-cause analysis and suggested remediations. It is available through programmatic APIs, MCP, and the product UI (Data Portal), all backed by a common entry point across the Netflix Data Platform. This shared model supports consistent behavior and custom product experiences.

This diagnosis still has to arrive quickly and remain consistent across investigations. A debugging response can take tens of seconds or a minute to generate, compared with a few seconds for error extraction. We run full AI-based debugging on demand instead of automatically, which keeps costs under control because most failed steps are never opened. We treat the failed step attempt, not the person who triggered it, as the unit of work, so everyone investigating the same failed attempt sees the same diagnosis.

That choice gives us a stable unit of work for caching. We cache each debugging response by step attempt. A second request becomes a key-value read instead of another AI inference call. The cache also prevents duplicate work, reducing latency and lowering costs further. This also means that a debugging generated in the Data Portal is cached and immediately retrieved for the MCP path instead of incurring another AI inference call. A conditional write creates an in-progress record, one caller starts generation, and other callers wait for the shared result to avoid conflicts when concurrent requests come in for the same step attempt failure. If generation remains in progress past an abandonment timeout, Pensive treats the record as abandoned and reclaims it. Completed responses expire after 90 days to help control costs and optimize storage because debugging requests for much older workloads are uncommon.

Pensive AI supports multiple ways to initiate debugging responses. A caller can (1) read a cached response, (2) generate one if no cache entry exists, or (3) regenerate and overwrite an existing response. Those semantics support the product UI, programmatic access, and the evaluation harness while keeping the response associated with a single immutable workload failure.

For users, the most important behavior is that a long-running answer feels alive. Pensive streams the response as it is produced. The service emits server-sent events, and the GraphQL layer republishes them as a WebSocket subscription for the Data Portal client.

Each event carries the complete response generated so far. That duplicates a small amount of text, but it keeps the client simple: it does not need a replay buffer or token reassembly logic, and the next event corrects the display after a dropped or late event. Generation is independent of the connection that requested it. A disconnect does not cancel the work, and the final answer still reaches the cache. A hard reload can show a loading state and poll for the completed result. Within the single-page application backing the Data Portal UI, active streams are maintained independently of view and component lifecycle, so navigation does not interrupt a response in progress.

Part 3: Measuring and improving the behavior we want

LLM-assisted systems need more than a demo or a few working examples. Before shipping in production, we need confidence in both error extraction and debugging guidance, and we need to know that we have not regressed on failures that legacy Pensive already handled. We built separate evaluation frameworks focused on the two distinct jobs.

Building labeled data is often the expensive part of evaluation. For error extraction, legacy Pensive’s hand-written rules provide reference labels for failures they recognize. They also serve as excellent ground truth in most cases. We run those rules over production logs and compare their extracted spans with Pensive AI’s output. This gives us a reliable regression set without relabeling every example by hand, though it is not an absolute ground truth (since our human-built rules could also have been stale). For now, we keep these rules frozen as a label source. This lets us continue to build comparison evals and run them on top of the existing system.

Stable inputs are essential for repeatable evaluation experiments where we A/B test new models/configurations or context injections to improve response quality. The logs Pensive AI uses can expire after as few as seven days or as many as 30 days. Production Pensive AI requests archive fetched logs to S3 object storage asynchronously during workload execution time so the upload is not on the user-facing critical path. Uploads do not overwrite an existing object, preserving the exact snapshot used to create an evaluation case. Evals use those archived logs, so score changes remain attributable to the system under test. The archive also prevents parallel experiments from overloading the source systems and lets us run them without affecting production. Cached S3 log inputs further reduce costs when running evaluations.

Concretely, we measure four complementary signals for error extraction:

• Inclusion: An LLM judge compares the legacy Pensive error for a failure with the errors Pensive AI extracted and determines whether the known error is still represented, even when wording differs.
• Precision: A second judge sees only the extracted AI errors and evaluates whether each is meaningful and actionable on its own.
• Repeated noise: A script looks across the whole dataset for lines that recur too often. A generic status line may look harmless in one answer but become distracting if it appears in a large share of failures. Lines appearing in more than about 30% of workflows are candidates for explicit exclusion. In practice, the exclusion list has only three lines, each of which otherwise appeared in nearly every extracted error.
• Latency: We record the latency of every extraction call. Quality scores alone cannot reveal when an apparent improvement requires too much additional work, which often correlates with increased AI costs.

This combination gives us an actionable feedback loop. Prompt changes can reduce general noise, while the dataset-level view exposes the stubborn boilerplate that prompt changes do not eliminate. Existing rules act as a regression floor, and archived logs give us a repeatable way to compare changes. At Pensive AI’s scale, model and system choices must satisfy quality, latency, maintainability and cost constraints together.

To seed the debugging ground-truth dataset for evaluation, we mine internal engineering-support Slack threads and use an LLM to identify threads that link to an orchestrated workflow, describe a resolved problem, and match the class of issue Pensive AI handles. A second model synthesizes a reference debugging response from the relevant discussion. Paired with logs from that workflow failure, it becomes a concrete evaluation row in our evals dataset. This approach works only retrospectively, because Pensive AI in production aims to provide debugging guidance before such a Slack discussion happens.

We also made the debugging generation process visible. Tracing follows the request through Pensive, with child spans for each log fetch and extraction tool call. Our tracing backend based on Braintrust brings the prompt, completion, tool calls, and latency into a single view. During early product development, that visibility helped us inspect low-performing debugging responses in context instead of guessing which input or operation caused them. Tracing helped us improve the responses and fill context gaps even before we had comprehensive evals. In production, the API and UI return the trace identifier with the response, so feedback can lead directly to the trace for in-depth analysis.

Part 4: Keeping quality in the release path

Evaluation is not only for experiments with LLM orchestration, models, prompts, or context changes. Changes to log selection, trimming logic, or a source system can change what the model sees and therefore change extraction quality. We use the same evaluation harness and evaluators throughout local development and Pensive AI deployment so those effects are visible before users encounter them.

For each pull request, the code branch is deployed to an isolated preview environment and a CI job runs an experiment with our evals against that isolated environment. Precision and inclusion thresholds gate the merge. The job also runs after a build is promoted, catching interactions that were not present when individual pull requests were evaluated in isolation. If the promoted build misses its thresholds, it alerts the on-call engineer to investigate or roll back.

The evaluator runs on a schedule against production to surface gradual drift or external changes. Developers can also trigger evaluation experiments remotely while iterating, using the same deployed-environment path as CI. Consistent infrastructure makes the results comparable across local iteration, preview builds, promoted builds, and production.

Part 5: Looking for feedback signal from users

The system is only valuable if it reduces the amount of manual debugging and log chasing done by our users. We wanted a measurement that was available from product telemetry and close to the behavior Pensive AI was designed to change. Manual debug-log exploration became one of the main signals. The Pensive interface already offers links to logs for a failed step attempt. To estimate whether the feature reduced raw-log hunting, we compared failed step attempts where someone interacted with Pensive AI with those where no one did. The second group averaged 321% more manual debug-log exploration actions. This observational comparison does not establish causality or a specific reduction in mean time to resolution, but it suggests that focused diagnosis often allows practitioners to inspect fewer raw logs. Programmatic MCP access also lets teams build operational flows with their terminals or custom agentic experiences around the same diagnosis service.

Another more explicit signal we collect is thumbs up and thumbs down on the debugging response. These feedback signals land directly alongside the internal traces so we can quickly associate feedback to the complete log of inputs/outputs and the path the AI took to arrive at the answer. More importantly, we created an internal business-hours alerting system where each thumbs down response sends us a Slack alert. This was valuable to resolve a lot of rough edges and context gaps initially in the product and is helping us improve the product continuously even today.

Part 6: Future Work building on these foundations

Pensive AI is the foundation for more work that goes beyond diagnosis and recommended remediations. We are exploring “Pensive AI Deepthink,” an opt-in managed agentic experience with higher cost and latency. It builds on the precise, low-latency analysis produced by Pensive AI Debugging and gathers additional context from internal documentation, manuals, past Slack discussions, source code, read-only warehouse queries, and Spark execution details. A routing step directs the investigation to the appropriate skill, such as Spark failures, data-audit failures, or late workflows. This keeps the shared starting point stable and deterministic while allowing the investigation to become more specialized, further improving recall. We are also in talks with the Spark team to leverage Pensive AI responses for failure debugging analysis and A/B comparison for upcoming platform migrations.

We also plan to invest in “Pensive AI Error Classification”. Hard-to-maintain legacy rules also classify failures by restartability and by whether they are platform or user caused. Maestro acts on those labels and changes its retry behavior accordingly like failing fast or tuning the default number of retries (platform errors are tried more than user errors by default). Error classifications must meet a higher bar than error extractions and debugging alone. Moving the majority of the error classifications to AI would let us retire most of the remaining rule-maintenance burden and avoid retrying unknown failures that were unlikely to succeed. For restartability, the costliest error is classifying a recoverable failure as non-restartable, because Maestro will turn a step that could have been recovered into a hard failure, leading to end user alerts and manual remediations. The evaluation framework will need to be tuned around that outcome before AI-enabled classifications can take on those decisions. Equally important, we must avoid retrying non-transient failures that will not succeed, because retries waste compute and delay data pipelines.

Our work on Pensive AI in Data Platform addresses concrete user pain points in debugging, improves maintainability for platform owners, and improves precision and recall for error extraction. Pensive AI extends coverage to failures that legacy Pensive previously classified as unknown. Reliability comes from the system around the AI model calls: bounded error extractions, compatible client data contracts, responsive delivery, reproducible evaluations, caching and tracing, and release controls that keep quality and cost visible as the platform changes.

Part 7: Acknowledgements

This work would not have been possible without our stunning colleagues across Data Platform and our Data Engineering and Analytics partners who provided early feedback. While we cannot name everyone individually, we are deeply grateful for their contributions, which shaped both response quality and the product experience. Special thanks to Adith Swaminathan for advising on evaluations and inference improvements and acting as a thought partner. Thank you to the Orchestration team members, especially Deepak Ramalingam and Malcolm Ramirez for their additional engineering contributions.

For dogfooding and integrating Pensive AI as part of user support, we also thank the engineering support organization (ESO Team). We also thank Abhishek Kapatkar (Data Developer Experiences Team) and Emmie Dong (Data Movement Team) for their feedback to improve the Pensive AI integrations with the rest of the ecosystem. Special thanks to the AI Platform team (Kyle Ahn, Rob Mann) and Braintrust Team for helping with distributed tracing, evals infrastructure and model gateway runtime support.

We also thank Kamran Masood, Ely Spears, Ronald Leung, Anjali Norwood, Prashanth Ramdas, Charles Smith, Harikrishna Menon and other leaders for their constructive feedback and support on the Pensive AI project.

Making Failure Diagnosis Reliable with Pensive AI was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

At Netflix, a crucial part of our mission is ensuring our content looks stunning on any device. Vertical formats are increasingly important for Netflix, as we have 1.2B followers across our social media accounts, and have recently rolled out Clips, our vertical video feed in the Netflix mobile app. Traditionally, creating vertical assets meant editors and producers manually reframed horizontal videos in editing tools, a process that doesn’t scale as our catalog and use cases grow. To support both our in‑app feed and external marketing campaigns at scale, we built an automated multi‑ratio video cropping system, tightly integrated into our internal production tools, that can generate high‑quality vertical and other aspect‑ratio variants with minimal manual effort.

Our Video Resizing Approach

Converting cinematic 16:9 footage into mobile‑friendly aspect ratios takes more than a simple center crop. Filmmaking often involves dynamic character blocking and complex camera movement. We need an intelligent “virtual camera” that can follow the key action, respond to how each scene is composed, and produce crops that feel as intentional as the original shot.

We address this challenge by framing video resizing as a tracking and temporal optimization problem. Our pipeline processes each asset in three main stages, designed to produce 9:16, 4:5, and 1:1 crops in a single pass, without incurring additional computational overhead.

Shot Detection and Splitting

First, we segment the video into atomic shots using a neural‑network–based shot boundary detector. In many cases, shots contain multiple characters conversing on opposite sides of the frame. To better handle these scenarios, we use an Active Speaker Detection based approach to logically split a single continuous shot into multiple sub‑shots. This allows our virtual camera to cut and re-focus on different speakers as the conversation flows. Our Active Speaker Detection leverages an internal face detector and LipFlap to reliably identify active speakers in both live‑action and animated titles.

Focus Region Discovery

Once shots and sub‑shots are defined, we identify the most important region in each frame. We combine face detections with visual saliency maps to filter and prioritize the faces that should be in focus. In scenes with multiple characters, we further rank faces using signals such as active speaker status and relative size. Our model aggregates these cues with shot‑level content understanding to establish a continuous “focus region” for the virtual camera.

Temporal Reframing

Once we’ve established the focus region, the next challenge is deciding how the crop should move over time. Tracking the focus region on a strict frame-by-frame basis would produce a shaky, distracting viewing experience, so we apply a set of heuristics that consider how the focus region moves over the course of a shot. When its displacement is small, we lock a static crop window that consistently covers the relevant area. When there is significant motion, we compute a dynamic tracking window and smooth its path so that any panning feels intentional and cinematic.

Tooling and Workflow Integration

We integrated the resizing algorithm directly into our internal Clips tool, so editors and producers can access it as part of their existing workflow. The interface enables them to automatically generate multiple aspect‑ratio versions of a clip with smart reframing. A unified in‑app module provides intuitive shot‑reframing controls, allowing users to review the algorithm’s crops, and quickly make targeted adjustments where needed.

For more complex shots or custom campaigns, producers can export the resizing parameters as XML and load them directly into external timeline editors such as Adobe Premiere. This allows agencies and external partners to start from the algorithm’s framing decisions instead of recreating crops manually or relying on tools like Premiere’s Auto Reframe. In our internal test, our resizing algorithm outperforms Auto Reframe on a curated evaluation set, so the XML export effectively brings those higher‑quality crops into external workflows. By bringing automated resizing and manual refinement into a single place, this pipeline significantly reduces production time and manual effort while preserving creative control.

Conclusion

Our work on multi-ratio video resizing is one step in a broader effort to make content creation more scalable, without compromising creative intent. By pairing algorithm-driven reframing with editor-friendly tools, we’ve reduced the time teams spend on repetitive resizing tasks and unlocked faster experimentation across mobile feeds and marketing campaigns. As we continue to broaden these capabilities to new formats and workflows, our goal remains the same: to give creatives more time to tell great stories, and to deliver a consistently high-quality viewing experience for our members and fans.

Multi‑Ratio Video Resizing for Mobile Feed and Marketing was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

At Netflix, shared Java libraries support applications developed across a large, independently evolving repository ecosystem. A library change begins in one repository, but its impact can reach many others. Producers can see the API diff in front of them. What they often cannot see is who still depends on the symbols that changed.

The need for that visibility grows as coding agents take on more of the software development lifecycle. An agent can accelerate a local edit, but without cross-repository context it cannot judge downstream risk or help plan consumer migrations. The evidence a producer uses to assess a change can also guide the agent working on it.

Consider a common case: removing a deprecated API. Deprecated entry points expand the surface producers must test, document, and support, and they can constrain how a library evolves.

Suppose a replacement has been available for several releases and the old method cannot express options required by new behavior. The library’s tests pass, and a compatibility checker flags the removal. The producer still cannot answer the questions that determine what to do next:

• Does any consumer still call this method?
• Are consumers concentrated on one version or spread across several generations?
• Does another library implement or wrap the API, multiplying its impact?
• Is direct coordination sufficient, or does this require a deprecation and migration campaign?

At small scale, producers can ask nearby teams or search a handful of repositories. Across many repositories, it is harder to find every caller of a method and the library version each one uses. Those answers are scattered across consumer codebases and builds.

Without that evidence, a producer may release a breaking change and discover the impact through consumer failures, or preserve an old API indefinitely because the risk is unknown. Libraries need to evolve, and their producers need evidence that helps them do so intentionally.

We built Quasar to index symbol definitions and references across Netflix repositories. For library producers, that shared index helps answer two questions:

1. What changed in the library’s public API?
2. Which repositories use the exact symbols that changed?

The first identifies a potentially incompatible change. The second estimates its observed blast radius. Together, they turn a generic compatibility warning into evidence a producer can act on before merging or releasing.

From API Changes to Consumer Impact

Existing techniques each see a different part of the problem. Binary and source compatibility tools identify contract changes, but cannot tell who uses the changed API. Dependency graphs show declared reach, but cannot prove that code calls a particular method. Text search may miss overloads, inheritance, or the artifact version behind a call. Consumer builds provide strong validation, but running them across the fleet is expensive.

Quasar builds on SCIP, an open interchange format for semantic code intelligence. Language-specific indexers encode definitions, references, and relationships using stable symbol identifiers. Quasar retains API definitions for library versions and commits alongside references from consumer repositories. That shared identity lets it connect an API change in one repository to exact usages elsewhere, even when those repositories were indexed independently.

On each enrolled CI build, language-specific indexers submit SCIP indexes to Quasar’s ingestion service. It writes API definitions and consumer references as Parquet in Amazon S3, retaining the symbol identity, dependency version, and source location needed to investigate a change.

During pull-request review, Quasar compares indexed base and head definitions and counts direct consumer references with Amazon Athena. Quasar exposes those findings through its REST and GraphQL APIs and MCP server; follow-up usage queries provide dependency-version breakdowns and source locations. A separate compatibility-impact assessment can also follow implementations and their callers when a producer supplies a changed JVM symbol.

Consider this simplified API change:

public interface Client {
- Result getLegacyValue(Request request);
+ Result getValue(Request request, Options options);
}

For a detailed compatibility-impact assessment of this removal, a producer-side tool supplies the old method descriptor. Quasar constructs its canonical symbol identity. This abridged excerpt from the direct-reference portion of Quasar’s Amazon Athena query shows the essential match:

SELECT c.change_id, 'DIRECT_REFERENCE' AS evidence_kind,
       r.scm_type, r.org, r.repo, r.dep_version, r.file_path
FROM requested_changes c
JOIN quasar.symbol_refs r
  ON r.symbol_canonical = c.target_canonical

requested_changes contains the API changes submitted by the producer. The SQL excerpt finds repositories that call the removed method. Quasar also checks implementations and calls through them, so it can count each affected repository once. It returns sample dependency versions and file locations to help investigate the result.

For the Client change, this tells the producer which repositories still use getLegacyValue, which versions they use, and where some of those calls occur. That helps the producer decide whether to coordinate a migration before removing the method.

Bringing the Analysis Before Release

Version-to-version comparison is valuable when planning an upgrade or writing a migration guide. Producers benefit even more when feedback arrives during code review.

For pull-request analysis, Quasar stores API definition snapshots keyed by commit. Given a base and head commit, it classifies symbols as added, removed, or signature-modified. Removals and modifications then become candidates for consumer-impact analysis.

An incompatible change is not automatically a high-risk change. Quasar reports the observed impact so the producer can decide what to do; it does not enforce a release policy. “No references observed” means only that Quasar found none in the indexed population. Reflection, configuration, dynamic loading, stale branches, or incomplete indexing can hide consumers. The result needs to show whether the requested commits were indexed, how fresh the consumer indexes are, and whether evidence was truncated.

Pull-request review requires indexed snapshots for the exact base and head commits. If either is unavailable, Quasar reports that it cannot compare them; it does not substitute a default-branch snapshot. This keeps a missing comparison from looking like a clean API diff.

Return to our Client change. This illustrative report brings together signals available through Quasar's API comparison, consumer-usage queries, and library-level criticality. It is not a literal response from one API:

{
  "change": "REMOVED",
  "symbol": "Client#getLegacyValue(Request)",
  "affectedRepositories": 137,
  "versionDistribution": [
    { "version": "4.2.1", "repositories": 86 },
    { "version": "4.1.7", "repositories": 38 },
    { "version": "3.9.4", "repositories": 13 }
  ],
  "sampleConsumerLocations": [
    {
      "repository": "checkout-service",
      "filePath": "src/main/java/com/example/checkout/CheckoutClient.java",
      "lines": [84, 129]
    },
    {
      "repository": "client-wrapper",
      "filePath": "src/main/java/com/example/client/ClientAdapter.java",
      "lines": [57]
    }
  ],
  "libraryCriticality": {
    "tier": 1,
    "basis": "highest-tier downstream application"
  },
  "comparison": {
    "baseShaFound": true,
    "headShaFound": true
  }
}

The repository count is symbol-specific: in this example, 137 indexed repositories refer to the removed method. The version distribution reveals a dominant cohort on 4.2.1 and a smaller long tail still on 3.9.4. A producer can inspect affected repositories and retrieve indexed file paths, with line numbers where the index provides them. The JSON shows only two sample locations to keep the example readable. The filePath and lines shape mirrors Quasar's query contract. The tier is broader context about the library, inherited from its highest-criticality downstream application; it does not imply that all 137 callers are Tier 1.

Together, those signals change the producer’s plan. Instead of merging the removal and waiting for failures, the producer can retain and deprecate the old method, publish a migration path, coordinate a broader migration, and track the long-tail versions before removal. File paths and available line numbers turn that scope into a concrete work list for engineers or coding agents. The same API diff paired with weak or stale evidence would lead to a different action: refresh the indexes before deciding. The value is not merely detecting the removal; it is choosing a proportionate next step.

A Polyglot Foundation with SCIP

The shared SCIP format also lets Quasar index several language ecosystems. Each language indexer resolves its own code, while Quasar can use the resulting symbol identities to find references across repositories.

Conceptually, client-lib indexes Client#getLegacyValue(Request) as a definition, while checkout-service indexes a reference to the same symbol. Quasar can then connect the library API to its consumer across repositories.

Relationships are especially useful when generated code separates the API a producer owns from the symbols consumers invoke. For example, Quasar can trace a Protobuf definition through its generated Java members to the repositories that use them, even when consumer code never names the .proto element.

SCIP records symbols and their references; it does not determine whether an API change violates an ecosystem’s compatibility rules. Today, Quasar compares indexed JVM and Protobuf API definitions and traces Protobuf changes through generated Java to find affected repositories. A separate JVM compatibility-impact path uses breaks identified by a producer-side tool to query affected repositories. Go, Python, JavaScript/TypeScript, and Rust support usage queries, but Quasar does not yet assess their breaking API changes.

Moving Risk Assessment Earlier

Today, Quasar surfaces API-change findings on pull requests, including observed consumer counts and confidence. Producers can use its other queries to inspect dependency versions and source locations, with library criticality as broader context. Together, those signals support a decision before merge.

We want to move that context earlier. An agent writing library code should recognize when it is changing a public symbol and consult Quasar before the change becomes a pull request. Quasar’s APIs and Model Context Protocol (MCP) integration provide an access path, but access alone does not create awareness. The coding workflow must also recognize public API edits so the agent knows when to query impact and when to involve the producer.

Earlier guidance will not replace pull-request analysis. The PR assessment will still verify the concrete diff and catch changes that were not recognized during coding. Coverage, freshness, ambiguity, unsupported platforms, and truncation must travel through both checks in machine-readable form. An agent should not turn “no observed usages in the available index” into “safe to remove.”

Today, agents can query Quasar through MCP when prompted. We are working toward making that context part of the coding workflow, then using identified call sites to prepare migration changes. Quasar supplies semantic evidence, an agent reduces migration toil, and the library producer remains responsible for API and release decisions.

What We Learned

Building this workflow led us to four practical lessons.

• Separate compatibility from reach. A removed method can break the API contract even when Quasar finds no callers. A list of affected repositories and their library versions shows the migration work ahead. Producers need both signals to choose a release path.
• Make the evidence inspectable. A repository count is more useful with sample call sites and a version breakdown. The result also needs to show which commits were indexed, how fresh the consumer indexes are, and whether evidence was truncated. Otherwise, “no usage found” can be mistaken for proof that nobody is affected.
• Keep usage and compatibility separate across languages. SCIP lets Quasar find symbol references in several ecosystems. Deciding whether an API change is breaking requires compatibility rules or a trusted producer-side signal for that ecosystem.
• Bring evidence to the producer while choices are open. Pull-request findings can prompt a producer to retain an API, publish a migration path, or coordinate consumers before merge. Moving the same evidence into the coding workflow could shape the edit earlier. After release, it still helps locate consumers that need to migrate.

Today, Quasar indexes JVM, Go, Python, JavaScript/TypeScript, and Rust code for cross-repository usage analysis. It compares JVM APIs and traces Protobuf changes through generated Java. Producers can review change findings through Quasar’s REST and GraphQL APIs or its MCP server, then investigate affected repositories and call sites. We are extending this foundation toward more ecosystem-specific compatibility checks, deeper code-review integration, and agent-assisted consumer migrations that keep release approval with the producer.

Quasar is internal to Netflix, but the pattern is portable: index code semantically, retain API definitions, compare them at the point of change, query affected consumers, and report the quality of the evidence. Library producers need more than a compatibility verdict. They need to know who may be affected and have enough time to act before release.

A huge thank you to the Java Platform team at Netflix. Their expertise, collaboration, and support made this work possible

Related Talks

• From Lag to Lightning: Transforming Dependency Update Timelines explores how Netflix shortened dependency-update timelines.
• Netflix’s Journey to Confident Automated Code Changes discusses building confidence in automated changes at scale.

Beyond API Compatibility: Building a Tool to Measure the Blast Radius of Library Changes was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

You click play on a Friday night, and in the second that follows, dozens of services answer in milliseconds. Most of them run on Titus, our internal container platform. Titus is, in essence, a Kubernetes-based container hosting platform: we keep both the control plane and the data plane (the container hosts that actually run containers) patched and up to date, so application teams get a consistent and reliable container experience. They hand us a container, and we run it, with no host, cluster, or kernel to own. Titus launches millions of containers every week across tens of thousands of EC2 instances in multiple AWS regions. It is multi-tenant by default: many containers from many teams share each host. That density is what makes container compute efficient, and it is why Titus has been the default home for so much of Netflix’s compute for years.

For most workloads, sharing a host works well. For a few, it does not, and those few include some of the most demanding real-time systems we run. This post is about them, and about single-tenant Titus, a mode that gives a task its own virtual machine while keeping everything else about Titus the same: the API, the deployment path, and the managed experience where the Compute/Platform team, not the application team, owns the host.

We will walk through why shared-kernel isolation runs out of road for certain workloads, how we measure isolation rather than asserting it, the options we weighed (microVMs, gVisor, EC2, and more shared-kernel tuning) and why a dedicated VM inside Titus won, how the mode is built on the Titus federation layer and autoscaling, and what we have seen along three axes: isolation, fleet efficiency, and the operational cost of keeping the fleet patched.

The cost of good neighbors

Multi-tenancy is a tradeoff. Packing many containers onto a large bare-metal instance keeps utilization high, but it also means those containers share a single Linux kernel and the underlying hardware. Titus isolates them with the tools the kernel provides: cgroups cap CPU, memory, and disk; namespaces give each container its own view of the system; seccomp and AppArmor restrict the system calls a container can make; the Completely Fair Scheduler divides CPU time; the iocost controller meters disk IO; and traffic shaping meters each container’s slice of the network. Each container also gets its own identity, IAM role, and security groups, so tenants are isolated for security and resource management.

These mechanisms provide isolation for what the kernel can partition. They cannot fully partition what is shared in hardware. A group of cores shares a last-level cache and a path to memory, so a container that suddenly pulls a lot of data from RAM evicts a neighbor’s cache lines and slows it down, even when both stay inside their CPU limits. There is no cgroup for memory bandwidth or cache residency, as there is for CPU shares. The interference is real, it is hardware, and it does not show up as either container exceeding its quota.

A noisy neighbor and a co-located app each stay within their cgroup CPU and memory limits, but both draw on a shared last-level cache and memory bandwidth that lack cgroup fences. The noisy neighbor takes most of it, so the co-located app stalls and its tail latency rises even though no one exceeded a quota.

We have written about this before. Predictive CPU isolation of containers at Netflix moved some placement decisions out of the kernel and into a service that forecasts each container’s CPU demand and re-pins containers to cores to reduce cache interference, formulated as a combinatorial optimization solved every minute. Noisy Neighbor Detection with eBPF instruments the kernel scheduler to surface per-container run-queue latency, so we can see a victim being starved in close to real time. And Mount Mayhem traced container-startup slowness to a global kernel lock amplified by modern CPU cache topology. Each of these raised the floor for everyone on the platform, and we are continuing that work.

A decade of shared-kernel isolation, at Netflix and across the industry, has not made co-tenancy invisible, because a noisy neighbor is not one problem but a family of them. Almost every physical resource a host shares is its own axis of interference, and a cgroup can fence very few of them:

• CPU performance indeterminacy. Co-located threads contend for the run queue, so scheduling latency rises even while a container remains within its CPU quota. Separately, a neighbor with a large working set evicts a tenant’s last-level cache lines and saturates memory bandwidth, neither of which a CPU share can partition.
• Network bandwidth and packets per second. Per-container shaping competes for link bandwidth, and receives processing runs in softirq context, charging one container’s packet work to whatever CPU a neighbor is running on. A flood of small packets exhausts a core’s softirq budget long before any bandwidth cap is reached.
• Storage bandwidth and IOPS. A neighbor saturating a device’s queue serializes everyone’s I/O, and on cloud block storage, the per-volume and per-instance throughput ceilings are a shared budget in their own right.
• Long tail. Page-cache reclaim, NUMA and cross-socket interconnect traffic, TLB-shootdown interrupts, PCIe and GPU contention, package-level power and thermal throttling, and contention on kernel global locks (the class of problem behind Mount Mayhem) each degrade neighbors in their own way.

What unites them is that they are physical, shared, and below the level a quota can reach. A platform can cap how much CPU time or memory a container is granted; it cannot hand each tenant its own slice of the memory bus, its own softirq budget, or its own portion of the last-level cache.

Of these, the network dimension has bitten hardest in our experience. Per-container bandwidth shaping uses a hierarchical token bucket (HTB) queueing discipline that serializes traffic with a global lock to account for each container’s bytes and packets. Under a flood of small packets from a single container, contention for that lock can burn kernel CPU and starve co-located containers of the time they need to process their own traffic, resulting in TCP retransmits and tail-latency spikes on services that did nothing wrong. The mechanism meant to prevent noisy neighbors becomes one.

The workloads most exposed to this are stateful, high-throughput streaming jobs: real-time data pipelines and stream processors that continuously move large volumes of data and maintain in-memory state. When one of these is perturbed, recovery is not cheap. A stream processor that falls behind or loses its place may have to restore from a checkpoint and pause processing for many minutes to catch up. For these jobs, intermittent interference from a neighbor is not a small slowdown that averages out; it is a stop-the-world event that pages an on-call engineer. The same teams report, accurately, that they never saw this on a plain EC2 instance.

Why we kept coming back to the VM

Full VM isolation, where a workload gets its own host and kernel, is the standard our customers are already happy with. The question was not whether VM isolation works; it was how to offer it without giving up what makes Titus worth using. We weighed four paths.

1. Keep improving shared-kernel isolation. This helps every workload on the platform, needs no opt-in, and we are still investing in it (more on the roadmap below). Its limitation is structural. Each round of tuning narrows the gap without closing it, because the underlying hardware remains shared. Cache, memory bandwidth, and the kernel’s own shaping and scheduling locks are not things a container boundary can fully contain. Shared-kernel isolation is the right default and the wrong promise for a workload that cannot tolerate a bad minute.
2. microVMs (Firecracker, Kata Containers). Lightweight VMs boot in about 150 milliseconds by emulating far less hardware than traditional VMs, which is why they power serverless platforms. They were the most appealing alternative on paper, and we carefully reviewed them. Three things made the cost hard to justify. First, the hardware virtualization they rely on is available only on bare-metal instances or through nested virtualization, and nested virtualization was not ready on the instance types we run at Netflix at the time, so adopting microVMs broadly would have constrained us to a narrow slice of the fleet. Second, there is no managed microVM service we could build on, which means we would have to run and maintain a virtualization layer ourselves: patching the hypervisor, live-migrating guests, and qualifying emulated devices. That is exactly the kind of operational surface we prefer not to own when we can avoid it. Third, even setting those aside, microVMs add virtualization overhead on CPU- and IO-bound work and do not support GPUs. Excellent isolation, but at a price in flexibility and operational load we were not willing to pay.
3. gVisor. A user-space kernel that intercepts system calls provides sandbox-grade isolation without a hypervisor, thereby avoiding the bare-metal constraint. The catch is performance and dependency. Intercepting syscalls in user space adds overhead on IO-heavy paths, which are precisely the workloads we most need to protect, and the open-source project lags behind its internal counterpart in fixes and is effectively maintained by a single vendor. The isolation we would gain came with a throughput tax on the exact jobs we were trying to help.
4. Move the workload to EC2. It is possible to run containers directly on EC2 instances. The problem is that the application team takes back the operational load that Titus exists to carry: choosing instance types, patching hosts, and sizing and scaling the fleet. For a team that came to Titus precisely to avoid those things, this is a regression in the experience, and the move between Titus and EC2 has historically been a one-way door that teams are reluctant to cross.

Spectrum of isolation options from lighter and more shared to stronger isolation: shared-kernel containers, then gVisor, then microVMs (Firecracker or Kata), then a full VM as used by single-tenant Titus, which is highlighted. Each option is annotated with its launch latency, overhead, and the entity that operates it; single-tenant Titus is a dedicated EC2 instance with no additional virtualization overhead.

There is also a question of what kind of isolation we actually need. microVMs and gVisor are built for security isolation between mutually distrusting tenants, the case where one tenant might actively try to attack or escape into another. Among Netflix teams, that threat model is more than the situation warrants. What these workloads need is resource and performance isolation: a guarantee that a neighbor cannot steal their CPU, cache, memory bandwidth, or network, not protection from a hostile co-tenant. A dedicated EC2 VM delivers exactly that, with hardware we do not have to emulate and a kernel we do not have to sandbox. We pay for the isolation we need rather than for a stronger security boundary.

microVMs and gVisor asked for a large, bespoke investment with real risk in order to approximate something we could get directly: a VM. Using EC2 to run containers provides the VM, but takes away the managed experience. So we took the direct route and brought single-tenant VM isolation into Titus itself, under the existing API and the existing shared-responsibility model.

How single-tenant Titus works

The idea is simple to state: a single-tenant task runs on its own EC2 instance, sized to fit it, with nothing else scheduled alongside. That instance is still a Titus container host that the Compute/Platform team owns and patches. The team running the workload sees the same Titus API and the same deployment path. Nothing about how they package or ship the job changes.

Side-by-side anatomy of a multi-tenant Titus, where multiple teams’ containers share one Linux kernel and the hardware beneath it and contend with one another, versus a single-tenant node where one task plus its platform sidecars owns the whole machine with no co-tenants.

Most of the work happens in the federation layer, the part of Titus that decides where a job runs (which cell, which capacity) before it is scheduled onto a node.

When a job is eligible for single tenancy, three things happen.

1. Federation decides eligibility and the envelope. Eligibility is governed by an allowlist maintained by the Compute/Platform team; a team opts in by targeting a single-tenant workload group rather than flipping a flag in the public API. For an eligible task, federation tags it as single-tenant and records the set of instance families it may run on, ensuring consistent performance across the workload. Consolidating this decision in federation follows the same principle we use for routing in general: placement intelligence belongs in one place, above the individual cells, where it can account for capacity and availability across the region.
2. Autoscaling provisions a dedicated node. Given a pending single-tenant task, our autoscaling implementation launches the smallest instance that satisfies the task’s requested CPU, memory, and GPU from the allowed families for the task just in time. Taking the tightest fit keeps the unused remainder of each dedicated node small. The next section covers this in detail.
3. The scheduler enforces one task per node. Single-tenant pods are configured with pod anti-affinity to prevent colocation with other pods, along with a priority class that exempts the task from preemption. Single-tenant nodes are also tainted to prevent unrelated workloads from being placed there, with only single-tenant workloads tolerating this taint. We also disable defragmentation/repacking for these nodes, so a single-tenant task is never relocated to repack the fleet. Together, these mean a single-tenant task has its host to itself for its entire life, undisturbed by a neighbor or by the platform’s own bin-packing.

Two design choices are worth dwelling on. First, single tenancy is allowlisted on purpose. If it were a free checkbox in the API, most teams would tick it, because who wouldn’t want a quieter host? And the efficiency that multi-tenancy exists to provide would erode across the fleet. Keeping the Compute/Platform team in control lets us direct dedicated isolation to workloads that need it and steer the rest toward shared capacity, while keeping the public API surface small as we learn. Second, we do not silently resize anyone’s job to fill its node. The job keeps the resources it requested; we recommend sizing it close to a node shape so the dedicated instance is well utilized, while keeping the workload portable and running identically whether in single-tenant mode or not, regardless of the host’s dimensions. Moving in and out of the mode is a configuration change, not a rewrite.

A few perpetrators, a lot of collateral damage

Noisy-neighbor pain is not spread evenly. In practice, a small number of workloads cause most of the disruption: a job that floods the host with small packets, churns the page cache, or saturates memory bandwidth degrades the performance of every other container unlucky enough to share its host. Some of these are also among the workloads most affected by contention, because a job can be both a source of interference and sensitive to it. The shape of the problem is a few perpetrators and many victims.

That shape is what makes single tenancy a precise lever rather than a blunt one. We do not need to give every workload its own machine. We need to pull the handful of perpetrators and the most disruption-sensitive victims onto dedicated nodes. Doing so protects their former neighbors directly and removes the reason the platform was spreading sensitive workloads apart with anti-affinity placement in the first place. The hosts left behind are quieter, and because they no longer have to be kept partly empty to honor spread rules, they pack more densely.

Before-and-after view of the Titus fleet. Before: a few red perpetrators are scattered across shared hosts, the amber workloads beside them are disrupted, and anti-affinity spreading leaves dashed stranded capacity. After: each perpetrator runs on its own dedicated single-tenant node, the remaining shared hosts are fully loaded with healthy workloads, and one host’s worth of capacity is reclaimed.

On the left is the fleet before: perpetrators sit on shared hosts, the workloads next to them suffer, and anti-affinity rules leave gaps that cannot be filled. On the right is the same fleet after: each perpetrator has its own right-sized node where it can be as noisy as it likes, with no one to disturb; the remaining hosts run full of undisturbed workloads; and the capacity that used to be stranded comes back. A few dedicated nodes buy isolation for the perpetrators and a calmer, denser fleet for everyone else.

Provisioning many small nodes

Single tenancy trades fewer large, shared instances for more small, dedicated ones, and that shift has consequences across the stack.

The first is the control-plane load. More nodes mean more for the Titus control plane to track and more VM launches against the EC2 control plane, the latter mattering most during a regional failover, when we scale up the fastest and can least afford a throttle. Titus is already a federated system: each region is divided into independent cells, which limit how large any single control plane can get, and let us add capacity by adding cells rather than growing a single stack without limit. We migrate workloads in stages rather than all at once, and we exercise the launch-rate path under controlled load on a schedule rather than discovering its limits during an incident.

The second consequence is capacity management, where the design changed the most. With one task per node, the fleet now spans many instance shapes whose demand shifts continuously as jobs come and go. Pre-staging static pools of every shape would be slow to manage and wasteful, so we instead provision single-tenant nodes on demand. It handles provisioning end-to-end, and several of its features line up with exactly what single tenancy needs:

• Just-in-time, right-sized provisioning. Single-tenant jobs are configured with a list of supported instance families based on their performance requirements. At launch, they are mapped to the best-fit node size by comparing requested resources against the selected instance families. New nodes of that instance type are brought up by the autoscaler to meet the pending demand. Single-tenant pods are configured with anti-affinity to all other pods, so the scheduler naturally lands one pod per autoscaled node. Newly vacated nodes may be reused by any pending deploys before they’re automatically torn down. The node pools themselves are shared across single-tenant workloads and describe a catalog of available instance families and sizes.
• Single tenant node pools. Single tenant may run in separate clusters or be shared with multitenant clusters. The pools are configured with a list of supported instance-type permutations and are pinned to a Titus container host AMI. This is where the managed experience lives: the host image and its configuration are ours, declared once and applied to every node created.
• Consolidation management. In multi-tenant environments, we default to a repack/defrag behavior that periodically relocates workloads. This doesn’t apply in single tenant, where nodes are explicitly each running at most one workload. We therefore reclaim only nodes vacated by workloads that have scaled in or terminated. Node termination is performed in two stages, allowing nodes to be reused for pending workloads before they are fully terminated.
• Drift-based patching. When we publish a new container host AMI, we support a “mutable” deploy to existing nodes without affecting any hosted containers, and an “immutable” deploy that updates the target AMI for newly created nodes. This means that many node upgrades don’t require disrupting workloads, but we reserve the right to roll the fleet with our users if needed.

The launch-latency tax, and why warm pools are harder here

Provisioning on demand does not make a node appear instantly, and launch latency is the tradeoff single tenancy asks us to manage most carefully. Bringing a single-tenant task to the point where it can serve traffic has two parts: standing the node up (booting a fresh EC2 instance, then pulling and extracting the container image, which is the largest part of host setup for most jobs) and the application’s own startup (a JVM loading and JIT-compiling its classes, a cache filling, and so on). A fresh instance takes tens of seconds before the image pull even begins.

In multi-tenant Titus, most of this is hidden. A new task usually lands on a Titus container host that is already running and already has the image layers cached, so starting it takes only a few seconds rather than minutes; the host boot was paid for once, long ago, and amortized across every task that has run there since. Single tenancy gives up that hiding place. Every single-tenant task waits for its own machine to boot and pull its image.

The usual remedy is a warm pool: keep some instances booted and idle so a pending task lands on one immediately. This works well for the multi-tenant fleet precisely because that fleet is close to homogeneous. A handful of large shapes cover almost everything, since any task can be packed onto a big shared host. A small amount of warm capacity in a few shapes gives a high hit rate, and its cost is spread across the many tasks that will use it.

Single tenancy inverts those economics, and this is the part worth being explicit about.

Press enter or click to view image in full size

Because every task gets a right-sized node, the fleet spans many instance shapes (several families across many sizes), and a warm instance only helps a pending task whose shape matches it. A warm r7i.2xlarge does nothing for a task that needs an m7a.8xlarge. To achieve a good hit rate, we would need to hold idle capacity in every family-and-size bucket, and idle capacity in a bucket with no task to fill is pure cost. The very property that makes single tenancy efficient at steady state, tight best-fit sizing, works against warm-pool reuse: more distinct shapes mean more buckets and a lower hit rate per dollar of standing capacity. And the pressure peaks exactly when it is least welcome. During a regional failover or a live event, when many tasks need fresh nodes at once, a thin warm pool spread across many shapes drains immediately, and the rest of the demand pays the full cold-boot cost.

So we treat warm capacity as a small, targeted lever rather than a default. We keep a little pre-warmed capacity for the most latency-sensitive single-tenant starts, and we deliberately narrow the menu of shapes to a curated set of sizes rather than the full continuum, so warm capacity concentrates into fewer buckets and trades a little best-fit efficiency for a better hit rate. Everything else provisions just-in-time and accepts the boot.

The more durable wins are in cutting the launch cost itself rather than pre-paying for it, and they help both single- and multi-tenant nodes, which is the better place to spend the effort:

• Faster image distribution. The image pull dominates the host setup, so newer OCI image formats and lazy-pulling snapshotters such as stargz and SOCI, which fetch only the bytes a container actually reads at startup rather than the whole image up front, reduce the largest component of the gap.
• Checkpoint and restore. For processes that are slow to initialize, snapshotting a warmed-up application with a tool like CRIU and restoring it, rather than cold-starting it, removes most of the second half of launch latency.
• Instance type selection. In multitenant configurations, we have historically used metal instances, which offer slightly better performance but come with a slower startup time. In a single-tenant configuration, node startup latency is more important, so we have moved away from these instances.
• Scheduling optimization. Through regular stress testing of fleet scaling, we found a scheduling bottleneck caused by Kubernetes preemption logic, leading to nodes being scaled up quickly but then left idle while waiting for workloads to be scheduled on them. This was ultimately unnecessary, as our single-tenant pools do not run preemptible workloads. Disabling preemption greatly improved scheduling performance when under strain.

None of this makes the tradeoff disappear, and it is worth stating plainly: a single-tenant node is slower to stand up than a slot on an already-warm shared host, and hiding that with warm pools costs more per node here than it does on the multi-tenant fleet, because the warm pool cannot be as homogeneous. But for the long-running services that single tenancy targets, paying that cost once at launch in exchange for a quiet life afterward is a trade these teams are glad to make.

Measuring the difference

Isolation claims are easy to assert and hard to trust, so we measure them with a repeatable benchmark and a single score.

The method is a differential one. We run a reference container, a healthy and representative application, and record its throughput and tail latency while it runs alone. Then we run it again next to a havoc container engineered to be a bad neighbor: a UDP flood through XDP to stress the network-shaping path, pointer-chasing and STREAM to pollute the last-level cache and saturate memory bandwidth, fsync storms against the disk, stress-ng to abuse the scheduler, and GPU-burn to monopolize an accelerator. We only trust a run once the host has reached a steady state, which we confirm with the kernel’s Pressure Stall Information: we wait until cpu_some or mem_some shows that tasks are actually stalling on a contended resource, not merely that the resource is busy. Each scenario is run many times with fixed seeds, and we report the median with a bootstrapped confidence interval, because noisy-neighbor distributions are heavy-tailed and a simple average can mask the spikes that matter.

From the reference container’s numbers, we derive a simple measure of how well isolation held. Throughput efficiency is the fraction of operations per second the app retains under attack; latency efficiency is the ratio of its undisturbed P99 to its P99 under pressure. A neighbor that doubles the reference app’s P99 drops the latency efficiency. As an illustration, a reference app whose P99 latency goes down next to a havoc container achieves a certain latency efficiency, and one whose throughput falls from a to b operations per second achieves a certain throughput efficiency. The composite score is a weighted blend of the two, with a latency-sensitive preset that gives greater weight to tail latency for interactive and streaming-path services. A perfectly isolated system scores 100%, and a single hard failure invalidates the run regardless of the average.

Noisy-neighbor isolation

cgroups container isolation isn’t perfect, and applications may experience worse performance due to sharing a machine with a determined neighbor. This could extend to disk and/or network IO where the same volumes or interfaces may be shared across workloads. For a distributed workload, this can lead to measurable inconsistencies in performance depending on where the replicas are placed. For more sensitive workloads, this variance can mean the difference between stability and getting paged.

The single-tenant option makes it possible to better isolate workloads that cause problems for their neighbors and/or are more sensitive to such problems. This can be used as a stopgap until the specific regression problem is resolved and consistency is restored, or kept permanent for load patterns that aren’t worth supporting in a shared environment.

Efficiency

The second effect is fleet efficiency, and it is the one people expect to go the wrong way. Dedicating a node to a single task consumes that node’s slack, so the naive expectation is that utilization falls. In aggregate, it can move the other way, for a reason related to how we dealt with poor isolation before single tenancy existed.

To protect sensitive workloads on shared hosts, the platform used anti-affinity placement, scattering them across the fleet to reduce colocation of the same workload on the same nodes. Spreading fights bin-packing: it strains capacity, because a host kept partly empty to honor an anti-affinity rule cannot be filled by a workload that would otherwise have fit there. When isolation-sensitive workloads move to dedicated nodes, the need for spreading and fragmentation goes away, and the remaining multi-tenant fleet can pack more densely.

Another source of fragmentation is services that are sensitive to disruption. These are typically stateful services where recovery is expensive. As these services become more common in a given population, they can lead to nodes being held up by pods with strict disruption requirements. This can lead to an inability to defrag and recompact the pool during off-peak times. The symptoms and resulting drop in fleet efficiency caused by disruption-sensitive workloads mirror what’s observed for workloads with anti-affinity rules.

Depending on the magnitude of the anti-affinity and/or disruption budget problem, moving such workloads to single-tenant pools can result in equivalent or even improved placement when all costs have been accounted for. It also becomes easier to quantify the cost to end-users. Rather than trying to attribute fleet-wide fragmentation to individual workloads, we can directly quantify waste as single-tenant node overhead.

The theory and the practice diverge here, and it was the practice that decided it. In principle, the efficiency ceiling is higher for multi-tenant bin-packing: pack enough tasks together, and a host fills almost completely. We do not reach that ceiling. AWS offers a discrete set of instance sizes, so a large shared host rarely matches the exact sum of the requests we place on it, and the gap between a host’s capacity and what we can actually pack onto it is a bin-packing hole we cannot fill. At the granularity of large instances, those holes are sizable. Right-sizing an instance to a single workload, picking the closest fit from many shapes instead of draping many workloads over one big host, captured more of the available efficiency in our experience than the higher theoretical ceiling ever delivered. The clean theory favored multi-tenancy; our measured fleet favored right-sizing.

Container-host maintenance

The third effect is operational, and it shows up in how we keep the fleet patched. A multi-tenant host is hard to take down for maintenance precisely because it is full: draining it means relocating many tenants at once, several of which carry disruption budgets that limit how fast they can move, so a routine patch becomes a careful, coordinated evacuation. A single-tenant host runs exactly one task, with consolidation and defragmentation turned off, so draining it means relocating only the one workload, and we can automatically handle that when a new AMI ships. The remaining multi-tenant hosts are easier to maintain, too, because the workloads that were most sensitive to disruption are no longer stranded on them.

What it changes, and what it doesn’t

Single-tenant mode is a configuration, not a migration. A team moves a workload onto a dedicated node, and back, with a labeling change, without rewriting anything and without leaving Titus. That turns the isolation decision into a reversible setting, which is the opposite of the one-way door between Titus and EC2 that teams used to face. A team can try single tenancy for a quarter, measure it, and step back if the trade-off isn’t worth it for them.

What does not change is the contract. The Compute/Platform team still selects the instance type, patches the host, and sizes and scales the fleet underneath. The application team still uses the Titus API, the same delivery pipeline, and the same observability. Single tenancy is a new point on the isolation axis offered within the existing managed experience, not a separate product that the team has to learn and operate.

When a dedicated node is the wrong call

A dedicated node is a real cost, and single tenancy is not free isolation. Two things are worth weighing before reaching for it.

First, a single-tenant task does not get the whole machine; it gets the machine minus overhead. Every Titus container host, dedicated or shared, runs the platform sidecars, system services, the kubelet, and the host environment itself, and that overhead is a roughly fixed slice of CPU and a few gigabytes of memory. On a shared host, that slice is paid once and amortized across the many tasks running there. On a dedicated node, a single task handles it. Because the cost is fixed rather than proportional, it is punishing as a fraction of a small instance and negligible on a large one: a small dedicated node can lose tens of percent of its resources to overhead, while on the largest shapes it rounds to a couple of percent.

Second, because a task is placed on the smallest instance that fits it, there is usually a gap between what the task asked for and the next available shape, and on a dedicated node, that gap is pure waste, with no neighbors to soak it up. The two effects compound: fixed overhead plus the best-fit remainder set a ceiling on how much of a dedicated node a workload can actually use, and that ceiling is lowest exactly where people are tempted to reach for isolation casually, on small instances. However, this sizing waste can be reduced through coordination with the service owner as needed.

The guidance follows from the arithmetic. Single tenancy pays off for large, long-running, performance-sensitive, disruption-sensitive workloads, sized close to a node shape and run as fewer, larger replicas rather than many small ones. It is the wrong call for small, bursty, or highly elastic workloads, where the overhead fraction is high, the dedicated node sits mostly idle, and multi-tenant packing is both cheaper and faster to scale. This is also why the mode is allowlisted rather than a self-serve checkbox: the platform can keep workloads on the side of that line where they belong, and leave the ones that would gain little from a node of their own on the shared fleet, where they are cheaper to run.

Who’s using it

The first adopters are the workloads that motivated the work: real-time data infrastructure. Managed Apache Flink stream processing, streaming data connectors, and telemetry and logging pipelines were both the most frequent sources of network contention and the most damaged by it, because their recovery is expensive and their disruption budgets are tight. These are stateful, high-throughput jobs where a few seconds of starvation can cascade into minutes of catch-up, and where the platform’s own repacking used to be a source of disruption in its own right.

Moving them onto dedicated nodes removed the class of problem rather than mitigating it. The teams running them report stable operation, having previously seen intermittent, hard-to-attribute interference, and their jobs are no longer relocated by fleet repacking, removing a second, self-inflicted source of disruption. Latency-sensitive serving paths and a handful of GraphQL and edge services have followed for the same reason: a predictable tail matters more to them than a few points of bin-packing efficiency.

This is an ongoing rollout, not a finished one. We are widening the set of eligible workloads as we learn, using the largest production jobs as the stress test for the platform’s scaling assumptions, and watching the control-plane and EC2 launch-rate metrics as the count of small nodes grows. We have also been running periodic stress tests to exercise scaling ahead of real user workload, and making improvements to autoscaling and scheduling logic along the way.

What’s next

We are not done on either side of the tradeoff. On single tenancy, we are reducing launch latency with smarter pre-warming of common shapes, broadening the allowlist as scaling data holds up, and extending the model beyond long-running services as cost attribution and capacity stories mature. On the multi-tenant side, which remains the right default for most workloads, we are addressing the dimensions that hurt the most. The largest is the network: replacing HTB’s global-lock shaping with an eBPF-based Expected Departure Time approach that rate-limits by timestamping packets rather than contending on a lock, and pairing it with a lower-overhead container network device. We are also evaluating hardware cache and memory-bandwidth partitioning (Intel RDT and AMD PQoS) to give the kernel a real lever on the last-level cache, hardware GPU partitioning (NVIDIA MIG) for accelerator isolation, and continued work on the cgroup v2 IO controller for disk. This isolation benchmark lets us rank these by return on effort rather than by intuition, and show customers, in the numbers, that the shared fleet is getting quieter over time.

Single-tenant Titus is also a step toward something larger. Netflix’s Platform teams offer two ways to get compute: virtual machines on EC2 & containers on Titus. We want to replace that with a single Compute API that takes intent rather than mechanism into account: how much isolation a workload needs, how quickly it must launch, how long it can run without disruption, and what business domain it belongs to. The platform then makes placement and runtime choices, and can make globally optimal ones because it sees the whole picture.

Conclusion

Multi-tenancy is the right default for a container platform, and we will keep making it better for everyone. But the right default is not right for everything. Single-tenant Titus gives the workloads that need it the isolation of a dedicated VM, provisioned and patched by our existing node management, under the same API and the same managed experience, without taking away the freedom to change their mind. We can now point to the difference rather than describe it: a tail that stops moving under a noisy neighbor, a fleet that packs more densely once spreading is retired, and hosts that are faster and safer to patch. For the streaming and data systems at the heart of Netflix’s real-time platform, a room of one’s own turned out to be the feature that mattered.

Single-tenant Titus: VM-level isolation on a multi-tenant container platform was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

When Jake Paul and Mike Tyson stepped into the ring in November 2024, we delivered the fight to roughly 65 million concurrent streams. On an ordinary day our infrastructure takes in more than 2 billion requests and handles upwards of 38 million real-time logging events every second. At that scale, the whole picture doesn’t fit into a single engineering team’s head or onto a single dashboard. With more Live content, Podcasts and real-time streams, our need to understand and update our systems as quickly as possible becomes imperative to provide a good user experience. Understanding the system at this scale and within the time constraints we have, often becomes a daunting exercise in finding a needle in a haystack, in a dark jungle, blindfolded.

This post is about that search for the needle.

The Four hours before a fix

Consider a regression of the kind we see in a given week. The home screen where you see the list of lists of movies, starts taking about 1.5 seconds longer to render. An alert fires. And then we go looking for what caused this issue and how do we fix it?

As it so happens, a regression on the application could be related to anywhere in the system — a client app regression from a new version deployment or A/B test rollout, a load shedding in the infrastructure, a GraphQL query responding with bad data because of a service rollout 6 hops downstream, are all suspects at this point. So, answering “what caused the issue?” needs insights from across the system, and this involves combing through layers of data in a meticulous manner.

The MELT Layer

“What caused the issue?”

Metrics, events, logs and traces, or MELT, in the industry’s shorthand, are the right raw materials for such an investigation, and each of them answers roughly a quarter of the question. We collect observability data across these four pillars, unified into a single observability data layer. This lets us observe our systems in real time, across what our users are experiencing, how our applications are doing and what our backend infrastructure and services are processing and responding to.

The investigation that follows may look like this in each layer:

The timeline of an investigation

So how long does it take to investigate an issue, figure out the root cause and fix it, so our users stop running into this issue again?

In our chosen example, four hours. Very little of it went into computing anything difficult. Most of it went into reconstructing relationships that our observability systems already knew but we had never written down anywhere a human or machine could simply read and discover.

The disagreement underneath

The instinct here could have been, around 2023, to reach for more data or better dashboards, and while both help at the margins, neither addresses what makes these investigations slow.

It’s 2026 now, so we might say, “hey, agents”, but without the right context delivered to the agents, AI systems hallucinate, and especially when teams are on-call and need a fix “now”, we cannot let inaccurate agentic analysis lead us down the wrong path. We need the right needle, not any needle.

Each of our systems is internally coherent and externally isolated. Atlas, our telemetry platform, knows applications and their time series. Radar, our alerting system, knows thresholds and firing history. incident.io knows severities and responders. PagerDuty knows rotations. Spinnaker knows what shipped and when. And our Slack history knows, in a form no query engine can reach, which engineer solved something very similar last quarter and what they found.

The difficulty is that they do not agree on what they are talking about. A thing that is an application in one system is a service in a second and a cluster in a third, with names differing by a suffix, a casing convention, or a rename only a couple of people remember (SMEs, ring a bell?). Joining across them requires a human carrying the mapping in their head, and that person becomes the bottleneck in such cross-domain investigations. The hard part of end-to-end observability is probably better understood as a data modelling problem.

What we lacked was an explicit, machine-readable agreement about the entities in our operational world.

The Ontology

This agreement has a name: an ontology. Ontology is a formal specification of types, the properties they carry, and the relationships they may have. Its unit is the triple, one fact expressed as subject, predicate, object. Consider the following example triples.

api-gateway   rdf:type       ops:Application .
api-gateway   ops:ownedBy    "Team Bedrock" .
api-gateway   ops:deployedIn "us-east-1" .

INC-5377      rdf:type       ops:Incident .
INC-5377      ops:affects    api-gateway .
INC-5377      ops:severity   "SEV 2" .

In the example here, we have already encoded six facts, and can answer in one query, a question that used to span three tools.

“Which incidents affected applications owned by Team Bedrock in us-east-1?” is a single query in place of looking up incidents, Spinnaker and Radar.

We call our ontology layer, The Observability Ontology. It acts as the schema, rule set and shared agreement that standardizes how different systems understand entities like applications, services, incidents, deployments, etc.

We built it on the W3C stack for this: RDF for the data model, OWL for the ontology, SHACL for validation, and SPARQL for querying. These standards are well specified and widely implemented, which keeps the model portable across graph stores.

The data from the ontology itself, is stored as a knowledge graph, persisted to QuipuDB, our homegrown RDF graph database, a distributed quad store based on redb, a Rust-based embedded key value db. Data here is generated by parsing raw telemetry, logs, and incident discussions into triples (subject-predicate-object facts).

Every harvest also produces a full Turtle serialization for archival, making it human-readable and diff-friendly, which has proven more useful for debugging than we expected.

Our ontology is organized into twelve namespaces, each owning one operational domain:

There are two decisions about this ontology that mattered while we built it, so they’re worth sharing.

The first is that we refused a generic Service type. “Service” means four different things across our tooling, so applications resolve to exactly one type, ops:Application, and every one of them is validated against our software catalog of roughly 49,000 entries before the entity is created. Unvalidated strings like usernames, team names, dependency names scraped from network topology, never become applications. We also cap dependency edges in both directions, because one popular service can carry thousands of reverse dependencies and would otherwise dominate the graph.

The second decision concerns what we chose to model. Incident chat carries the reasoning behind an incident. The hypotheses tried and discarded, the mitigation that worked, who knew why, is all context in the incident’s Slack channel and that knowledge evaporates when the channel goes quiet. So a 200-message, 50-person, 15-service incident channel becomes a structure, converting the operational chaos of an incident into ordered triples:

Creating determinism from non-determinism

Building this graph means using tools that are inherently stochastic. LLMs summarize discussions. Fuzzy matching resolves service names. A probabilistic engine proposes causal chains. None of these produce the same output twice with certainty, but what comes out at the end of our system is a deterministic graph that answers a SPARQL query the same way every time.

The ontology is the mechanism that creates this determinism through its pipelined approach. We think of the pipeline as a ladder that drives entropy down, where entropy means the proportion of information still trapped in unstructured form. For example, at the start of this pipeline, an LLM could say anything, but it cannot emit a triple that violates the type system. We constrain the output space tightly enough and a stochastic process yields a deterministic contract:

# The model may produce any prose. Its output MUST fit these predicates:
category  IN ["incident", "question", "discussion", "decision"]
sentiment IN ["positive", "neutral", "negative", "urgent"]

# Result — a deterministic triple:
thread:123  slack:category          "incident" .
thread:123  slack:isIncidentRelated "true"^^xsd:boolean .

There are five layers that get us down this ladder. Any single step of this ladder is non-deterministic. But in aggregate, the chain becomes deterministic. That inversion is the whole trick. We describe them below:

Enrichers: Agents as entropy reducers

We use what we call, enrichers, to reduce entropy across the knowledge graph. Dozens of enrichers run over the graph. Each one of them takes noisy input and emits typed triples, and each reads what earlier enrichers wrote. The contract for an Enricher is deliberately narrow:

class Enricher:
    async def _enrich_async(self, ...):
        raw    = await self.fetch(...)      # 1. read noisy input
        result = self.analyze(raw)          # 2. process (may be stochastic)
        graph.add((uri, RDF.type, OPS.X))   # 3. emit TYPED triples
        graph.add((uri, OPS.pred, Literal(result.value)))
        return EnrichmentResult(success=True, triples_added=count)

Three invariants that are standard across the enrichers:

• every entity carries an rdf:type,
• every URI follows a single naming scheme, and
• the catalog gates every ops:Application.

Two operational choices are worth naming here, as well. The enrichment pipeline runs sequentially — load the graph, add triples, and save.

This sequencing removes an entire category of race conditions at a cost in wall-clock time we are happy to pay. And calls out to external systems like metrics, logs, events and the rest could be best-effort, so a slow downstream API doesn’t block the graph harvest, albeit some degradation. To that end, we are also working to translate these separate systems into the ontology, such as the Metrics Ontology and the Alerts Ontology, so everything could be queried from the Ontology itself.

The Correlation Engine: finding the correlation chains

The correlation engine assimilates the vast number of signals from diverse input sources and condenses them into causal or correlatory chains of events. Today, the engine reads thirteen signal types, including log errors, metric anomalies, slow and failed traces, alerts, incidents, deployments, config changes, experiments, autoscaling, commits, end-to-end bottlenecks, and error patterns. It then searches for paths among them. Some ordered pairs are known to be causal and carry a high prior weight. For example, deployment before log error, config change before metric anomaly, failed trace before incident, and so on. Others, which show high correlation are considered correlation chains. Such chains are scored on temporal proximity and severity, inside a ten-minute window, with anything under 0.6 confidence getting discarded.

The role of the engine here is to propose with evidence, not to make conclusions. Hence, the engine stores its chains along with the graph records that were used to create them.

A 0.87 chain in the above example, is a strong hypothesis for a human or an agent to check, and the graph records the confidence alongside the claim so that consumers can understand why a chain is what it is and weigh their options to make decisions.

The knowledge flywheel

Each harvest cycle turns the same four steps, and each rotation leaves the graph richer for the next. This is the Knowledge Flywheel, leading to continuous enrichment and improvement of the context about our system, stored in the knowledge graph.

Observe collects the signals. Enrich adds typed triples. Infer finds patterns invisible in the raw data. Adapt feeds knowledge back, so that established patterns become detection rules. The compounding result of the four phases leads to a self-improving system. A causal chain discovered in one incident becomes a known pattern for the next, and the ontology gradually stops being a schema and starts being a memory.

Consider the example above. When we first see an incident in api-gateway caused due to a timeout and resolve it, we convert this knowledge into entries to the ontology. This becomes encoded knowledge. As we go through multiple cycles of this encoding, the graph starts learning patterns. After N cycles, patterns become wisdom — “api-gateway deploys cause 40% of SEV-2s” and “30% of deployment issues in api-gateway come from Connection Timeouts”. Surface this as an insight and we proactively start reducing incidents, not just react quickly to new ones.

Putting the graph to work

On top of the graph sits AutoRCA, our internal agentic AI solution, built with a set of specialist agents coordinated around one question. Walking the home page regression from the top of this post, the agentic system’s reasoning chain might look something like this:

Each specialization to the inputs is built by people who understand their respective domain deeply. That division of labour only works if the specialists can pass findings to one another without having to agree on the terms of communication or even have knowledge of each other. The ontology acts as the interface. Each specialist learns one vocabulary — the graph’s. A team writing specialized use cases for how they use traces record their findings as triples about an ops:Application, the logs specialization reads triples about that same ops:Application, and neither needs to know the other exists. A new specialist learns the same vocabulary once and can immediately read everything the other six have written.

Set against the investigation we opened with, the change is narrower than it might sound. The alert still fires at 3:30pm, an engineer still has to weigh the evidence, and the fix still has to be written by someone who understands the code. The hour or two that used to go into assembling context — finding the owning team, reconstructing what shipped, correlating information from different parts of the system, recalling that something similar happened last quarter — all that work is the part the ontology and the graph absorbs. We are early enough that the ranked hypotheses are sometimes wrong, which is why each one carries the evidence that produced it. But when the engineer now starts the investigation, they are already holding the picture that used to take half an afternoon to assemble.

If you want to build one of these

A fair amount of what we have built, we learned by getting it wrong first. Based on this experience, these are the principles we would carry into a second attempt.

• Model identity before anything else. Choose one authoritative catalog per entity kind and validate against it before creating an entity.
• Treat the ontology as code. Version it, review it, test it, and let changes arrive as pull requests. Treating the ontology as code lends itself to agentic development and lets teams scale and contribute to it.
• Separate concepts from instances. Concepts change on a human timescale and instances change continuously, conflating them makes both harder to evolve.
• Build on open standards. RDF, OWL, SHACL and SPARQL are specified, implemented many times over, and portable. A bespoke graph model ties a future consumer to one implementation.
• Constrain stochastic tools with a type system. This is the principle we would defend hardest. Models are useful precisely where determinism is impossible, and a sufficiently narrow output contract converts their output into something a query engine can trust.
• Make enrichment best-effort and ordered. A slow dependency should utmost degrade the graph, never block it. And ordering enrichers so each builds on the last buys more accuracy and that’s worth the tradeoff with parallelism that buys speed.
• Model human reasoning alongside machine telemetry. The most valuable edges in our graph came from discussion. Telemetry records what the system did, discussion records what people understood and interpreted. The reasoning is more valuable to store and serve as wisdom when we need it.
• Keep humans as the approvers. Let agents propose fixes, enrichers, and ontology changes, and route every proposal through a review process with humans in the loop.

What comes next

The graph makes three things possible. First, automatic root cause analysis, where a regression arrives with a ranked set of graph-backed hypotheses attached. Next, automated remediation, where a sufficiently confident diagnosis carries a proposed action like a targeted rollback, a traffic shift, a code change. And eventually infrastructure that heals itself, acting on a predicted failure before members feel it.

Alongside this, we plan to integrate the Observability Ontology with Upper, the domain-modeling language at the heart of Netflix’s Unified Data Architecture (UDA). Upper lets teams describe business and system domains as interconnected, machine-readable models and preserve their meaning and constraints as those models are projected into concrete schemas and systems. This integration would connect operational concepts such as applications, alerts, deployments, and incidents to the broader set of domains modeled across Netflix, giving engineers and agents a shared semantic layer for querying, validation, and reasoning. We will begin with a proof of concept that models incidents in Upper and use what we learn to determine where the two systems should converge.

Acknowledgements

None of this would exist without the teams who share their data, expertise, and patience with an early system. We want to specially acknowledge the efforts of our own team members who made this possible from very early on in this lifecycle, Kaela Marsheck and Aditya Gaur. We are also especially grateful to our collaborators in Netflix’s ontology working group, Peter Royal, Alex Hutter, and to Alexandre Bertails and members of the Unified Data Architecture (UDA)/Knowledge Foundations team for sharing Upper and their domain-modeling expertise, and for helping us explore how the Observability Ontology can become part of a broader semantic foundation at Netflix.

Ontology-Driven Observability: Building an End-to-End Knowledge Graph at Netflix Scale was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Authors: Nick Rutigliano, Argha C, Samartha Chandrashekar

Open Netflix, and the home screen is assembled for you: which rows appear, the order they appear in, the titles within them, and the artwork for each title are chosen by machine learning models trained on how hundreds of millions of members watch. Personalization is one of the most widely used things we build, and members feel it every time they open the app. Like a growing share of Netflix, it runs on GPUs, which train those models and increasingly serve them live.

Those GPUs have to stay healthy for that experience to hold up, and keeping them healthy is harder than it sounds. A single large training job can run for weeks across hundreds of GPUs spread over many machines. It runs as a gang: the ranks move in lockstep, exchanging data every step, so the job advances only as fast as its slowest participant and stops entirely if one participant dies. Every GPU, every high-speed network adapter, and every interconnect must keep working throughout the run. At the scale at which modern machine learning runs, individual GPUs and network links can fail during a run, so the underlying platform must be prepared for this and handle it without a person in the loop.

This post describes how we keep our GPU infrastructure healthy for training and inference. We cover how we detect failing hardware before and during jobs, how we automatically remediate a bad machine, and how we let a long job survive the loss of a machine. These training workloads run on our modern Kubernetes-native Compute platform.

Why GPU reliability matters

Netflix aims to entertain the world, and machine learning has been part of how we do that for years, from the recommendations that help members find something to watch to the systems that help produce and promote titles. As generative AI advances, we are investing in it on both sides of the business: for members, through better recommendations and higher-quality promotional assets; and for storytelling, through creative tools for our production partners.

All of this runs on GPUs, for both training and inference, and GPU capacity is among the fastest-growing and most expensive parts of our infrastructure. That makes reliability a business concern on two fronts.

• Availability: a training run that loses a machine, or an inference service that cannot get healthy capacity, delays work that members and creators depend on.
• Cost efficiency: a GPU sitting idle behind a slow neighbor, or a whole machine pulled from the pool because one device is degraded, becomes capacity we paid for and are not using. Higher reliability and higher utilization mean more useful work per GPU, which is equivalent to spending less to do the same work.

Inference is a growing share of that GPU footprint as more of these AI features reach members, so the reliability and efficiency that matter for multi-week training runs increasingly matter for always-on serving too.

Why basic health checks are not enough

We start from the baseline provided by the cloud provider. Cloud providers run instance status checks on virtual machines: a system status check for the underlying host’s power, network, and hardware reachability; an instance status check to verify that the operating system is up and reachable; and a check on attached storage. It emits scheduled maintenance events when it plans to retire or reboot a host, and it can automatically recover an instance that becomes impaired. We use all of these, and the maintenance events feed one of our own checks. They catch a machine that is down, unreachable, or slated for maintenance, and they stop at the instance level. They confirm the box is up and reachable. They do not verify that each GPU, its on-chip memory, its interconnect, and its network fabric are healthy enough to support distributed training.

The next step most teams take is a GPU-level check to confirm that the driver responds and that the GPUs are visible. That catches a dead or missing GPU, but misses the cases that take the most time.

• A GPU can answer a status query and still be faulty once a workload touches it. A fault in on-chip memory or in the firmware that manages the device can remain invisible until a job allocates memory or runs sustained compute, at which point the job fails or slows.
• A high-speed network adapter may be present but show fewer links than expected. Collective operations still run, at a fraction of the bandwidth, so the job does not crash. It crawls, and the cause is easy to miss.
• A check can fail in ways that leave a faulty machine looking healthy. The diagnostic tool can hang when the driver hangs, and a naive check that waits on it hangs too. A binary that the check depends on can go missing after a driver upgrade, causing the check to stop running and report nothing. A check that can silently stop running is worse than no check, because it gives false confidence while the machine keeps working.

Because a job runs as a gang, any one of these on any one machine can degrade or stop the whole job, and none of them shows up in an instance-level check or a driver-responds check. The health system has to assume the hardware can lie and the tools that inspect it can break.

Our approach

We treat reliability as four layers, each catching what the layer below cannot:

1/ Node and hardware health: find bad GPUs, network adapters, interconnects, memory errors, and thermal issues, then taint or recycle the machine.

2/ Scheduler and platform: retry at the task & job level, validate a machine before it takes work, and drain & recycle machines safely.

3/ Training loop: checkpoint and use elastic workers so a job survives losing a machine.

4/ Observability: metrics, service-level objectives, and failure attribution across the other three.

The first two layers keep hardware problems away from jobs. The third accepts that some loss is unavoidable and makes it cheap. The fourth tells us whether the first three are working.

How the health checks work

Checks run in two tiers. Device-level checks look at each GPU. A node-level agent, built on the open-source Kubernetes node-problem-detector, runs on every GPU machine every 30 seconds and turns check results into conditions on the Kubernetes node, the same mechanism Kubernetes uses to mark a node NotReady. Controllers watch those conditions and decide whether the machine can take on work.

GPU device checks are built on NVIDIA’s DCGM for driver-reported error codes, uncorrectable memory errors, interconnect integrity, and thermal state, and networking adapter checks inspect the host state for the count and link state of the Elastic Fabric Adapter interfaces that carry cross-machine traffic. A few of these checks deserve detail.

• Failing closed: if the driver or the diagnostic tool is unresponsive, the check times out and marks the machine unhealthy. A check that cannot reach the hardware reports a problem rather than passing, so a wedged driver takes the machine out of rotation rather than letting it hide behind a hung tool.
• A fallback path: driver error codes are the strongest signal that a GPU has faulted, but the tool that reads them can itself be broken in certain failure modes! When it does, we read the kernel log dmesg directly for those codes, so no Xid or SXid errors are missed.
• Our own fatal-error map: vendor driver error catalogs lag newer GPU generations and do not always match what we see in production, so we keep our own mapping of which codes are fatal for each GPU family. A code that means reset-the-GPU in one generation can be benign in another, and one GPU generation emits a burst of normally-fatal code on every healthy boot, which we learned to treat as noise rather than drain fresh machines.
• No silent no-ops: a check fails when a required dependency is missing. This one came from an upgrade that dropped a binary the network-adapter check relied on. The check quietly stopped running, and machines with the wrong number of adapters shipped for days before the slow jobs were traced back to them. Now a missing dependency fails the check.

Turning a bad machine into healthy capacity, automatically

When a check fails, the machine’s node condition changes, and controllers take over. We distinguish two levels. A problem condition blocks new work from landing while jobs are running, because pulling a job off a machine is disruptive, and many problems clear on their own. If the condition persists beyond a tolerance window, it escalates to a failure condition, which evacuates the machine, drains its tasks (overriding the normal disruption limits to move quickly), terminates it, and replaces it.

Karpenter node auto-repair is used on our compute stack. We have customized it with a policy and a controller that maps each node condition to the duration we tolerate it before acting, so a fast-fatal error is actioned within minutes. At the same time, a flaky signal gets a longer window, and a cap limits how much of a cluster we will disrupt at once, so a bad rollout of the checks themselves cannot cascade into tearing down a whole cluster. We plan to upstream this feature to all Karpenter users.

Not every fault requires a full replacement, and replacement is the slowest way to restore capacity. We match the action to the fault. Killing the offending job clears a software wedge. Resetting a single GPU clears many transient device faults. Rebooting the machine clears the firmware state that has drifted over a long uptime. Terminating and replacing is reserved for hardware that fails a test we trust. For that last case, we record the GPU’s immutable hardware identifier, which stays with the physical device across reboots and relaunches even when the instance changes, report it to the cloud provider, and keep our own list, so a device that fails repeatedly does not quietly rejoin the pool under a new instance.

One subtlety sits between detection and scheduling. A freshly checked machine was handed to the scheduler just as its checks were still settling, so a job could land on it a moment before a slow check flagged it. We now hold capacity until the checks have run and settled, which closed that window.

Validating a machine before it takes work

A machine that just booted is not trusted until it proves itself. Before we release its capacity, it passes system checks, confirms the GPUs and their topology are present, and runs a round of GPU diagnostics. We then wait for the health checks to run twice, because the longer checks, such as counting network adapters, take more than one cycle to fire, and releasing after a single pass would let work land before those checks are reported. Only then do we remove the startup taint. On teardown, we run deeper diagnostics and record the GPU identifiers, which feed the same bad-hardware list, so hardware that failed once does not silently cycle back into service.

Letting long jobs survive machine loss

Even with all of the above, some machines are lost mid-job, so we make that cheap. Jobs checkpoint their state, and on a lost machine, they resume from the last healthy checkpoint rather than restarting from the beginning of a multi-week run. Workers are elastic and retryable, and the scheduler automatically retries platform-caused failures.

Attribution is what makes automatic retries safe. If we retried every failure, we would mask genuine bugs in a job and burn GPU hours looping on an error the platform cannot fix, so we classify each failure as platform or application before retrying. One example of why this matters: a class of timeouts in NCCL, the collective communication library GPUs use to exchange data, appeared to be a network fault and was traced to data-loading stalls at high worker counts. The GPUs and the network were fine, and the fix was in the data pipeline. Treated as hardware failures, those timeouts would have cycled healthy machines without fixing the cause.

Measuring whether it works

We track GPU reliability along three service-level objectives, each split by cause, so a miss points at the layer responsible:

• Availability: the amount of GPU capacity available for scheduling at any given moment.
• Job-start success: whether a job starts promptly once its capacity is available.
• Job success: whether a job finishes without a platform-caused failure.

Each objective is split into hardware, driver, platform, and user causes. Hence, an availability dip from bad hardware reads differently from one caused by a scheduler change or a user misconfiguration. We optimize for goodput, the useful work the fleet completes, because a job held back by one slow GPU can look healthy on an uptime chart while getting almost nothing done. Specific targets for each objective are provided here for illustrative purposes.

A closed remediation loop

These pieces form a loop. Hardware is validated at allocation, before a job starts, with a short collective communication run and pairwise network tests. Hence, any machine that fails within the first minute fails before the job is admitted. During the job, we watch passively for driver errors, memory errors, thermal events, and more. We view each job as a distributed system, which lets us identify a single slow rank dragging the rest, rather than only whole-machine failures. When something is wrong, remediation is automatic and graded. The loop is the same whether a job runs through our batch scheduler, Kueue, or Ray.

Sharing GPUs safely

Not every workload needs a whole GPU, and leaving small inference jobs on dedicated GPUs wastes capacity. We partition GPUs with MIG for memory-level isolation between tenants, so a fault or a memory hog in one partition does not affect another. We also reconfigure those partitions without restarting the machine, so that a pool can shift between many small slices and a few large ones as demand changes. GPU and network adapter allocation is device-plugin-based today, to move to Kubernetes Dynamic Resource Allocation once it is supported in Karpenter. This matters because a GPU talking to a network adapter or working with a CPU on the wrong PCIE root of the machine incurs a bandwidth penalty.

Device-level allocation also changes how we handle a single bad GPU. With DRA support, instead of losing the whole machine, we could isolate the one faulty device and keep the others serving, which matters most for inference, where many small jobs share a machine. And because allocation is expressive, a job can request the capabilities it needs, such as memory or tensor-core support, and the platform matches a suitable GPU type, so users are not pinned to a model they had to choose in advance.

Results

Moving from basic checks and manual draining to layered checks and automatic remediation changed two things users feel. More of the GPU fleet is available for scheduling at any time because bad machines are caught and cycled quickly, rather than sitting in the pool or waiting on an on-call engineer. And fewer jobs fail due to platform issues because most faults are caught before a job lands, and the rest are automatically remediated and retried. Graded remediation also returns capacity in minutes for faults that only require a job kill, a GPU reset, or a reboot, rather than the longer wait for a replacement machine.

Since the above improvements have landed, we’ve seen:

• Consistently achieving >99.95% availability of our GPU nodes
• Job success rate has gone from ~98% to above 99.5%

What is next

We are extending active checks to cover additional failure modes under load, widening burn-in so more classes of failures can be cleared before a user encounters them, and making remediation more granular so more error types are cleared in place rather than requiring a full node replacement loop for certain Xid errors. Additionally, we want to do more periodic diagnostics between jobs.

We are working to make placement more topology-aware using Kubernetes Dynamic Resource Allocation (DRA) once it lands in Karpenter, for example, by aligning on Compute/IMEX Domains, PCIe roots, and NUMA nodes.

We also plan to run a custom Linux scheduler built on sched_ext. This enables us to prioritize processes during collective coordination, such as when GPUs exchange results before the next round of compute, thereby reducing overall GPU idle time by establishing a fleet-wide priority for these tasks. The effect is small on one machine but grows across hundreds of machines, where a few percent of throughput amounts to a large amount of capacity. Additionally, it allows for prioritizing data-loading processes to minimize bottlenecks that occur when data is not loaded onto GPUs quickly enough to maintain steady progress.

While we already monitor throttling events, GPUs do not have to report throttling for it to hurt a job. One running ten to fifteen degrees hotter than its peers, even within healthy limits, sets the pace for the gang, so we can treat a large temperature gap as a fault, stop the job, and move it to cooler GPUs.

Additionally, we plan to expand our NCCL RAS (reliability, availability, serviceability) monitoring to identify NCCL issues sooner, such as mismatched collectives or hangs, to more quickly resume work.

Closing

Hardware fails, and at this scale, it fails often enough that the platform has to treat failure as normal. Detecting problems before and during jobs, automatically remediating machines, and allowing long jobs to resume after a loss lets teams run large-scale training and inference without babysitting the hardware. If you like these kinds of problems, we are hiring.

Reliable GPU compute for large training & inference jobs was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

Organizations that have been around for a while usually run two identity systems side by side. One belongs to the cloud provider: IAM roles, instance profiles, execution roles. The other is your own, and it is the one your internal services actually check when they decide whether to answer a request.

On infrastructure you build yourself, you can bootstrap your own identity however you like. On managed compute you cannot. The provider hands your process a cloud identity and nothing else.

This post describes how we close that gap for Apache Spark workloads running on Amazon EMR. A workload that starts with only an AWS identity has to end up holding a first-class internal identity. Doing the exchange is straightforward. Making it trustworthy is the part that took the design work. Very little of what follows is specific to Spark or to EMR.

Two identity systems

Internal service-to-service authentication at Netflix runs on a private PKI called Metatron. Every workload gets a short-lived X.509 certificate, and services authenticate each other with mutual TLS. A workload cannot simply ask for a certificate. It is issued only after the identity service has satisfied itself that the requester really is the workload it claims to be. That verification step is called attestation. In each environment we support (VMs, containers, functions) attestation rests on some environment-specific proof that the platform can check on its own.

The second concept is the Data Project, which is our unit of ownership for data. A Data Project owns tables, has a set of authorized users, and has an identity of its own. Jobs run as the Data Project rather than as the person who launched them. That is what keeps table-level access control and audit consistent across scheduled and interactive use.

So the problem is this. A Spark job on managed compute starts with an AWS execution role and no internal identity. Everything it needs from inside the company (reading an encrypted column, evaluating table ACLs, holding an interactive session open longer than a short-lived token lasts) requires the internal one.

One identity, one role

The design rests on a decision made outside the Spark stack entirely. Each Data Project identity maps 1:1 to a dedicated IAM role, and the service that owns Data Project metadata records that mapping.

The mapping is what makes translation possible. It lets a statement in the provider’s vocabulary (“this process is running as role R”) become a statement in ours (“this process is workload W”). Without it there is nothing to translate into, and no amount of cryptography helps. The hard part of bridging two identity systems is rarely the protocol. It is committing to a mapping and then keeping it authoritative.

One practical objection arrives immediately. A single AWS account cannot hold tens of thousands of IAM roles, and we expect data projects in the order of ten thousands. We handled this by sharding data project roles deterministically across a small pool of dedicated accounts, which scales well past the projected number of projects and has the useful side effect of keeping workload roles on the far side of an account boundary from the control planes that launch them.

Components

Five components take part.

The control plane is the only service allowed to launch Spark jobs. It resolves the Data Project’s IAM role, builds and signs a workload metadata payload, and submits the job with that role as the execution role. It holds a signing key issued to it for this purpose alone.

The Data Project service is the authority for the identity-to-role mapping.

The Identity service receives attestation requests, verifies them, and issues certificates.

A Spark plugin provides the hook. Spark 3.0+ added a plugin interface with driver-side and executor-side components that are initialized while those processes are bootstrapping.

AWS STS acts as a notary, which is not its usual job.

Step one: the control plane makes a signed claim

When a job is submitted, the control plane validates the caller, looks up the Data Project’s IAM role, and builds a small payload describing the workload it is about to launch. The payload names the application identity, the mapped role, and the environment, stack and detail that the resulting credentials should be scoped to. The control plane signs the payload and passes it and the signature through as ordinary job configuration.

Two properties of this step matter. First, only one service can produce a valid signature, so the set of things that can assert “this workload is legitimate” is small and auditable. Second, the payload is a claim rather than an authority. On its own it proves nothing, because job configuration travels through infrastructure we do not fully control and anything that can read it can replay it.

Step two: the workload proves it holds the cloud identity

When the managed service launches the driver process, it does so under an OS user that has the execution role’s credentials available. The driver-side plugin initializes and uses those credentials to sign, but not send, a request to the provider’s identity endpoint (sts:GetCallerIdentity). The result is a short-lived pre-signed URL.

That URL is a transferable proof of possession. Anyone can fetch it. Only the holder of the role’s credentials could have produced it. And the response comes from AWS rather than from the workload, stating which role signed the request. The workload cannot lie about the answer because the workload does not supply the answer.

The pattern is not new. It is the same idea behind AWS IAM authentication in HashiCorp Vault and in AWS’s own function attestation flows. It generalizes well: any environment that gives a process cloud credentials and nothing else can still produce a verifiable statement about its own identity.

The plugin sends one attestation request carrying both artifacts, the pre-signed URL and the signed metadata.

Step three: corroboration

The identity service does four things.

1. It fetches the pre-signed URL against AWS STS and reads back the role that signed it. This is the provider’s statement.
2. It verifies the metadata signature against the key issued to the control plane. This is the platform’s statement.
3. It corroborates the two. The role AWS reports must be the role the control plane said it dispatched, for the identity the metadata names.
4. It issues certificates for that identity, scoped to the environment, stack and detail from the signed metadata.

Step 3 is the point of the whole design. Neither statement is sufficient by itself, and they fail in complementary ways. The provider’s statement is unforgeable but under-specified: it gives you a role, and a role is not a workload. It says nothing about why this process exists or what it should be allowed to be. The platform’s statement is well-specified but unverifiable on its own, for the reasons above. Put together, the signature says the workload was legitimately dispatched and the pre-signed URL says this really is it. Neither party has to trust the workload’s own account of itself.

There is a tension here that we settled deliberately, and other teams should expect to meet it. The provider’s answer identifies a role, not an application. Deriving an application name from a role name is fragile string matching, and the session identifiers the platform assigns are not under our control. We chose to take the application identity from the signed metadata and use the provider’s answer only for corroboration. That costs a network round trip and a stricter signing requirement, and it buys a much clearer trust story.

The fan-out problem

A Spark application is one driver and up to thousands of executors, created and destroyed throughout the life of a job. Attestation designed around one process per host does not survive that.

There are two defensible answers.

In the first, each executor attests independently. This is uniform, adds no new trust boundary, and grounds every process’s identity in the same provider-verified proof. The cost is amplification. One large job can produce thousands of STS calls and thousands of attestation requests in a burst, which looks a lot like an attack and creates a hard dependency on provider-side rate limits.

In the second, executors inherit credentials from the driver. The driver attests once and distributes credentials to executors over Spark’s internal RPC, which we configure with authentication and AES-GCM encryption so that only processes in the same application can take part. Load on the identity service stays constant. The cost is a second trust boundary and a driver that is now a credential distribution point.

Considering our scale at Netflix here we went with the second approach. The general lesson is that identity amplification is a capacity question, and it is better to answer it on purpose than to have it answered for you during an incident.

Lifecycle

Certificates are short-lived by design, so bootstrapping is only half the work. On self-managed hosts a system service handles renewal. Inside managed compute there is no equivalent hook, so the driver JVM runs a timer that re-attests on a fixed interval, well ahead of expiry. Credential material is written to a temporary directory readable only by the process owner and removed when the process exits.

Executors do not refresh. They are usually short-lived, and an executor that somehow outlives its credentials can exit and be replaced, which is cheaper than making every executor a renewal client.

The rule worth carrying into any similar system is that attestation has to be a repeatable operation rather than a bootstrap step. Anything that can only happen once at process start will eventually be the reason a long-running job dies nine hours in.

Why this generalizes

If you run your own identity system and are moving workloads onto managed compute, the specific services here matter less than the shape of the solution.

1. Anchor on one exchangeable primitive. A single authoritative 1:1 mapping between your identity and a provider identity is what makes translation possible. The rest is plumbing.
2. Require two independent claims and corroborate them. One from the provider, unforgeable and under-specified. One from your control plane, well-specified and replayable. Trust the intersection and never either alone.
3. Keep the signer scarce. If exactly one service can make the platform’s claim, your trust story fits in a sentence.
4. Hook the layer you still own. On managed compute you rarely control the host or its init system, but you almost always control the runtime: a plugin, an agent, an entrypoint. Attestation belongs there.
5. Decide the amplification policy on purpose. Distributed engines multiply every per-process operation by their parallelism.
6. Make attestation repeatable and credentials short-lived. Renewal is a requirement, not a follow-up.

What makes the result trustworthy is that no single participant issues an identity by itself. Not the workload, not the control plane, not the provider. And the workload, which is the party in the weakest position to be trusted, is never asked to vouch for itself.

Acknowledgements

Thanks to Amer Hesson for the Data Project abstraction, Nick Siow for the Data Project IAM role sharding, and Doug Clark for Metatron attestation.

Trading a Cloud Identity for Your Own: Workload Attestation on Managed Compute was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Stratum is Netflix’s internal serverless media processing platform, shaped by over a decade of experience developing large-scale distributed systems. It powers a wide range of media workloads — including encoding, packaging, inspection, and quality scoring — behind playback, advertising, studio workflows, AI embeddings, and more.

Engineers develop Stratum functions by packaging application code and OS dependencies into OCI images. Stratum then schedules these containers as batch jobs on Netflix’s Titus compute platform, with upstream services triggering function executions.

Every day, Stratum manages hundreds of thousands of containers across bursty, high-throughput, and latency-sensitive workloads. As Netflix’s core media processing platform, this load needs to scale as new content formats enter the catalog, such as podcasts, live events, sports, ads, or regional ingests like TF1 in France.

To request resources for these functions, owners specify their CPU, memory, disk, and network limits as YAML configuration, as shown below:

functionName: VideoEncoder.encodeV1
resources:
  numCpus: 8
  memoryInMB: 18000
  diskSizeInMB: 50000
  networkInMbps: 1000

Maintaining optimal values manually is challenging amidst evolving Netflix infrastructure, EC2 instance upgrades, and application-level changes. Consequently, configs are often over-provisioned upfront or slowly drift away from their ideal values over time. Mis-sized configurations are a universal pain of large-scale serverless platforms, causing additional cognitive load for function builders and operational costs for platform owners.

In this post, we share how we built the Stratum Resource Tuner (SRT) — an automated system that safely right-sizes running containers in production without human intervention. We will walk through real-world examples, our safety architecture, and what we learned when an automated change went wrong.

The Challenge: Over-Provisioning

When engineers write a new Stratum function, they typically copy a resource YAML from an existing function or size it for the absolute worst-case scenario. Once running in production, engineers rarely revisit these configurations until a failure occurs, prompting them to over-provision resources out of an abundance of caution.

Manual resource tuning does not scale for the following reasons:

• High Operational Friction: Downsizing requires days of manual metric monitoring and trial-and-error testing, diverting valuable engineering cycles away from other high-impact work. If a cut goes too far, engineers face production incidents — making manual tuning a high-risk, low-reward task.
• Continuous Infrastructure & Codec Evolution: Cloud providers regularly upgrade their hardware. A resource allocation that was optimal on an older hardware generation quietly becomes inefficient on newer hardware. At the same time, Netflix continuously advances media standards and improves encoding algorithms (such as AV1, AV2, and spatial audio). An allocation tuned for one codec might not be optimal when an improved algorithm is released.
• Dynamic Workload Evolution: As Netflix expands into new media types — such as streaming video podcasts, sports, interactive AI-adapted ad formats, and major broadcaster integrations like TF1 in France — media processing footprints can change unpredictably, rendering static container resource limits imprecise.
• Rising Industry Hardware Costs: Driven by massive global AI infrastructure demand, server hardware and DRAM prices continue to climb, making idle container memory significantly more expensive to waste.
• Compounding Platform Scale: At Netflix scale, even minor over-provisioning multiplies exponentially across hundreds of thousands of daily container executions, compounding wasted capacity across the entire Stratum platform.

To understand the scale of the challenge, we analyzed historical usage across Stratum’s workloads. The telemetry confirmed our hypothesis: over-provisioning was frequent, wasted compute was compounding, and the sustainable path forward required a fundamental re-evaluation of how resource configurations are managed.

Meet the Stratum Resource Tuner

To address the challenges listed above, we built the Stratum Resource Tuner (SRT) to automatically right-size Stratum containers. Our north star is to ensure that users never have to manually specify the resource configurations. We started with a handful of high-throughput workloads with relaxed latency requirements, which account for the majority (70%–80%) of resource usage.

As a quick note on terminology, we use “right-sizing” and “tuning” interchangeably in this post. “Downsizing” means decreasing resource limits, while “upsizing” means increasing them. Since our goal is to optimize for cost, this post focuses on downsizing.

Automatic right-sizing is not enabled by default. Function owners must manually opt in on a per resource type basis (e.g., CPU, memory, network, or disk). Once enrolled, SRT operates without a human in the loop across five steps:

1. Collect resource usage metrics across all executions of that function.
2. Generate a recommendation for each resource type (e.g., CPU or memory).
3. Verify the recommendation through experiments.
4. Apply or reject the recommendation based on the results of experiments.
5. Monitor post-apply metrics, watching for anomalies after applying the recommendation.

“Recommendation” is a key construct in SRT and serves as the primary unit of operation. As SRT executes its steps, it promotes the recommendation through several stages:

• CREATED: A scheduled workflow generated the recommendation.
• EVALUATING: Recommendation verification is running, and SRT is analyzing the results.
• APPLIED: All experiments passed, and SRT automatically rolled out the recommended configuration.
• REJECTED: At least one experiment failed, leaving the function configuration untouched.
• MONITORED: SRT monitors the recommendation during a post-apply soak period to verify stability.

The sections below examine each of these stages in detail. To illustrate the flow, we will follow the real-world lifecycle of the VideoEncoder.encodeV1 function — a high-throughput video encoding function whose over-provisioned memory and CPU footprints were successfully tuned.

Note that all operational thresholds and safety limits discussed below are fully configurable per function to accommodate varying workload characteristics and risk tolerances.

Metrics Collection

To make these recommendations possible, we need a solid metrics collection mechanism. Every Stratum container reports its function execution metadata, infrastructure dimensions, and resource utilization (CPU, memory, disk, and network) at sub-minute resolution.

These metrics flow through Keystone, Netflix’s real-time event streaming platform, and land in two separate datastores:

• Execution summaries store: Stores execution-level summary metrics indexed in Elasticsearch for fast point lookups, serving as our short-term datastore for querying and inspecting individual runs.
• High-resolution long-term store: Stores raw, high-resolution usage metrics in Iceberg tables. While slower to query, it serves as our long-term datastore, holding months of history for resource utilization trend analysis.

One record is written per execution, which is represented as a span. Here is how these records look for a single span of our VideoEncoder.encodeV1 function.

In Elasticsearch, the record captures the high-level summary of the completed span, identified by tags.stratum.spanId:

{
  "tags.stratum.spanId": "38536313f65f060a",
  "tags.functionName": "VideoEncoder.encodeV1",
  "tags.stratum.phase": "run",
  "tags.stratum.success": true,
  "tags.stratum.retriedWithMoreResources": false,
  "tags.stratum.function.execDurationSecs": 26,
  "tags.cpuCount": 8,
  "tags.memoryInMB": 18000,
  "tags.diskInMB": 50000,
  "tags.networkInMbps": 1000,
  "tags.titusTaskId": "06e1a7c0-2502-4a54-905f-b13a1e786107",
  "tags.instanceType": "r7a.24xlarge",
  "tags.nf.region": "us-east-1",
  "ts": 1780743028000
}

In Iceberg, the record contains the same metadata, but adds the sub-minute resource utilization timeseries tracked during that span:

{
  "span_id": "38536313f65f060a",
  "function_name": "VideoEncoder.encodeV1",
  "phase": "run",
  "success": true,
  "start_epoch_secs": 1780743002,
  "end_epoch_secs": 1780743028,
  "runtime_secs": 26,
  "cpu_requested": 8,
  "memory_mb_requested": 18000,
  "disk_mb_requested": 50000,
  "network_mbps_requested": 1000,
  "titusTaskId": "06e1a7c0-2502-4a54-905f-b13a1e786107",
  "instance_type": "r7a.24xlarge",
  "tags.stratum.retriedWithMoreResources": false,
  "env": "prod",
  "resource_usage": [ 
   { "cpu_usage": 1.13, "memory_mb": 3701.37, "disk_mb": 20120.4, "net_in_mb": 221.09, "net_out_mb": 0.12 },
   { "cpu_usage": 2.25, "memory_mb": 3584.09, "disk_mb": 25410.1, "net_in_mb": 0.02, "net_out_mb": 0.05 },
   { "cpu_usage": 4.16, "memory_mb": 6750.79, "disk_mb": 30400.5, "net_in_mb": 165.90, "net_out_mb": 0.38 },
   ... 
  ]
}

Recommendation Generation

A scheduled Conductor workflow runs daily, reading the past month of usage data from Iceberg and generating recommendations for every function and resource type. The workflow also generates recommendations for functions that have not opted into automatic right-sizing to preview potential savings.

Each recommendation is saved in the recommendations datastore, enabling us to query and filter recommendations by function, resource type, or status on an ad hoc basis.

For each function and resource type, the workflow applies the following generation logic:

• Filter by workload type: We only tune high-throughput workloads. This focuses our efforts on right-sizing, which yields the most meaningful fleet-wide savings.
• Aggressively tune CPU and network: We target p99 usage with no extra headroom. CPU and network are compressible resources; if a container occasionally hits these limits, the function might execute slightly slower.
• Conservatively tune memory and disk: We target p99.9 usage and include a safety buffer. Running out of physical memory or disk space causes hard failures — such as Out of Memory (OOM) or Out of Disk (OOD) crashes — making a protective margin essential.
• Apply safeguards to changes: We discard recommendations if the absolute reduction is too small to justify operational effort. For memory and disk, we clamp the maximum reduction allowed in a single cycle to 15% — a safety limit introduced after learning from early production runs.

The workflow creates new recommendations in the CREATED state. If a recommendation already exists in this state for a given function and resource type, SRT overwrites it with updated calculations. Once a recommendation advances past the CREATED state, its recommended_value and stats fields become immutable. Subsequent daily runs generate a new recommendation with a unique ID instead of modifying the existing record.

Here is how these recommendations look across resource types in their initial CREATED state for our VideoEncoder.encodeV1 function:

• CPU recommendation

{

  "function_name": "VideoEncoder.encodeV1",
  "env": "prod", "resource_type": "CPU",
  "current_value": 8, "recommended_value": 4,
  "status": "CREATED",
  "resource_cost_usd_per_hour": ***,
  "stats": {
    "usage": { "p99": 4.16, "p999": 4.57 },
    "execution": {
      "total_fn_executions": 748762182,
      "success_fn_executions_pct": 99.75,
      "total_fn_exec_runtime_hr": 88102723
    },
    "window": { "start": "2026-05-05T00:00:00Z", "end": "2026-06-05T00:00:00Z" }
  },
  "experiment_inputs": null,
  "experiment_workflow_execution_id": null,
  "outcome_message": null,
  "finalized_at": null
}

• Memory recommendation (15% reduction in one step)

{
  "id": "2edb734a-6184-4f5e-8ffa-23004bd665d7",
  "function_name": "VideoEncoder.encodeV1",
  "env": "prod", "resource_type": "MEMORY_MB",
  "current_value": 18000, "recommended_value": 15300,
  "status": "CREATED",
  "resource_cost_usd_per_hour": ***,

  "stats": {
    "usage": { "avg": 5296.43, "p99": 6750.8, "p999": 7820.3 },
    "execution": {
      "total_fn_executions": 748762182,
      "success_fn_executions_pct": 99.75,
      "total_fn_exec_runtime_hr": 88102723
    },
    "window": { "start": "2026-05-05T00:00:00Z", "end": "2026-06-05T00:00:00Z" }
  },
  "experiment_inputs": null,
  "experiment_workflow_execution_id": null,
  "outcome_message": null,
  "finalized_at": null
}

As the recommendation moves through its lifecycle, SRT populates the remaining parameters.

When proceeding to the verification stage, SRT transitions the recommendation status to EVALUATING, records the target experiment settings, and links the active workflow execution tracking the experiment.

Finally, once verification runs complete, the status changes to APPLIED or REJECTED, storing the outcome details and completion timestamp for analysis.

Retry with More Resources

We take a conservative approach to generating recommendations, prioritizing system stability over extreme efficiency. Because recommendations rely on historical usage data, they cannot always anticipate sudden spikes or novel workload profiles when functions encounter heavy, unseen requests.

To handle these unexpected spikes, we developed a fail-safe mechanism. A watcher process running in each tuned container tracks memory and disk usage during function execution. If the container exceeds the pre-set thresholds, 85% for non-cached memory usage or 95% for disk usage, the watcher flags the execution as high risk, cancels execution, and re-queues the task onto a larger container:

new_container_size = original_container_size * 1.5

This 1.5 multiplier only applies to memory and disk dimensions.

This “retry with more resources” mechanism serves as our safety net: larger container configurations are applied only when retrying tasks to self-heal from edge cases. Although these infrequent retries incur additional compute costs and latency, they allow us to keep day-to-day configurations highly optimized without having to size for rare, costly outliers up front.

For our VideoEncoder.encodeV1 function, if a heavy run causes memory usage to exceed the 85% threshold (15,300 MB) within its current 18,000 MB container, the system aborts the execution and reruns it on a 27,000 MB container.

While we prefer to trigger retries based on actual OOM or OOD events rather than pre-set thresholds, technical constraints in our current architecture make true event detection challenging. Using pre-set thresholds serves as a reliable stopgap while we develop improved detection capabilities.

SRT automatically enables the retry mechanism for any function opted into memory or disk right-sizing. Each Stratum container reports retry events, which flow into Elasticsearch and Iceberg alongside standard performance telemetry.

Recommendation Verification

Every recommendation must pass a two-stage verification process managed by a Conductor workflow before rollout. During this EVALUATING stage, SRT executes two distinct experiment types:

1. Synthetic experiment: A low-cost, low-risk test to catch obvious regressions before production rollout.
2. Production experiment: A live evaluation that validates changes across diverse production workloads to increase confidence.

A recommendation advances to production testing only after passing the synthetic experiment. When an opted-in function has multiple pending recommendations, SRT prioritizes and evaluates the candidates with the highest estimated savings first.

Synthetic experiment:

• Runs in a test environment isolated from production traffic.
• Sets up two clusters: one for the current and one for the recommended config.
• Replays identical request sets across both clusters simultaneously without splitting traffic, exposing both configurations to 100% of test requests.
• Typically runs for 1–2 hours.
• Runs to completion without early termination checks.

Production experiment:

• Runs in production on real user traffic, split three ways:
  ◦ ~90% remains on the main cluster (current production config).
  ◦ ~5% goes to a baseline cluster (current production config).
  ◦ ~5% goes to an experimental cluster (recommendation config).
• Evaluates recommendations against a similarly sized baseline cluster rather than the main cluster to preserve comparable traffic profiles.
• Typically runs from one day to one week, depending on traffic volume.
• Fails fast: cancels the experiment early if regressions are detected.

The Stratum platform executes other experiments alongside SRT experiments. To avoid over-splitting traffic, SRT experiments run at lower priority and yield resources to active, higher-priority experiments.

Traffic arrives in irregular bursts separated by multi-day or multi-week lulls. The verification pipeline accommodates these variable arrival patterns rather than expecting steady traffic.

Back to our VideoEncoder.encodeV1 memory recommendation (18,000 MB → 15,300 MB), verification is triggered with:

{
  "recommendationId": "2edb734a-6184-4f5e-8ffa-23004bd665d7",
  "syntheticExperiment": { "numberOfExecutions": 3000 },
  "productionExperiment": {
    "enabled": true,
    "maxContainers": 20, // max containers shift to baseline & experimental clusters
    "upGateTimeoutMinutes": 10080, // time to wait for traffic to start flowing
    "maxRunDurationMinutes": 1240, // max time to run the experiment once traffic starts
    "minSamples": 500 // min executions needed on both clusters before evaluating
  }
}

Both verification stages evaluate recommendations against two sequential checks, terminating immediately if either fails:

• Retries with more resources: Restricts memory and disk downsizes to a maximum 1% retry threshold over the experiment duration.
• Execution time: Compares median runtimes to prevent outlier distortion, permitting up to a 10–15% slowdown relative to baseline, depending on workload, and is configurable by function.

If a recommendation passes both verification stages, SRT updates its status to APPLIED and rolls out the configuration to the main production cluster. If a check fails at any point, SRT marks the recommendation as REJECTED and preserves the existing configuration. SRT sends notifications for both outcomes.

For VideoEncoder.encodeV1, both stages passed cleanly:

{
  "outcome": "APPLIED",
  "syntheticExperiment": {
    "result": "SUCCESS",
    "samples": { "baseline": 3000, "experimental": 3000 },
    "medianExecutionSecs": { "baseline": 59.0, "experimental": 59.0 },
    "slownessIncreasePct": 0.0,
    "retryRatePct": { "baseline": 0.0, "experimental": 0.0 }
  },
  "productionExperiment": {
    "result": "SUCCESS",
    "samples": { "baseline": 50361, "experimental": 49790 },
    "medianExecutionSecs": { "baseline": 196.0, "experimental": 198.0 },
    "slownessIncreasePct": 1.0,
    "retryRatePct": { "baseline": 0.0, "experimental": 0.0 }
  }
}

The recommendation moved to the APPLIED state, permanently setting the memory allocation for VideoEncoder.encodeV1 to 15,300 MB. We observed similarly clean results for the CPU recommendation, which reduced CPU allocation from 8 cores to 4.

Post-Apply Monitoring

Passing verification experiments does not mark the end of the recommendation lifecycle. Updated function code, modified input parameters, or underlying changes in the compute environment (Titus/EC2) can alter production behavior over time. To ensure configuration safety and foster feature adoption, we instituted continuous post-apply monitoring for applied changes.

We monitor metrics for 90 days across two rolling time windows: a 30-minute fast window and a 24-hour slow window. Each window maintains specific alert thresholds: a 3% retry rate over 30 minutes or a 2% rate over 24 hours triggers an automated notification.

At steady state, workloads exhibit baseline retry rates around 0.1%, with rates up to 2%–3% falling within acceptable operational variances. Most applied recommendations remained stable over the past three months.

VideoEncoder.encodeV1 has operated on its tuned configuration for months without breaching thresholds. However, VideoEncoder.encodeV2 presented a challenging edge case: a 50% memory reduction passed all verification checks and ran stably for two weeks before an unexpected traffic shift drove retry rates above 60%.

Because early versions lacked automated rollback capabilities, engineers manually triaged exception cases. This incident highlighted the need for automated mitigation, prompting us to construct self-healing rollback functionality.

Today, the two auto-revert windows look like this:

• Last 30 minutes: 3% retries warn us; 10% triggers an auto-revert.
• Last 24 hours: A 2% retry rate triggers both an alert and an automatic revert.

Breaching an auto-revert threshold immediately restores the function configuration to its pre-recommendation baseline.

We intentionally maintain straightforward revert logic — restoring the last known stable configuration rather than attempting dynamic resource adjustments during an incident.

The VideoEncoder.encodeV2 incident also reshaped our recommendation generation rules for memory and disk allocations. Rather than applying full target reductions immediately, SRT now applies smaller, incremental 15% step changes, advancing only after confirming stability at each step. Additionally, we introduced a global kill switch to disable automated right-sizing across all functions in case of unexpected issues or side effects.

Additionally, we enforce a mandatory cooldown period — typically one week — before initiating new experiments on a function. Pausing adjustments allows us to observe performance across diverse production traffic patterns, identifying potential risks before applying subsequent recommendations.

Where Are We Today

Oversized containers and config drift don’t have to be an accepted tax of running serverless workloads at scale. By building the Stratum Resource Tuner, we transformed manual, risk-averse resource sizing into an automated, self-healing pipeline. Through real-world workloads and hard lessons from unexpected spikes, we’ve proven that continuous right-sizing can safely deliver significant gains in infrastructure efficiency.

We initially focused on right-sizing high-throughput video encoding workloads, aiming for an initial 10% reduction in overall Stratum compute costs. For instance, VideoEncoder.encodeV1 achieved a 50% reduction in CPU usage and 40% in memory usage without compromising performance or retry rates.

However, edge cases like VideoEncoder.encodeV2 highlighted key lessons: large single-step reductions can introduce hidden risks, and manual incident response slows recovery. These insights led us to enforce stricter safety guardrails in the Stratum Resource Tuner, including capping memory and disk reductions at 15% per cycle and adding automated rollbacks when retry thresholds are breached. Guided by these safeguards, re-tuning VideoEncoder.encodeV2 yielded a stable 20% memory reduction with zero subsequent incidents.

Looking ahead, our key priorities include:

• Expand SRT beyond high-throughput encoding workloads to support a wider variety of functions and workload types.
• Further tune the resources in regions with constrained resource availability.
• Move from reactive alerts to proactive, automated upsizing for under-provisioned functions.
• Experiment with intentional upsizing for latency-sensitive workloads to evaluate performance gains and reduce function execution time.
• Upgrade our failure signals for OOM/OOD events from lagging indicators to leading indicators to improve tuning precision.

If solving problems like this sounds exciting, we are hiring in our Warsaw office. Check out open roles here.

Credits

Special thanks to Olof Johansson, Jakub Karczewski, Adam Mazur, Ethan Mathieu, Rosanna Lee, Ameya Vasani, Jean Barkhuysen, Paul Hieromnimon, and Adithya Prakash.

Netflix Stratum Media Processing: Automated Container Right-Sizing was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Danny Thomas, JVM Ecosystem Team

Recent work on the Java language to pave the on-ramp has made it easier than ever to start a Java program and evolve it using the full language and platform. At the end of that on-ramp lies Java’s mature build and dependency management ecosystem, capable of carrying software to enormous scale and complexity.

That ecosystem reached its maturity by developing strong models for projects, dependencies, and builds. When the Java Module System arrived, those models were already serving developers exceptionally well. The module descriptor consequently became just another description of the project to keep in agreement.

We’re excited to announce a preview of ja and its family of composable tools, that build on the capabilities of the Java Module System to provide a modern command line development experience for Java. We take the module descriptor and make it a complete description of a project, with dependency versions sitting naturally beside its requires directives and module metadata provided through documentation tags:

/**
 * @mainClass com.example.application.Main
 */
module com.example.application {
    requires com.example.framework; // @1.2.3
}

Combined with command line ergonomics you’re used to in other languages, creating and consuming Java modules has never been easier.

Composable Tools

Java developers have long been exceptionally well served by graphical tools. An IDE formats source, navigates between declarations and usages, presents API documentation, and maintains a compiled view of the project. That experience has been so complete that Java has had less need to expose the same capabilities through small, composable command line tools. Those gaps become quickly apparent when coding agents work with the Java language, with agents frequently struggling to locate dependencies, documentation and sources.

ja only provides command line ergonomics and tool orchestration, each feature is underpinned by a standalone tool. You don’t need to adopt ja to get the benefit of these tools, you can compose them in any way you choose:

• jig performs module version resolution, compilation and assembly, outputting standard module system arguments for use with other tools. It is also the bridge to and from Maven repositories providing a standalone module proxy and publishing commands
• jfmt formats source using the Code Conventions for the Java Programming Language, adapted for the modern Java language. Avoids the very common whitespace, indentation, import ordering and qualified class references introduced in agent written code
• jist provides source aware symbol search, providing a grep style interface for understanding class files and their associated sources. Gives coding agents access to symbols and sources without indexing, LSPs or MCPs while interoperating with other build tools via an argument file contract
• jdocserver serves locally browsable API documentation

These projects use the tool discovery and execution capabilities of the platform, and are intended to be installed in your JDK along with the standard tools. They all implement Tool or ToolProvider, allowing them to be run in process.

This is also the tool discovery and execution model for ja. There we use OptionChecker and optional custom metadata to discover which module system options are supported so it can resolve the arguments on behalf of the tool. This provides a seamless transition from your source path modules to the standard JDK tooling such as jdeps, jlink and jshell.

Maven as a foundation

In a recent survey of the 1,000 most popular artifacts on Maven Central, just 232 had explicit module definitions and another 248 declared automatic module names. The remaining 520 expressed no Java module name opinion. The module system also makes no distinction between namespace and module name, so module-first tooling requires a solution to module naming and location in existing repositories.

Fortunately, Maven Central already gives published artifacts a verified namespace. Publishers prove control of reverse domain group IDs, reflecting Sonatype’s long standing case for namespaces in public repositories.

We use these conventions to establish a canonical Maven module coordinate, paring a verifiable DNS namespace with the complete module name, for example pkg:maven/com.netflix/com.netflix.tools.ja. For existing modules, authors choose to publish a single Maven relocation pom at the canonical coordinate, to allow for discovery of the original coordinate.

When neither are available, candidates are walked from the root of the namespace using common Maven artifact conventions inferring coordinates from module names. We also bundle a short list of aliases for the most popular modules that don’t use a reverse DNS module name, but we suggest authors should always namespace their modules. The module proxy in jig presents resolved modules using the filename based conventions for module naming, making even automatic modules without stable names safe when used with these tools.

These conventions and location strategies allow the majority of existing artifacts to be discovered using only the module name and version.

Integrity by default

ALL-UNNAMED has become unfortunately common in Java access options, because of the heavy use of the class path. It hides the source of the technical debt that applications are incurring by allowing such access and becomes increasingly consequential as Java moves toward Integrity by Default. For example, Preparing to Make Final Mean Final asks applications to explicitly authorize the modules allowed to mutate final fields.

We allow runtime access requirements to bedeclared as module metadata and carried with the module descriptor throughout the module’s lifecycle. For example a library may record the access it requires:

/**
* @enableFinalFieldMutation com.example.framework
*/
module com.example.framework {
}

However, the consuming application remains in control and must explicitly authorize the framework, for it to be available at runtime:

/**
* @mainClass com.example.application.Main
* @enableFinalFieldMutation com.example.framework
*/
module com.example.application {
    requires com.example.framework; // @1.2.3
}

The command line interface for ja allows the dependency and authorization to be added together:

ja require com.example.framework@1.2.3 \
 --enable-final-field-mutation com.example.framework

Without that authorization, dependency resolution fails with an unsatisfied access requirement. Native access follows the same model through @enableNativeAccess and qualified exports and opens are also supported.

Module integrity is ensured by persistent hashes of resolved binary dependencies in a module-info.hash file, sequent resolution verifies those hashes and rejects an artifact that has changed.

We also take a step further than the recent improvements to annotation processor security by treating annotation processing as an explicit code generation step. The resulting sources are alongside regular module source, making them visible in code review and allowing a module to be assembled without executing generator code.

Make modules your default

We think every Java project should be modular, regardless of the build tool you’re using. If you’re a library author producing automatic modules, we’d encourage you to avoid split packages and produce explicit modules.

You can get started with our tools today with our installation guide.

Leave the Class Path in the Rearview Mirror was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

Any team generating text with an LLM at scale hits the same problem. You cannot evaluate everything manually. You don't know which outputs are great, which are mediocre, and which are harmful, because you simply cannot read them all. The standard solution is LLM-as-a-Judge, a second model that scores the first model’s output as a proxy for human raters. It is fast, cheap, and easy to deploy. However, it introduces a new and critical question: Who evaluates the evaluator? Furthermore, if the judge drifts, quietly degrading and starting to approve bad output, nothing downstream will flag it.

This post is about how we solve that problem, by building, aligning, and continuously monitoring LLM judges so they stay trustworthy over time. Our core principle is simple: An LLM judge must never operate alone. It has to be anchored in human judgment. We keep a human-in-the-loop (HITL) at every phase of the lifecycle. Humans define what “good” means, create the labeled benchmark datasets to train and validate the judge, monitor its behavior while it runs in the system, and review and approve any updates before they’re rolled out.

We believe:

A reliable LLM judge is not a static artifact. It has a lifecycle.

It needs to be built against trusted human labels, aligned with human preference and reasoning, run under human-set policy, and maintained as the data and usage pattern shift. Each of these stages has human owners. In the sections below, we walk through these four phases, the key design decisions in each, and the lessons we’d share with a team building their own LLM judge, using one concrete task as our running example: recommendation explanations.

A recommendation explanation is a short natural-language message that accompanies a recommended item, and it shapes how users perceive and trust a recommender system. In our work, an explanation is human-readable evidence for why a title was recommended based on what the member has previously watched, shown on the title’s detail page, which is the screen a member reaches by selecting a title from the homepage. We focus on similarity-based explanations, which connect a recommended title to one reference title the member has interacted with via shared attributes such as genre and tone. For example,

A funny, heartfelt holiday romance about love and new beginnings, much like “My Secret Santa.”

Quality matters here because a misleading or poorly grounded explanation erodes the very trust the explanation is meant to build.

Everything that follows comes out of a series of controlled, user-facing online experiments we ran on the explanation feature, including a month-long experiment on the Netflix mobile app. In these tests we used an LLM to generate explanations, hundreds of thousands a week across a constantly changing catalog. Manually reviewing them at this scale isn’t feasible, so we deployed a second LLM to evaluate them.

What counts as a good explanation is specific to our task. However, the way we build, align, and monitor the LLM judge for evaluating explanations should be reusable and generalizable across different tasks.

The Lifecycle at a Glance

Figure 1 summarizes the four phases of our LLM-judge lifecycle and the data flow connecting them.

We start with Birth: Benchmark Data Creation, where human experts craft hard or near-boundary examples and human raters label them with pass/fail decisions and written rationales. We augment this with LLM-generated gray-area examples to build a richer benchmark dataset.

Next, in the second phase of Training: LLM Judge Development, we tune the LLM judge to align with human labels and rationales from the Birth phase, using self-reflection to identify mismatches between the judge and humans, and updating the rubrics based on those mismatches and anchor examples. We iterate in this phase until our metrics show that the judge is sufficiently aligned with human raters.

In the third phase, Deployment, we integrate the judge into the explanation generation system as both a gate and a critic. The LLM generator writes explanations, and the LLM judge scores them and either passes them through or blocks them and supplies a rationale that the generator can use for self-refinement and retries. Passed explanations are served to members, and low-quality explanations are gated.

Finally, in the last phase, HITL Alignment & Drift Monitoring, we keep the judge aligned with humans over time. “Human-in-the-loop” here means a person is directly involved in the decision, not just reviewing outcomes after the fact. We benchmark the judge against trained raters, and a human approves every change before it is rolled out to the system. During each test we sampled a set of explanations each week (both passed and gated, with a bias toward newer items), had human raters review them with labels and rationales, and compared the judge against these human ratings. If drift exceeded a threshold, we triggered retraining and benchmark data augmentation. This closes the loop by feeding new human-labeled data and updated signals back into the benchmark and development phases.

System components

Four automated components appear across all phases, alongside the human roles that define, validate, and supervise them.

• Generator: produces the explanation text, optionally using prior judge rationales and traces as part of its prompt for revision.
• Judge: evaluates an explanation and returns a pass/fail label together with a written rationale. In the deployed pipeline, it acts as both a guardrail (gating bad outputs) and a critic (providing feedback for self-refinement).
• Reflector: revises the Judge’s rubric when the Judge makes mistakes. It updates the rubric based on mismatched examples, human rationales, and anchor examples. It tunes the rubric and does not directly evaluate explanations.
• Meta-Judge: checks whether the Judge failed an explanation for the right reason by comparing the Judge’s rationale against a human rater’s rationale.
• Human experts and raters: the experts who define the quality criteria; the raters whose labels and written rationales the Judge is tuned and measured against; the reviewers who monitor the Judge while it is running and explicitly approve any changes before they were rolled out.

Phase I, Birth: Establishing Ground Truth

Everything downstream depends on knowing what “good” means, so this is the most human-intensive phase. Our internal writing experts define the must-have criteria an explanation must satisfy, author detailed labeling guidelines, and hand-craft tricky adversarial examples. Those same guidelines do double duty. They’re both the rulebook for human raters and the seed rubric for the LLM judges we tune later.

Why we built our own benchmark

Off-the-shelf LLM-judge benchmarks focus on general-purpose text evaluation and miss what makes our task specific. Explanations are short, item-specific, and context-dependent (each one is tied to a target title and one or two reference titles the member has watched). We also care about alignment to our own editorial bar, not just generic language quality. As a result, we built our own custom benchmark from three complementary sources:

• Expert-crafted adversarial cases: hand-authored pass/fail examples with known failure modes, the kind of edge cases nothing else reliably produces.
• Boundary examples: LLM-synthesized explanations that are deliberately pushed near the pass/fail boundary, then rated by humans. These capture the ambiguous cases that sampling from live traffic rarely surfaces on its own.
• Real sampled explanations: explanations sampled from the output of our LLM generator during test runs.

Two details that mattered

• The benchmark is not frozen. A monitoring pipeline (Phase IV) appends a fresh batch of human-rated explanations on a weekly cadence, so over the course of a test the benchmark tracked the current catalog and new usage patterns instead of going stale.
• Every “fail” label carries a free-text rationale explaining why the explanation failed. This looks like a small bookkeeping detail, but it is actually critical for the next phase, since those rationales are what we use to tune and audit the judge’s reasoning.

Phase II, Training: Aligning the Judge to Humans

The failure mode: right answer, wrong reason

There is a failure mode that does not show up in label-agreement metrics at all. A judge can agree with humans on the label, passing when they would pass and failing when they would fail, while reaching the right verdict for the wrong reason. On a dashboard it looks perfectly aligned. But as soon as you ask it to explain a rejection, which our pipeline does to steer self-revision, it can return misleading guidance.

We therefore care about not just label agreement but also reasoning alignment. In our setup, the Judge is always asked to output:

• a label (e.g., PASS/FAIL), and
• a short natural-language rationale for that label.

Here is an illustrative example. The member watched a musical, a family fantasy with a young heroine and a well-known ensemble cast. We recommend an adventure movie that is also a family fantasy with a young heroine, which is why the recommender paired them. But it shares neither the songs nor the ensemble cast.

The label (FAIL) is correct, and the stated reason is reasonable at first glance. But it turns out to be incomplete in a way that simple label agreement cannot detect, as we show below.

Reasoning-Aligned Rubric Tuning

We tune each criterion’s LLM Judge with a procedure we call Reasoning-Aligned Rubric Tuning (RART). As introduced in the system components above, the Judge, Meta-Judge, and Reflector are distinct components. The Judge scores an explanation against a rubric and gives a rationale, the Meta-Judge compares that rationale against a human-written one, and the Reflector edits the rubric text itself. RART is how the three are combined into a training loop.

Every Judge is a fixed prompt template parameterized by a rubric, so tuning it does not mean retraining the underlying model. Instead we iteratively refine the rubric text until the Judge’s behavior matches human labels and human rationales. At iteration i, the Judge is conditioned on the current rubric text and maps an explanation to a pair, a label and a rationale.

At a high level, each RART iteration proceeds as follows.

1. Score the training data with the current rubric.
2. Evaluate weighted alignment metrics on a held-out validation set. If all per-criterion metrics exceed their thresholds, stop.
3. Otherwise, construct an error-driven focus set of misaligned examples and hand it to the Reflector, which proposes a revised rubric.

The key difference from a standard loop is how we define a “mistake.” Most self-improvement loops look only at label disagreements. RART adds a second class of error. The Meta-Judge examines the cases where the Judge and a human both said fail and compares the Judge’s stated reason against the human’s rationale. When the two diverge, that is a right answer, wrong reason error, and it is also fed to the Reflector. Concretely, the focus set handed to the Reflector is the union of two error classes:

• Label mismatches: examples where the Judge’s label differs from the human’s.
• Right label, wrong reason: examples where the Judge and the human both said FAIL, but the Meta-Judge finds their two rationales don’t match.

So the Reflector revises the rubric from both ordinary label errors and right label, wrong reason errors. That second term is the core of RART. The Judge is pushed not only to match human decisions but to justify them in a way the Meta-Judge recognizes as aligned with human reasoning.

Returning to the earlier example, the LLM Judge said FAIL and a human said FAIL, so a label-only check would register agreement. Placing the two reasons side by side shows what that check misses.

RART routes it into the focus set, so the next rubric revision has to account for it.

Since the Meta-Judge is an LLM too, we checked it against humans first. Trained human raters labeled a sample of agreed-fail rationale pairs, and the Meta-Judge matched their verdicts on nearly all of them.

Alignment metrics

We run one Judge per criterion, each with its own rubric, tuned and scored on its own. So everything below is measured per criterion on held-out data. For each one we track three metrics.

• Specificity (fail-recall): how often the Judge correctly rejects bad explanations. This is the one we care about most, because a bad explanation that slips through the gate is a trust hazard.
• Recall (pass-recall): how often it correctly passes good explanations, which keeps coverage high and revision cost low.
• Reasoning agreement: on the explanations both reject, how often they reject for the same reason.

Each criterion’s Judge gets its own three, and one can be healthy while another slips. That is also why the gate is strict. An explanation has to pass every must-have criterion to be eligible to serve.

Ablation: does the reasoning signal help?

To isolate the impact of the reasoning signal, we compare RART to an otherwise identical loop that uses only label disagreements. We call this baseline vanilla: same Reflector, same Judge, same data, but no Meta-Judge and no use of human rationales.

Figure 2 shows the result. On two of the three must-have criteria, vanilla and RART land in the same place, because the starting rubric was already near ceiling and the metrics had very little room to gain. On the third criterion, the vanilla (label-only) method degrades. The Reflector in the vanilla loop rewrites the rubric into degenerate forms, and specificity and reasoning agreement fall with every iteration, ending up worse than the rubric it started from. RART stays stable on that same criterion and lifts both, with a small recall gain as well. The reason is that a vanilla (label-only) optimizer can reach the right label through unsound reasoning, and unsound reasoning doesn’t generalize to new data. Anchoring the signal on the rationale prevents that. Reasoning alignment helps where there is headroom and doesn’t hurt where there isn’t.

Phase III, Deployment: Putting the Judge to Work

Once deployed, the judge sat inside the generation pipeline, gating and critiquing explanations as they were produced during our tests.

Two roles for one judge

Once we align the LLM Judge with humans, we reuse it in two distinct roles for every generated explanation:

• Role 1, the gate. The Judge scores each explanation against the must-have criteria and rejects anything that fails.
• Role 2, the critic. When an explanation fails, we don’t just throw it away. The Judge’s free-text reason gets appended to the Generator’s prompt, and the Generator tries again.

The second role turns the pipeline into a generate → judge → revise loop, with the Judge’s critique steering each revision.

Bounded retries, and why we drop rather than serve

We cap the retries. Each revision costs another round of generation and judging, and the gains flatten quickly. Figure 3 shows what the loop actually buys, on 1,000 generated explanations run through as many as 12 revisions with four generators. Every curve climbs, which is the Judge’s critique doing its job, but the steep part is early, and the gap between generators is obvious. Revision amplifies a capable generator rather than rescuing a weak one. In our test runs we used a budget of 3 retries with our strongest generator, and an explanation still failing after that was dropped rather than served.

That “drop rather than serve” rule is deliberate. The two outcomes aren’t symmetric. A bad explanation is a trust hazard, while a missing explanation just forgoes an opportunity. When in doubt we’d rather show nothing, and most explanations cleared the bar within the retry budget.

Do judge-approved explanations help members?

Passing the Judge proves an explanation is good by our standards, but it doesn’t prove that judge-approved explanations are useful to members. To test that connection, we ran a one-time controlled experiment on the Netflix mobile app, explanations vs. no explanations on the title detail page, over roughly a month, with the pipeline generating and judging hundreds of thousands of explanations a week during the period.

Members who saw explanations shifted their viewing toward titles they hadn’t watched before, and more of their browse sessions ended in a meaningful play. Both effects were statistically significant, and we saw no member-initiated takedowns or escalations about explanation quality during the test. The result worth carrying over isn’t the size of that effect. It’s that offline agreement with our raters was necessary for these explanations to work but never sufficient to show they help members. Only the online A/B test could establish that judge-aligned explanations are not just quality-compliant but genuinely useful.

Phase IV, Monitoring: Keeping the Judge Aligned

A judge that’s well aligned at launch won’t stay that way. The catalog shifts, and so does what counts as high quality. The lifecycle has to close the loop.

The weekly human review

For each week a test was running, we pulled roughly 300 generated explanations for human review, stratified across the Judge’s decisions (served, revised, dropped) and weighted toward newer catalog items, where drift is most likely. The stratification stayed fixed week to week, so successive weeks were comparable. A panel of at least three raters reviewed each explanation, and the majority label was our ground truth. Those labels did two jobs. They told us whether judge-human agreement was slipping, and they grew the benchmark.

Setting the bar with human disagreement

The panel also tells us how much the raters disagreed with each other, which turns out to be the right yardstick for the Judge. Rather than hold it to a fixed threshold, we score the Judge and each rater against the same majority label and ask that the Judge fall within the raters’ own spread.

Take one criterion and one of its metrics, M. That week’s sample gives one value M(J) for the Judge and a set of per-rater values M(H) for the three or more raters on the panel. We require

so the Judge must score no worse than two standard deviations below the average rater. A harder week widens human disagreement and widens the band with it, so the Judge isn’t penalized for difficulty the humans also felt.

Falling below the band on any metric-criterion pair is a drift event, which triggers re-tuning (back to Phase II) on the augmented benchmark. A re-tuned rubric never goes live automatically. Someone on the team reads the proposed changes and signs off before it reaches the running pipeline, and we keep the previous rubric for instant rollback.

What human review catches that metrics can’t

Beyond asking whether the Judge still agrees with human raters, the weekly review answers a second question. Are the current guidelines still asking for the right things?

In the weekly review process, reviewers surfaced issues that did not show up in Judge–rater agreement. An explanation can use accurate attributes and still overstate how similar two titles really are. A single vague adjective (“exciting”, “fun”, “emotional”) can technically satisfy the rubric while saying almost nothing. Sometimes the reference itself is the problem, something the member wouldn’t think of as a comparable title at all, so the comparison lands oddly no matter how the attributes line up. Some genres show the same problem at scale. Stand-up specials can share similar metadata tags but differ completely in tone and cultural context, so a comparison may look justified on paper, not be bad enough for a rater to fail, and still land poorly with members. And some explanations meet every must-have criterion while still reading like generic “word soup” that does not help the user much.

When recurring patterns kept appearing, we updated the Phase I guidelines, which seed both the raters’ instructions and the judges’ rubrics, so one revision updates both sides. Other times we added product rules outside the Judge, for example by blocking certain title types from being used as references.

Drift detection catches a judge that has stopped agreeing with people. Only the continued human review tells us when our definition of a “good explanation” is too narrow or incomplete. We need both.

• Drift detection keeps the Judge aligned with the current rubric.
• Continuous human review keeps the rubric aligned with what actually works for members.

What We Learned

Five lessons we think transfer to any team shipping an LLM judge:

1. Invest in the benchmark before the judge. A modest, rationale-annotated benchmark beats a large label-only one. Those written rationales are exactly what make reasoning-aligned tuning possible. You can’t correct a judge’s reasoning if you never wrote down the human’s.
2. Tune on reasoning, not just labels. A judge can reach the right verdict for the wrong reason, which looks fine on a dashboard but gives misleading feedback and doesn’t hold up on new data. In our ablation, the label-only loop degraded a rubric that the reasoning-aware loop kept stable.
3. One aligned judge can serve many roles. Reusing the same judge as both gate and critic amortizes the alignment cost and keeps production behavior consistent across roles. Alignment is the expensive part, so get one judge right and use it more than once.
4. Plan for monitoring from day one. A recommender platform’s items and members never stop changing, so an LLM judge, like any ML model, has to be watched and re-tuned if needed, not shipped and forgotten.
5. Keep people in the loop by design, not by exception. Automating evaluation is not the same as delegating responsibility for it. In our lifecycle, humans set the standard (Phase I), supply the rationales the judge is aligned to (Phase II), decide the conservative default of dropping rather than serving a doubtful output (Phase III), and review a standing sample of live decisions and approve every rubric change before it ships (Phase IV). These checkpoints are much harder to retrofit once a judge is already making decisions on its own. The standing review also tells us what our rubric is missing, not just when our judge has slipped.

Taken together, the judge extends the reach of our reviewers, and those reviewers are the basis for trusting the judge. That relationship is what makes it responsible to run this system at scale. An automated evaluator that no one audits provides no assurance about the decisions it is making.

Acknowledgements

This post summarizes our research paper, “The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations,” which has the full methodology, metrics, and ablations.

Our thanks to the internal human writers and review experts who defined what a good explanation is and keep those guidelines current; to our Human Data Operations team and the raters they train and calibrate, whose labels and written rationales are what the judges are aligned to; and to the engineers across the team who built and operate the pipeline; and to our UI engineering partners, who built the member-facing surfaces that bring these explanations to the homepage.

The Lifecycle of LLM-as-a-Judge: Building, Aligning, and Monitoring at scale was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.