Introduction

In our multi-part blog series, we have discussed why we chose Ray as a last mile data processing framework, how we integrated Ray into our existing ML Training Infrastructure, and how we enabled efficient Batch Inference at scale. In this blog post, we will discuss how we were able to apply Ray to another critical part of our ML Infrastructure stack: Model Evaluation.

In the Ray Batch Inference Blog Post, we ended on a section on accumulators (counters that would allow us to calculate metrics like ROC-AUC and cross entropy). This post is where those accumulators pay off: a model evaluation platform that combines the strengths of Ray and Apache Spark to compute metrics efficiently at scale.

Model evaluation may appear to be a relatively straightforward step after inference or training. But as ML workflows scale, evaluation quickly becomes an infrastructure challenge. Systems that run many experiments in parallel, such as feature importance, must efficiently compute and aggregate metrics over large volumes of inference data, as shown in Diagram 1.

Later in this post, we’ll show how we amortize the metrics compute step, overlapping it with inference for minimal added runtime cost, cutting end-to-end evaluation time by up to 32% while scaling linearly as we run more experiments.

Terminology

• Feature Importance (FI): A technique for quantifying how much each input feature contributes to a model’s predictions. In this post, we will reference a perturbation-based method where we perturb one or more columns, run inference, and compare the results against an unperturbed dataset.
• Experiment / baseline: One perturbed run vs. the unperturbed run.
• Model head: One output of a multi-task model.
• Batch: an arbitrary, compute-sized group of rows. This is the unit of data that one Ray task processes.
• Bucket: a batch that is group-preserving. It contains one or more complete segments, and no segment may span buckets.
• Segment: The semantic unit over which a group-wise metric is defined, such as a request.
• Serde [serialization/ deserialization]: The process and cost of writing data out and reading it back.

Background

Traditionally, evaluation and inference are two separate stacks stitched together. Diagram 2 below contrasts that unfused pipeline against the fused pipeline we’ll build up over the rest of this post.

Concretely, this evaluation step is typically implemented as a Apache Spark job that computes metrics across one or more inference experiments. As shown in the diagram above, we previously used Ray inference to write predictions to S3, then have a follow-up compute job to reread all this data and compute the metrics.

For a FI run, this approach introduces two sources of overhead:

• Cluster startup: Acquiring and warming up a cluster for the metric job.
• Data movement: Writing predictions to S3 and reading them back to compute metrics.

Because inference is done in batches, and metrics typically depend on all of the inference results, the unfused approach can be used to calculate any metric. To save the overheads, we need to fuse the stages, but this also implies that we need to be able to calculate metrics on batches of rows. Therefore, we need a metric platform that treats decomposability as a first principle.

Metrics fall into three groups:

Metric Contract

We define a common metric contract that captures each metric’s compute layout and provides a consistent interface across execution frameworks. We express this interface concretely as an Abstract Base Class.

from abc import ABC, abstractmethod

class ReducibleMetric(ABC):
    @abstractmethod
    def dataset_preprocessor(self, ds):
        """Prepare the dataset to satisfy the metric's data layout requirements."""
        pass

    @abstractmethod
    def compute_partials(self, batch):
        """Compute a partial result from one batch"""
        pass

    @abstractmethod
    def reduce_partials(self, partials):
        """Combine partial results into the final result."""
        pass

Decomposable and holistic metrics can calculate partials from arbitrary batches because they do not require related rows to be colocated.

Group-wise metrics introduce an additional constraint. Before handing data to Ray, we organize complete segments into buckets. A bucket is therefore a group-preserving batch: it contains one or more complete segments, and no segment crosses a bucket boundary. This lets us independently tune bucket size for efficient execution without changing the semantic unit of the metric.

Common Metrics found in Recommender Systems

We will walk through three metrics, MAE, ROC-AUC, and Precision@K; one from each group above. We will use the following dataset to compute each metric.

For the examples below:

• model_output contains the predictions, and label_continuous contains the target values.
• label_binarized is derived as 1 if label_continuous ≥ 0.5 and 0 otherwise.
• Batch A contains ReqA, and Batch B contains ReqB. This split lets us demonstrate batch-level computation and aggregation.

MAE (Decomposable)
Mean Absolute Error measures the average absolute difference between predictions and their actual values.

MAE_b = (1/n_b) · Σ_{i ∈ batch b} |pred_i − label_continuous_i|

Unlike MSE/RMSE, MAE does not penalize large errors as heavily.

Per-row absolute errors:

Global: Σ = 1.45, N = 8 → MAE = 0.18125.

Batch decomposition:
Batch A partial (Σ|err| = 0.65, n = 4)
Batch B partial (Σ|err| = 0.80, n = 4)
Merge (0.65 + 0.80) / (4 + 4) = 0.18125

class MAE(ReducibleMetric):
    def dataset_preprocessor(self, ds): 
  # No dataset preparation needed. Rows are independent.
 return ds

    def compute_partials(self, batch):
  # Emit summed absolute error and row counts from each batch.
        error = np.abs(batch["model_output"] - batch["label_continuous"])
        return error.sum(), len(error)

    def reduce_partials(self, partials):
  # Sum partials and divide once globally. 
        error, count = np.sum(partials, axis=0)
        return error / count

ROC-AUC (Holistic)
ROC-AUC is a binary classification metric which measures the ability of a model to distinguish between classes across different decision thresholds.

Method 1 — Global Sort (what sklearn does)

Global Sort

We sort the rows by model_output in descending order, keeping each prediction paired with its corresponding binary label.

Cumulative Sum

After sorting, we calculate cumulative true-positive (TP) and false-positive (FP) counts and prepend 0 to both vectors.

Calculate the TPR and FPR

We calculate the TPR at each threshold by dividing the TP vector by the number of positive labels.

We calculate the FPR at each threshold by dividing the FP vector by the number of negative labels.

Calculate the Trapezoidal area under TPR/FPR

We calculate the AUC of TPR/FPR to get the final ROC_AUC value.

Sorting our eight rows by score:

With 4 positives and 4 negatives, the area works out to AUC = 0.9375 (or 15/16 positive/negative pairs are ranked correctly).

We observe that ROC_AUC using this method is calculated over the entire dataset due to a global sort. We could break the dataset into batches, calculate ROC-AUC for each batch, and average the results. However, this does not produce the same result as a global ranking.

Consider the following calculation:

• Batch A (ReqA) = 1.00
• Batch B (ReqB) = 0.75
• Average = 0.875, not 0.9375

Averaging a ranking number across batches is wrong, because AUC depends on how rows rank against all other rows, not just those in their batch.

We could force a real global sort in Apache Spark, but doing that across all metrics, model heads, and experiments is exactly the cross-stack cost we’re trying to avoid.

Method 2 — Confusion Matrix

Consider the standard method to approximate ROC_AUC.

Define thresholds

Pick a fixed set of thresholds in [0, 1].

Calculate Confusion Matrix

Calculate the confusion matrix (TP, FP, TN, FN) for each threshold

Calculate the final metric

Calculate the TPR/FPR and finally derive the AUC.

This method can be calculated batch-wise because the global confusion matrix can be calculated from the batch confusion matrices.

We can create the following confusion matrices per batch and then sum:

From this approximate example, when we derive the area under the curve, we get 0.875. If we use more thresholds, for example [0.00, 1.00] with step size of 0.01, we would get 0.9375. Note: This does not mean that threshold-based aggregation generally produces the exact global AUC. In this toy example, a step size of 0.01 recovers the exact value because the prediction scores align with the threshold grid.

class ROCAUC(ReducibleMetric):
    def dataset_preprocessor(self, ds): 
 # No dataset preparation needed. Rows are independent.
return ds

    def compute_partials(self, batch):
  # Emit confusion matrix across thresholds. 
        pred = batch["model_output"]
 truth = batch["label_binarized"].astype(bool)

        positive = pred[None, :] >= self.thresholds[:, None]
        return np.stack([(positive & truth).sum(1), (positive & ~truth).sum(1),
                       (~positive & ~truth).sum(1), (~positive & truth).sum(1)],1)

    def reduce_partials(self, partials):
  # Sum confusion matrices and add ROC endpoints.
 # Integrate in ascending FPR order. 
        tp, fp, tn, fn = np.sum(partials, 0).T
 tpr, fpr = tp / (tp + fn), fp / (fp + tn)
        return np.trapz(np.r_[0, tpr[::-1], 1], np.r_[0, fpr[::-1], 1])

Precision@K (Group-wise)
Precision@K is a common metric used for ranking systems. It measures the fraction of the top K ranked items that are relevant.

This metric is calculated within an individual segment of data, such as a session or request. A segment may be identified by a single key or a composite key.

The procedure for calculating Precision@K is as follows.

Group/Rank Dataset
Group the dataset by segment and sort by predicted score.

Window and Count
Select the top K items and count the relevant items within that window.

Calculate Metric
Divide the number of relevant items by K to calculate Precision@K for each segment. Average the per-segment scores to obtain the final result.

On our data with K = 2:

Average = (1.00 + 0.50) / 2 = 0.75.

We observe this metric has a similar decomposition pattern. We can break it down into subproblems where we look at each segment, calculate the metric, then average final scores of all subproblems.

We have two problems when trying to scale this approach:

1. The subproblem (batch_size) is tied to the size of each segment. This is inefficient as each executor may receive a small and variably sized amount of data.
2. If we combine segments together, we must ensure that no segments span a bucket boundary.

Our goal is to keep every row for a given segment in the same bucket while ensuring that each bucket contains enough segments to saturate the compute resources.

More succinctly, we want all rows within each segment to be colocated and each bucket to be sufficiently large. Since we are operating at the global level, we can define this as an Apache Spark procedure. It keeps all rows for each segment together, then packs complete segments into buckets large enough to use each Ray worker efficiently without splitting a segment across buckets.

class PrecisionAtK(ReducibleMetric):
    def dataset_preprocessor(self, ds):
  # Colocate each segment and pack into buckets.
        return bucket_complete_segments(
            ds,
            by="request_id",
        )

    def compute_partials(self, bucket):
  # Rank within each segment and calculate top-K score.
        scores = []
        for segment in group_by(bucket, "request_id"):
            ranking = np.argsort(
                -segment["model_output"]
            )
      # Assume each segment contains at least K items.
            top_k = ranking[:self.k]
            relevant = segment["label_binarized"][top_k].sum()
            scores.append(relevant / self.k)

        return sum(scores), len(scores)

    def reduce_partials(self, partials):
  # Sum segment scores and divide by segment count.
        score, count = np.sum(
            partials,
            axis=0,
        )
        return score / count

Fusing Metrics with Inference

In the upper figure in diagram 3, we show a typical Sequential Inference-Metrics multi-experiment pipeline. We run model inference over all the experiments, serialize, and persist those results along with the carryover labels. Note this inference dataset scales with the input data and the number of experiments. Next, we deserialize this data and compute metrics. GPU inference is typically more resource intensive than metrics computation. This design causes the CPU workers to sit idle during inference time.

The lower figure in diagram 3 illustrates a fused approach where we pipeline the metric computation once inference completes for each bucket. The shared object store minimizes serde overhead. This enables us to effectively overlap metric computation with inference, reducing total runtime.

Results

We benchmarked two setups against a Pin Ranking Model, sweeping across three dataset sizes (Small ≈ 30K, Medium ≈ 300K, Large ≈ 3M rows) and several experiment counts:

• Unfused: Ray inference, then a separate Ray metric job.
• Fused: Ray inference with metrics in the same pass.

When we gather metrics during inference, we do not see any meaningful runtime increases. The time measured here includes startup time (acquiring the cluster) and runtime (actual computation).

Furthermore, we can see that runtime grows linearly with experiment count while startup stays flat.

As the number of experiments increases from 1 to 300 on the Large dataset, fused runtime grows approximately linearly (~43 min fixed + ~41s per experiment), while startup time remains roughly constant. The separate metric stage (~11s per experiment) is work that fusion absorbs through overlap with inference, keeping the marginal cost per experiment small and predictable.

What’s next

Other upcoming topics include:

• Ray Tune at Pinterest
• Improving the data loader performance in Ray
• Fault-tolerant Ray Batch Inference and Training
• Feature Importance on Ray

Acknowledgments

Related Pins: Shivin Thukral, Zilin Ren
AI Platform: Rubin Fergerson IV, Qingxian Lai
Data Platform Sciences: Darren Reger, Adam Obeng
Agent Platform: Tan Wang

Scaling Model Evaluation at Pinterest with Ray (Part 4) was originally published in Pinterest Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

Modern recommendation systems, including those used for ads ranking and content discovery, have grown substantially in scale and complexity. This growth stems from both increasingly large model architectures. The resulting memory footprint often exceeds the capacity of a single GPU, making it necessary to shard production models across multiple GPUs using distributed training techniques such as Distributed Model Parallelism (DMP).

This evolution brings challenges to the everyday engineer work on testing and debugging the model code. The bottleneck includes:

• Local Machine Hardware Limits: Because production models require multi-GPU setups, initializing a full model locally on a single-gpu workstation is virtually impossible.
• Input Data Dependencies: When an engineer modifies code inside a specific downstream module, getting realistic input data is a major hurdle. Traditionally, obtaining that input requires initializing the entire multi-GPU pipeline from scratch and executing every preceding module in the network — wasting immense amounts of time and computation resources.
• Slow Iteration Feedback Loops: Engineers face prolonged iteration loops when attempting to verify sub-module changes, as every code edit requires re-executing or re-initializing heavy distributed trainer setups.

To eliminate these bottlenecks, we build ModuleLens, which pivots from a legacy “Model-First” paradigm to an isolated “Module-First” engineering framework. The following table contrasts these two operational paradigms across key engineering metrics:

Under the module-first paradigm, engineers can isolate any given module, avoid multi-GPU model initialization entirely, and reconstruct and profile it locally within seconds. In addition to accelerating developer workflows, this modular foundation offers a fast, structured execution environment that empowers autonomous AI agents to analyze, refactor, and optimize PyTorch code independently.

Architecture

The architecture of ModuleLens is organized into three primary components: ModuleTracer for capturing runtime data, Module Repo for central storage, and a Consumption Layer serving human interfaces and AI-based tooling. The following diagram depicts this structure.

ModuleTracer

ModuleTracer is designed to capture and record all essential data — including input and output tensors, model parameter weights, and initialization arguments. This comprehensive tracking allows any target PyTorch module to be reconstructed seamlessly within the consumption layer, enabling offline execution with captured inputs and validation against expected outputs.

To carry out trace capture, ModuleTracer relies on several key mechanisms:

• Non-Intrusive PyTorch Forward Hooks: Attaches listeners directly to PyTorch modules to seamlessly log parameter weights (state_dict), input and output tensors, and execution metadata during runtime.
• In-Memory Module Init Tracking: Ensures target modules can be reconstructed by dynamically tracking initialization arguments (__init__ args and kwargs). It achieves this in memory by inspecting imported classes and patching Python’s __import__ function.
• Resource-Efficient Tracing Loops: Reduces GPU compute overhead by registering early_stop_func and post_train_step_func callbacks. These wait for model warmup, capture data across a single training step, and terminate early.
• Optimized Handling of Large Tensors: To efficiently profile large tensors, ModuleTracer automatically enables shape-only tracing (log_shapes), logging solely tensor dimensions so that synthetic inputs can be reconstructed during restoration. Additionally, if an input tensor is shared across multiple modules, ModuleTracer retains only a single copy and uses reference links for all other instances.

ModuleRepo

Extracted module state is persisted within a queryable backend built on top of the Weights & Biases (WandB) Registry. All captured modules are stored into one registry and organized around Collection and Versioned Artifacts:

Collection Indexing: Target collections are indexed using an MD5 hash of the model and module names: hash(model_name, module_name). Additionally, top-level trace executions log overall run history to special collections structured as registered_model_xxx, indexed directly by model name. The registry supports both attribute-based and tag-based search filtering.

Versioned Artifact Payload: Every captured Module produces a distinct version entry containing two standardized artifacts:

• JSON Metadata File: Contains runtime execution metadata including the model and module metainfo, and the autocast info.
• Binary State Payload: Serialized via torch.save, containing the precise input tensors, baseline output tensors, and module parameter state_dict.

Consume Layer

ModuleLens exposes its capabilities through two tailored surfaces to consume the traced data:

Human Engineers: Interactive Jupyter Extension and IPython CLI for hands-on debugging, exploration, and manual profiling.

AI Agents: Agent Skills that allow LLM agents (e.g., Claude Code, Cursor) to autonomously discover, rebuild, benchmark, and profile modules without human intervention.

The consume layer provides the following capabilities:

The screenshot below illustrates the command-line interface available to users.

Use Cases

Ever since its deployment across our Ads and Core models, ModuleLens has also been adopted for various internal tools built by our engineering teams.

Training Analyzer

Evaluates module-level metrics across different architectures. Through hierarchical visual representations of latency, FLOPs, and memory usage across the module tree, engineers can rapidly identify training bottlenecks.

KernelAgent

An LLM-driven agent designed to generate high-performance GPU kernels. It leverages captured ModuleLens data to benchmark, profile, test, and validate the accuracy and speed of generated kernels.

Scaling Law ROI Tools

Evaluates the return on investment (ROI) for model scaling. By utilizing ModuleLens to track training and serving FLOPS, activation and parameter memory usage, and execution throughput, it helps assess infrastructure costs against model scaling gains.

Looking Ahead

Our immediate priority is stabilizing the core system and fostering adoption across machine learning teams. Concurrently, we are enriching our Agent Skills repository to furnish AI coding agents with advanced optimization capabilities, including automated Triton/CUDA kernel synthesis as well as autonomous training and serving optimization workflows. By building reliable, execution-friendly interfaces for both developers and autonomous agents, ModuleLens optimizes human engineering velocity while laying the groundwork for scalable, fully automated performance optimization.

Beyond optimization, ModuleLens can serve as an execution substrate for model autoresearch and RSI (recursive self-improvement). By giving agents reproducible access to isolated modules, captured inputs and outputs, and trustworthy performance signals, it enables them to formulate experiments, modify model components, benchmark results, and retain successful changes in a tight feedback loop. Over time, this foundation can support systems that continuously improve both model quality and the tools and strategies used to optimize the model.

Acknowledgement

This project benefited from the valuable contributions of many individuals. We express our gratitude to Archer Liu, leader of Product ML Infra (PMLI), for his leadership and support, Chen Yang (ATG) for insightful design discussions and feedback; Abhinav Naikawadi (HomeFeed) and Paulo Soares (Ads ranking) for leading testing, adoption, and advocacy efforts; and Jiasong Sun, Xinyi Zhang, Diane Li and Hongze Zhao (Product ML Infra) for their contributions to code development and review.

Introducing ModuleLens: Zooming into PyTorch Modules for Human and Agentic Engineering was originally published in Pinterest Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Pinterest is built to get inspired and then turn the inspiration into realization — a dinner, a renovation, a wedding, a new skill. That only works if we understand more than the Pin a user just clicked. The useful unit of personalization isn’t the thing someone just clicked, it’s what they’re trying to accomplish. Such goals unfold over weeks and months, across dozens of sessions, through hundreds of saves.

We call those goals journeys: the intersection of a user’s interests, intent, and context at a point in time. A journey is a sequence of interactions, often spanning sessions, centered on one interest and a clear intent. For example, “Summer dresses” are an interest. “Figuring out what’s in style for a summer wedding I’m attending in August” is a journey. Someone might be interested in five at once: a kitchen remodel, a vegetable garden, learning sourdough and the mix changes as life does.

If we can name those journeys, recommendations and notifications can help someone finish a project instead of bouncing between unrelated interests. This post is about how we infer them in production: a fine-tuned 4-billion-parameter large language model (LLM) that reads a chronological activity log and emits a ranked list of journey names, in the user’s locale.

We didn’t start there. The initial user journey system clustered keywords and attached attributes with separate machine learning models. This post is the follow-up which covers how we formulate journey inference as generation, how we distill frontier-model quality into a 4B student we can actually serve, how the serving stack holds up at notification scale, and what we’re building next. The recipe is domain-agnostic: any platform with sequential user activity and structured intent targets (e-commerce, streaming, search) can adopt the same activity-to-intent formulation, collapsing classical multi-stage recommendation pipelines into a single generative model, making generative journey inference a general recipe for personalization beyond Pinterest.

Why interests aren’t enough

Most recommenders optimize the next item. However, journeys on Pinterest are longer-horizon. Planning a kitchen is not the same as a one-off click on a cabinet Pin, and it isn’t well described by a durable interest like “Home Decor”. The cabinet Pin, the backsplash search, and the garden-bed save might be one project, two projects, or noise. Getting that distinction right changes what we retrieve, what we write in a notification, and when we should stay quiet.

So the output we want is not a taxonomy label. It’s a short, specific, ranked list of a user’s active journeys, with specific names they’d recognize.

The legacy clustering system

We had almost no labeled journeys when we started, so we stayed lean. We reused pretrained text embeddings, started from a few hundred human examples, and built a multi-stage pipeline:

1. Extract keywords from search queries, annotations and interests on engaged Pins, and Pins on owned boards.
2. Embed and cluster those keywords hierarchically. Each cluster is a journey candidate.
3. Name the cluster by ranking its keywords and taking the top one.
4. Rank and diversify the clusters with a pointwise model, then a similarity penalty so the top of the list isn’t five near-duplicates.
5. Classify stage with heuristics: evergreen vs. situational (roughly, does engagement persist past a few weeks?), and for situational journeys, ongoing vs. ended (time since last engagement).

That pipeline powered the first journey-aware notifications. Against interest-based notifications, it lifted email click rate by 88%, push open rate by 32%, and positive survey feedback by 23%.

It also compounded its own errors. Every stage was locally reasonable; together they fragmented coherent goals.

• Fragmentation: Cabinets, countertops, and backsplash are one kitchen, not three journeys fighting for the same notification slot.
• Generic buckets: When a cluster was messy, the naming model fell back to the most frequent keywords — “Art”, “Recetas”, “Shopping”. Taxonomically true, useless for personalization.
• Incidental noise: A handful of meme Pins or celebrity clicks could occupy a top slot. Clustering doesn’t know which activity is a goal and which is a passing click.

Journey inference as generation

To solve these problems, we stopped treating journey inference as a chain of transformations and started treating it as one act of generation: read the activity, write the intent. The model gets a single input prompt containing a semi-static profile plus a chronological activity log, and returns strict JSON — a ranked list of journey names:

User Profile:
gender: female; language: en-US; country: US; age: 30-35
User Owned Boards: Kitchen Reno 2026; Weeknight Dinners; Garden Plans
User Activities:
2026-06-14 search shaker cabinets.
2026-06-14 save subway tile backsplash, kitchen backsplash; quartz countertop, white kitchen.
2026-06-12 click raised garden bed, vegetable garden; tomato trellis.
2026-06-09 search sourdough starter discard.
2026-05-30 save cast iron skillet cornbread, dutch oven recipes.
...

{"journeys": ["Modern Farmhouse Kitchen", "Raised Bed Vegetable Garden",
              "Sourdough Bread Making", "Cast Iron Recipes"]}

Notice what’s gone: no keyword extraction step, no embeddings, no clusters, no separate naming model. One model reads the log and writes the answer.

Some deliberate choices in the prompt:

• Actions collapse into a tiny vocabulary: search, save, like, click, comment, dislike. The model doesn’t need to distinguish a repin from a save-to-device; it needs to know both are strong positives. The prompt states the ordering explicitly: search/save > comment > like > click > dislike.
• Events are grouped by (date, action), with a semi-colon (;) separating Pins and a comma (,) separating tags within a Pin. That’s token economy, and it also gives the model a cleaner view of same-session bursts.
• Dislike is deliberately included in the input, as hides are evidence. The prompt says to suppress a journey whose support is predominantly dislikes unless there is evidence of significant related positive engagements.
• Temperature 0, reasoning tokens off, max 300 output tokens. We want a deterministic, replayable read of the same log, and with a fixed output schema for decoding purposes. Reasoning is turned off to reduce online inference cost.
• A 360-day lookback, up to 150 events. Journeys are long-horizon by definition, we experimented with different lookback windows and # of events to balance cost and relevance.

The output contract is deliberately simple: drop non-strings, dedupe case-insensitively, and the first occurrence wins. The output is parsed defensively because a generative system that occasionally emits prose where you expected JSON is a system that occasionally drops a user’s entire personalization payload.

The other thing worth saying out loud is that once you collapse the pipeline, the prompt becomes the ranking policy and the safety policy. Everything the staged system encoded in model weights and thresholds now has to be clearly stated. In production it asks for up to about twelve distinct journeys, each 1–5 words, in the user’s locale, ranked by:

1. Engagement strength (search/save over click)
2. Repeated support
3. Recency
4. Coherence
5. Specificity (“Mid-Century Coffee Table”, not “Furniture”)

It also encodes the failure modes we observed with the legacy journey clustering system: don’t merge unrelated topics; drop one-off outliers; treat auto-generated tags as noisy; lightly diversify the top of the list only among journeys that are already well supported. Empty output is allowed when the activity doesn’t support anything but inventing unsupported journeys is not.

These instructions aren’t just decorative. The diversity pass replaced a similarity-penalty diversifier. The dislike rule replaced a hand-tuned negative-signal filter. Few tag-noise instructions exist because auto-generated annotations are genuinely noisy, and the model will happily build a journey out of a classification artifact if you let it. Prompts like this deserve the same review discipline as model code, so we treat changes to them as launches.

Clustering was the right cold-start system, and it still exists as a fallback. Production inference is now LLM-first: one model, one decode, structured output.

Distilling a frontier teacher into a 4B student

Frontier models can infer coherent journeys without task-specific training, but they are too expensive to run for hundreds of millions of people on Pinterest regularly. Instead, we use them as teachers to generate synthetic training examples: each activity history is paired with a high-quality target journey. We also found that teacher outputs improved when the model was allowed to use additional reasoning tokens before producing its final structured response.

We fine-tuned open-weight Qwen3 students with 0.6B, 1.7B, 4B and 8B parameters. Supervised fine-tuning (SFT) teaches the model from the teacher-generated examples, while low-rank adaptation (LoRA) lets us update small adapters rather than every model parameter.

Quality improved rapidly during the first few thousand examples, then largely stopped increasing. Adding more similar examples provided little benefit; selecting diverse, high-quality examples — especially difficult consolidation and multilingual cases — was more valuable.

Model size presented a separate trade-off. The 0.6B and 1.7B students were faster, but produced less relevant and consistent journeys on the harder cases. Among the sizes we tested, the 4B model offered the best balance between output quality and serving throughput, so it became our production student.

Table 1: Same-user comparison of the staged clustering pipeline against the fine-tuned LLM. Fragments consolidated (Cabinets + Countertops → Modern Farmhouse Kitchen). Generic buckets got specific (Art → Penguin Art Drawing). Dropped intents came back (Cast Iron Recipes). Noise disappeared (Celebrity News, Memes). And user B got journeys in Spanish — something the keyword pipeline structurally couldn’t do, because it named journeys out of an English-annotation vocabulary.

Serving at notification scale

Scoring hundreds of millions of people on Pinterest is a batch problem with online-shaped latency. We run a full refresh when the algorithm changes, and a daily incremental pass over recently active users, so journeys move when behavior does.

We benchmarked the offline workflow against serving the same 4B model behind an online inference stack. Online serving won on throughput. The reason is a set of now-mature decode optimizations that a naive offline loop typically doesn’t get for free: PagedAttention, so the key-value cache isn’t wasted on padding; continuous batching, so a finished sequence immediately yields its GPU slot to the next user instead of waiting for the slowest prompt in a fixed batch. The serving stack is NVIDIA Dynamo in front of vLLM decode workers on L40S GPUs. Frontend and decode workers scale independently, and the frontend routes with key-value (KV) cache awareness so requests sharing prompt structure land on workers that already hold the relevant prefix, which matters because every request shares a long, stable system prompt. We enable prefix caching and chunked prefill and cap concurrent sequences per worker.

In production, a cluster on the order of 100 GPUs sustains about 775 requests per second end-to-end, with a median around 1.2 seconds and p95 around 3.0 seconds. The batch side is where the cheaper wins live. Inference runs as a daily fan-out over journey-eligible users, sharded into small per-job batches with bounded concurrency inside each batch, so a slow model response back-pressures one shard instead of the pipeline. The single most effective throughput lever we added wasn’t a GPU at all: we track when each user was last inferred and skip anyone processed within the last couple of days. Journeys don’t change hour to hour, and paying to re-derive an unchanged journey is the most expensive kind of waste — the kind that looks like progress on a dashboard.

If a decode fails, we retry that user rather than storing a partial list. An empty but valid JSON list is a successful inference; some users don’t have a supportable journey that day, and we’d rather send nothing journey-specific than a generic “Art”. We suppress inference of sensitive traits at multiple levels before anything is stored:

• Input keywords pass our notification content filter before reaching the prompt, so the model can’t build a journey on a keyword we wouldn’t have sent anyway.
• Output journey names pass that filter and our generative content safety filters, so names that are unsafe in a journey-inference sense are dropped even if they would pass a generic keyword list.
• The model is instructed never to infer or target topics related to health conditions, politics, religion, sexual orientation, gender identity, race, etc.
• Profile inputs are minimized, with ages bucketed at the low end and grouped, because the model needs a rough life stage rather than a birthday. Under-18 users in the EEA, Switzerland, and the UK aren’t eligible yet due to local laws.

Inference writes a ranked list of journeys per user into a store that notification serving reads at send time, and each journey name then becomes a retrieval query. We built embedding-based retrieval models to pull Pins for each user journey which are further ranked by our Pin ranking models. Journeys the user was recently notified about are fatigued, and journeys they’ve given explicit negative feedback on are suppressed outright. Only then does the notification framework decide whether to send at all, and on which channel.

How we know it’s better

There is no ground-truth label for a user’s latent goal. We evaluate at three levels, from inspection to a causal test.

• Same-user qualitative comparison: For journey-eligible users, we put clustering output and LLM output side by side on the same user activity. We keep both the strong cases and the honest misses: over-generalization (“Home Cooking” when the log is clearly “Spam Kimchi Fried Rice”) and under-generation (dropping a well-supported side interest). This is slow, but it’s the only way to see fragmentation and language quality that metrics flatten.
• Evidence-grounded diagnosis: A separate language model scores a sample of emitted journeys on axes we care about in production: relevance to the visible context, support from the profile and activity log (is this name faithful to what we showed the model?), diversity / non-redundancy across the list, and granularity. The support axis is the important one. A fluent, on-brand name that isn’t in the log is a failure, even if a user might have clicked it.
• Online experiment: Initial experiment ramped to the US and Canada users, with the clustering production system as control, showed lifts of 1.1% on email click-through and 1.3% on push opens, plus multiple statistically significant sitewide engagement and retention gains. A subsequent multilingual international rollout replicated the direction, and shipping both experiments increased the daily, weekly, and monthly active users on the platform.

What we’re building now

The fine-tuned Qwen-4B model is the production default. Everything in this section is under experiment — trained or rolling out, not what every user gets today.

Semantic IDs, so Pins aren’t a bag of tags

Tags mean every Pin costs a handful of tokens and drags along a dependency on Pin-to-text mappings. Semantic IDs (SIDs) are the alternative: each Pin is compressed by a residual-quantized variational autoencoder (RQ-VAE) into five levels of 11 bits each, which we pack into a single 64-bit integer and unpack at prompt-build time into special tokens:

<sid_begin><sid1_1263><sid2_973><sid3_725><sid4_793><sid5_1727><sid_end>

A general-purpose model has no idea what <sid3_725> means, so we bridge the gap with continued pre-training (CPT): extend the tokenizer with the SID vocabulary, then run next-token prediction over interleaved SID sequences, Pin titles, and tags until the model has mapped its world knowledge onto our catalog. SFT goes on top, so that given a SID sequence the model writes journeys.

SIDs only work better for users with dense engagement, so production is a fallback ladder: fewer than ten SID-carrying events and we fall back to tags for the whole prompt; a single event missing a SID falls back to tags for that event; and search queries always keep their literal text, because a typed query is the highest-fidelity signal we have. The bet is fewer input tokens, no dependency on annotations, and a shared item language between recommenders and LLMs. The cost is interpretability — a tag-based prompt can be read by a human mid-debugging session. A SID prompt cannot.

Offsite conversions as a complementary signal

Onsite search and saves remain the strongest intent signals, but they can be sparse for dormant or low-activity users. We therefore use recent offsite actions — such as page visits, add-to-carts, sign-ups, and checkouts — as a complementary signal, adding a compact, safety-filtered summary to capture timely shopping or planning intent beyond Pinterest.

Offsite evidence remains secondary: onsite outweighs offsite, and checkout is stronger than add-to-cart or a page visit. We only add it when onsite activity is sparse, cap its prompt footprint, and avoid ranking offsite-only journeys at the top. We are testing whether better prompting is enough or whether a fine-tuned offsite-aware model is needed.

One model, several products

Once inference is generative, adding an output is a prompt-and-fine-tune change rather than a new service.

• Multi-granularity journeys: Rather than a flat list of names, have one call emit each journey together with deeper cuts of it and adjacent topics that broaden discovery, so “Sourdough Bread Making” comes back with a specific next step into the interest and a sibling topic a user might jump to. That gives downstream retrieval a ladder of intent per user instead of a single rung.
• Copy titles: A good journey name isn’t always good notification copy, and we had engagement data showing which names underperform as titles (“Photography”, “Hairstyles”, “Quotes” — the generic ones again). One model now produces both, with a hard ordering constraint: finalize the journey names first, then write copy, and never reshape a name to make its copy sound better.

Closing the loop with User feedback

Distillation got us strong journeys, but it has a ceiling. The student is optimized to match the teacher, so its quality is bounded by the teacher’s sense of what users want. To go further, we need to optimize directly toward user preferences, using real feedback from users themselves.

Much of that signal already exists. Journey emails include a module asking “Are you interested in <journey>?” with Yes and No buttons. Each answer is logged, and a No immediately suppresses that journey for that user. Beyond these explicit answers, engagement inside journey-aware notification sessions gives us a much larger implicit signal: saves, clicks, and hides on the Pins we retrieved for a journey.

This feedback comes with a caveat: it is pointwise. Each response is a verdict on a single journey, while the model generates a ranked list. To bridge that gap, we split training into two stages. First, a mid-training stage applies preference optimization to single journeys across a large volume of feedback. Given a user’s activity, the model learns to make a journey that users confirm or engage with more likely, and a journey they reject less likely. Then, we formulated feedback-as-edit: the chosen response is the model’s list with the Yes journeys kept or the No journeys removed, and the rejected response applies the reverse edit. Starting from the mid-training checkpoint, fine-tuning restores list-level behavior: ranking, diversity, and the output format described earlier.

Improving the LLM with preference optimization and real user feedback is still an active area of experimentation. We are exploring how to weight explicit versus implicit feedback, and how to formulate a loss that is amenable to scaling, one whose gains follow a scaling law as feedback volume grows. Offline results are promising, and online A/B tests are now running. We hope to share more details in a follow-up blog post or publication.

Acknowledgements

We’d like to thank Justin Tran, Darien Boyd, Sarah Schwager, Shipeng Yu, Randy Tumalle, James Hernandez, Mengtian Hu, Dimitra Tsiaousi, Yaxin Li, Ying Wei, Sam Owens, Gloria Lee, Justin Pangilinan, Ye Tian, Rui Liu, Anya Trivedi, Binghui Gong, Tianqi Wang, Fangzheng Tian, Eric Tam, Mengying Yang, Lauren Fratamico, Andrew von Rosenbach, Ignacy Debicki, Zhenyu Tan, Jiajing Xu, and other contributors from Notifications, ATG Applied Science, ML Platform, GenAI Serving, and Content Safety teams for their insights and support!

From Activity to Intent: Generating User Journeys with LLMs was originally published in Pinterest Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

At Pinterest, reliable and trustworthy metrics are behind every decision: from measuring company-wide business performance to evaluating each feature experiment results. Hundreds of data producers — analysts, data scientists, and engineers across Pinterest — create thousands of metrics from our petabyte scale data lake. A metrics ecosystem this size can’t be ad hoc; it has to be organized, scalable, and resilient. So we built Metrics Board: an end-to-end platform that automates metric creation, enforces quality and enables agent-driven analytics.

In this post we walk through the product experience, architecture, integrations and explain some of the design choices we made to make metrics work at planet scale.

Why Metrics Matter at Pinterest

Most organizations rely on a familiar set of metrics — revenue, growth, engagement — to steer the business.

Metrics are how we understand whether Pinterest is delivering value to users and advertisers. Measures related to areas such as revenue, growth, and engagement turn the complexity of user behavior and Pinterest performance into a shared view of what is working, where we are falling short, and what matters most. These metrics drive the decisions our product, finance, and executive leaders make every day, from what to build to how we forecast and steer company strategy.

Metrics also power Pinterest’s product experimentation practice, which runs at massive scale — thousands of experiments are active at any given time. Those experiments are used as gates for each new feature launch. For results to be fair, comparable, and statistically significant, metrics must be clearly defined, reliable, and reusable. Beyond common core metrics, individual experiments often need bespoke metrics scoped to a single module, feature, or device. For example, a feature to add a new form of interaction with a Pin might need its own measure to evaluate engagement with the feature — even if core product engagement metrics are flat, the experiment might be successful if it creates a new form of user-engagement that a new metric would need to measure.

As much as key decision-makers need metrics, the way they get answers from data has changed too. Increasingly, the question isn’t typed as a SQL query or a dashboard filter — it’s asked in natural language to an agent. But agents are only as good as the trusted data they can access. For agentic analytics to work, metrics have to be discoverable, quality-checked and machine-readable.

Before Metrics Board

In the past, users were defining their metrics in bespoke data pipelines and wiring them by hand into our reporting and experimentation products. That approach had several problems:

• Time spent building similar data pipelines by hand was slow
• Inconsistent monitoring and availability were inconsistent, so breakages often only surfaced downstream
• Low awareness of existing metrics, leading to redundancy and trust erosion
• Metric metadata drifted from one place to the next with no source of truth

Put together, these problems slowed teams down, held experiments back, and left us with more metrics than we could trust. We needed a solution that would:

• Unify metrics across the organization
• Give us transparent definitions and code versioning
• Automate pipeline creation, orchestration and operations as much as possible
• Require and auto-generate documentation and metadata
• Automatically validate and monitor metrics for accuracy, reliability, and anomalies
• Give every metric and dashboard a clear, shared signal of quality and trust

Our goal was to make metrics easy for everyone who touches them — the people who produce them, the people who consume them, the applications that serve them, and, increasingly, the agents that reason over them.

The Solution

Define a metric once, Metrics Board handles the rest.

We built Metrics Board as a thin semantic layer for metrics on top of the data tools Pinterest already runs: a metric is defined once, and that definition drives everything downstream. It’s built around five pillars: inventory, compute, quality, publishing, and serving & discovery.

1. Inventory

The inventory pillar is our single source of truth for metric definitions. Every metric lives as a YAML spec in a dedicated Git repo, and is treated like any other production code, with version control, code review, and change history built in.

version: 1
metric_id: mb__metrics
display_name: Metrics Board metrics count
description: |-
  Count of metrics defined in MB. Test metrics aren't included.
domain: tech_data
metric_sql: |-
  with mb as (
    SELECT urn
    FROM query_analytics.pincat_db_latest_state_v2
    WHERE dt = '{end_date}'
    AND aspect = 'structuredProperties'
    AND is_active
  )
  SELECT 
    COUNT(mb.urn) as mb_metrics,
    '{end_date}' as dt,
  FROM mb
temporal_columns:
  - column_name: dt
    format: YYYY-MM-DD
    granularity: DAY
integrations:
  superset:
    enabled: true
  starrocks:
    enabled: false
ownership:
  nimbus_project: ii
compute:
  schedule: 0 9 * * *

One important consideration at the time of design was how much should Metrics Board be a true semantic layer. Do we enforce a model on top of existing datasets and have metrics defined as expressions leveraging the model? Or do we accept any SQL query and ask users to provide modelling information (temporal columns, metric or dimensions definitions) for the resulting data? We decided on the latter, driven by two main considerations: user feedback (“Put Pinners First” is one of our company values) and the sheer variety of tables which are used for metrics definitions — enforcing a semantic layer for each was simply not realistic.

We don’t ask teams to craft definitions by hand. The Metrics Board UI is a dedicated interface: users define metrics through guided forms, and the system generates the corresponding YAML and opens a pull request on their behalf. The result is one place to manage ownership, governance and metadata, that still integrates cleanly with the rest of the data stack.

2. Compute

To compute metrics, Metrics Board turns each definition into its own scheduled Airflow DAG. A definition carries all the metadata we need — schedule, SQL, validations, integrations — to generate a dedicated workflow. Isolating each metric in its own DAG lets us tune schedules, retries, and resources individually, and prevents a failure in one cascading to others.

At execution time, Metrics Board compiles each metric’s SQL into SparkSQL or PrestoSQL depending on the engine named in the definition. It then analyzes the SQL code to automatically create dependency tasks which wait on upstream data to be ready (we rely on sqlglot for parsing). It also automatically generates standard data quality checks and lets users define additional ones too (more on that below). Backfills are managed through Airflow, so owners can recompute history easily. Metrics Board also provides a layer for automated operations. For example if upstream data is late, Metrics Board automatically detects when it lands and resumes the metric computation DAG.

Taken together, the above listed features make metrics creation and management easy even for non-data teams. What used to take days or weeks to build can now be achieved in 1–2 hours — a significant boost for users’ development and experiment velocity.

3. Quality

A metric nobody trusts is worse than no metric. Every definition can declare validations that run as part of the metric’s own pipeline, backed by two engines:

• Rule-based checks for explicit expectations: day-over-day, week-over-week, month-over-month, and year-over-year thresholds, custom threshold rules, and comparisons against a parent metric — each applied to either a single point in time or a trend across time.
• ML-based anomaly detection that models a metric’s historical patterns — seasonality, trend, and normal variation — and surfaces deviations which a static rule would miss (you can read more about Warden here).

When a check fails, the right people hear about it in real time. Alerts fire as soon as validation runs, route to the owning team, and can be snoozed or acknowledged so the signal stays meaningful and doesn’t turn into noise.

All of this rolls up into a quality report per metric — per-check pass rates, a time series of results, gaps where validation didn’t run, and annotations (comments or linked tickets) — available via API and as a PDF export. Because the validations live in the same spec as the metric, quality is versioned and reviewed alongside the definition itself.

4. Publishing

A metric that lives only in its own pipeline isn’t useful yet — it has to reach the products people actually work in. Publishing does that automatically, driven entirely by the definition. Each metric declares its integrations, and turning a surface on is as simple as a flag: enable it, and Metrics Board fans the metric out to that surface with no per-surface wiring by hand. Today a single definition can publish to:

• PinCat (based on DataHub), our data catalog and metadata hub, so the metric is discoverable and its lineage is tracked
• Helium, our experimentation platform, so the metric is available to experiments the moment it’s defined
• Superset, our BI layer, where Metrics Board creates or updates the dataset and can stand up a monitoring dashboard automatically
• StarRocks, our analytics warehouse, for fast interactive querying

Publishing happens in two phases:

1. Metadata-level integrations (DataHub, Helium) publish at commit time as soon as the definition merges. The metric is catalogued and available to experiments before it has even run.
2. Data-level integrations (Superset, StarRocks) publish at compute time after each run when there’s fresh data to export.

Either way, the outputs (dataset IDs, the dashboard link, the output table, the pipeline URL) are written back to DataHub, so the catalog stays the single place to find a metric and everything attached to it.

5. Serving & Discovery

Discovery & governance. Metrics Board is where the org finds and evaluates metrics — but consumers have to be able to find a metric before they can evaluate it. Metrics Board centralizes definitions into a single source of truth, capturing rich metadata alongside them: ownership, description, certification grade, and quality signals.

That metadata is what turns centralization into discovery. Every metric automatically flows into PinCat, so users can discover them in the same place they already go for datasets. With metrics easy to find, the next question is what to trust. Two signals do that work, and they operate at two levels.

At the metric level, every metric carries a grade — a certification level that climbs from Unknown through C, B, and A up to A+ (fully certified and manually verified by a Data Quality Steward), with each rung demanding more quality work. It tells consumers what to trust and shows owners exactly how to level up. Furthermore, a metric’s key metadata — grade, certification state, and owner — is surfaced directly within its Superset chart.

At the reporting level, dashboards in Superset are classified into their own three tiers that reflect how critical and durable they are, and a dashboard’s tier drives three concrete things: how long it’s retained when unused, which Presto cluster its queries run on (and therefore its performance), and what documentation it must carry.

• Tier 1 is a small, curated set of company-critical dashboards. They carry the fullest documentation and are formally certified, and in return get the strongest guarantees — priority query performance and protection from automatic cleanup — with certification renewed periodically to keep them trustworthy.
• Tier 2 is for team-level dashboards others rely on. They require basic ownership and documentation and are retained as long as they see regular use.
• Tier 3 is the default for exploratory or test dashboards. They need no extra metadata and are cleaned up automatically once they go unused.

Together, metric grade and dashboard tier give consumers a trust signal at every layer of the stack, while retention and query-lane routing fall out of the dashboard tier automatically.

Agent enablement. The same structured metadata that powers discovery makes Metrics Board a foundational platform for analytics agents. We expose an MCP server so any LLM-based tools can find metrics, read their definitions, and reason about them. The clearest example is our Analytics Agent, which can answer natural-language questions about metrics. By referencing golden queries and certified metric definitions, the agent ensures its responses are grounded in trusted, governed data. As it turns out, the work we did to make metrics trustworthy for humans is what makes them usable by agents.

Impact

Adoption is the primary success measure we track, and we’ve exceeded adoption goals quarter after quarter — both in metrics created and in active users. Today nearly 150 new metrics are created each month (we also built out a lifecycle policy and engine to garbage-collect metrics no longer in use). Over 98% of metrics created for experimentation go through the Metrics Board today.

That adoption rides on dramatically improved velocity. On one recent high-profile project revamping key engagement metrics, Metrics Board saved roughly two engineering-months versus the old approach; metric additions that once took weeks now take days. At scale those savings translate to thousands of hours saved each month.

And velocity isn’t the whole story: the governance, quality, and cataloging we built along the way are quickly becoming the foundation for the LLM and agentic use cases we’re now shipping.

Acknowledgments

Analytics Platform: Krishna Gopal
Workflow Platform: Dinghang Yu, Martin Yau
Metrics Quality: Dennis Marcu, Raj Phull
Experiment Platform: Lu Yang, Xinyue Cao
Real-time Analytics: Charles Wu

Metrics Board: Building an Agent-ready Metrics Layer was originally published in Pinterest Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

This is the third post in our series on Pinterest’s next-generation database ingestion framework. Part 1 introduced the DB ingestion framework built on Kafka, Flink, Spark, and Iceberg, and Part 2 covered automated schema evolution. This post tackles another challenge in migrating downstream customers to the new ingestion framework: knowing when data is complete enough to read.

We’ll walk through that data completeness challenge and how we solve it: what “partition finalization” means, why it matters to downstream customers, how we built a unified mechanism to generate partition finalization markers, and how those markers are consumed. We’ll also look at how this mechanism generalizes beyond DB ingestion.

Background & Motivation

Before explaining partition finalization, it helps to define what a partition is and why a downstream consumer cares when one partition is “done.”

What is a Partition

A time partition is a subset of a table whose rows are grouped by a time column, e.g. all rows whose timestamp falls within a given hour. Grouping data this way lets a consumer read just the slice it needs, such as the 1 AM hour, instead of scanning the whole table.

How that grouping is physically represented depends on the table format:

• Hive uses explicit, directory-based partitioning: each partition maps to a physical directory in S3.
• Iceberg uses hidden partitioning: the partitioning is defined in the metadata layer rather than by directory layout, so consumers don’t need to know the physical file layout to query a partition.

What Finalization Means

Now consider a typical consumer of these tables: a downstream batch job that processes a single hour’s partition and runs only once for that hour. The consumer faces the question: when should my job run?

The answer is straightforward: Because the job runs only once, it should run only after the target partition is ‘finalized’, i.e. data in that partition is less likely to change. If it runs too early and the partition is still being updated after the run, it will miss later updates in that partition. And to know when a partition is finalized, it relies on the producer to put some signals on the data. For example, the producer can emit a lightweight marker in the table location when it thinks the partition is finalized, and the consumer will use a sensor to check that marker.

Challenge: Finalization in the Stream-based DB Ingestion Framework

In the previous batch-based system, finalization was trivial. The data was produced by a single batch job that read the whole source DB and wrote it in one shot, so once that job finished and there is data in a partition, we can consider that as finalized and downstream jobs could start consuming.

The new framework is stream-based. A Flink job writes change events continuously, committing many small batches into a partition over time. Because a record’s event time (when the change happened in the source) can lag when we process it, a partition can keep receiving data after its wall-clock hour has passed and no single moment marks it “done”, and the presence of data in a partition no longer means the partition is complete.

So finalization is no longer a byproduct of a job finishing. We have to infer it from the stream itself uniformly across the thousands of pipelines the framework supports, and a new mechanism is needed to support this.

Our Solution: EventTime Based Partition Finalization

We introduced a partition finalization mechanism into the streaming layer of the pipeline. The same Flink job that writes the CDC table also measures how far its event time has progressed and publishes a finalization marker into the destination Iceberg table, where downstream jobs can wait on it via a sensor before they start consuming.

Architecture Overview

There is no change in the underlying data flow as in Part 1 (see below figure, blue components are added for partition finalization): Kafka carries CDC events, Flink writes them into the CDC Iceberg table, and Spark upserts into the base table. The partition finalization logic mainly happens inside the Flink-to-Iceberg sink, plus one piece of published metadata that can be carried over in the Iceberg table.

Partition Finalization Logic

We’ll start with how a Flink-to-Iceberg pipeline works in general. As Flink processes the event stream, it first writes incoming records into data files. Those files aren’t visible to readers right away; they become visible during Flink’s checkpoint process, which runs every few minutes (configurable) to persist the job’s progress for fault tolerance. Every checkpoint produces a new snapshot in the Iceberg table: an atomic commit that also carries a small summary of metadata describing it. It works much like a Git commit, with each snapshot recording a new visible state of the table on top of the last.

With this process in mind, our partition finalization logic executes in the following procedures:

Event time extraction

Everything starts from a single value per record: the CDC event’s event time. Each pipeline configures which column carries it, and an extractor reads that column and normalizes it to a timestamp. Because the extractor is pluggable, the same machinery works for any table by simply pointing it at that table’s event-time column.

Fact collection: event time statistics

Between checkpoints, we collect the event time of every record processed within that window and track statistics over them. The core facts are the minimum and maximum event time seen in that window. On top of those, we also capture percentiles, e.g. the 99th and 95th percentile of event time, so we retain the shape of the distribution rather than just its extremes.

Fact storage: Iceberg snapshot summary

When a checkpoint commits, these statistics are written into the Iceberg snapshot’s summary, the per-snapshot metadata Iceberg already maintains for each commit. Every commit therefore carries its own self-describing record of what event times it contained, with no external store to keep in sync.

To keep the footprint small, we store the percentiles in a t-digest, a compact sketch that approximates a distribution in a small, fixed amount of space. The summary stays around a few hundred bytes whether a checkpoint saw a thousand records or millions, which is what makes it affordable to attach to every commit. The sketch is also mergeable, so partial statistics computed in parallel combine into a single summary at checkpoint time.

From fact to opinion: partition finalization watermark

After each commit, the framework turns those facts (the per-commit event-time statistics, such as the minimum event time seen in each checkpoint) into an opinion: how far the partition is finalized. It runs an algorithm over the statistics from recent commits and produces a watermark that marks the point up to which the partition is considered complete. By default, it takes the minimum of the per-commit minimum event times across the last X commits, then rounds down to the hour boundary.

The watermark is kept monotonically increasing. If late-arriving data would pull it backward, the value holds instead of regressing, so a finalized hour stays finalized. When that happens, the application can also send an alert to the data consumer, flagging that late data arrived after the partition was finalized.

Opinion storage: Iceberg table property

Unlike the facts which live on individual snapshots, the opinion is a single table-level value, so we store the partition finalization watermark as an Iceberg table property. Downstream jobs read it with an ordinary metadata lookup and gate their processing on it, without the need of any side channel or separate service.

Finalization Metadata Propagation

Since the statistics and finalization watermarks all live in Iceberg metadata, they are easy to propagate. Finalization is computed on the CDC table, but many consumers read the base table produced by the Spark upsert job, which carries the data but not the signal.

We close that gap in the upsert job: on each run it reads the finalization watermark from the CDC table and writes it onto the base table in the same commit. The signal follows the data to where consumers read it, and because these are ordinary Iceberg properties, propagation is just copying metadata during a commit the pipeline already makes without the need of side channel or services. Downstream jobs can then query those table metadata via sensors.

Underlying Implementation: Extensible Iceberg-Flink Sink

The above partition finalization logic is built entirely on top of the Flink-to-Iceberg sink. We needed two capabilities the standard sink did not have: (1) a way to collect per-record metrics inside the writer and merge them at commit time, and (2) a way to run custom logic immediately after every successful Iceberg commit. Rather than hard-coding partition finalization into the sink, we introduced two generic extension points — Accumulators and CommitProcessor — that any Flink-to-Iceberg pipeline can use.

Operator Topology

The standard sink has two Flink operators in sequence: a writer (one instance per parallelism unit) that flushes records to data files, and a committer (single instance) that makes those files visible by appending a new snapshot to the Iceberg table. The extensible sink keeps exactly the same topology and adds no extra operators.

When either extension point is configured, the writer and committer swap in extensible variants that carry accumulator and commit processor logic alongside. When neither is configured, the pipeline falls back to the standard operators with zero overhead.

Accumulators: collecting facts per record

An accumulator is a lightweight aggregator that runs inside each writer subtask and builds up a summary of the records it has seen since the last checkpoint. Between checkpoints it records and aggregates each record, extracting a field value and updating running statistics such as a count, a min/max, or a custom metric. At checkpoint time, each subtask takes a snapshot of its accumulator’s current state and resets it for the next window.

Those per-subtask snapshots travel to the committer alongside the data-file manifests. The committer, which sees results from every subtask, combines the individual snapshots into a single merged result. Because the subtasks may complete in any order, the merge operation must be order-independent: two snapshots combined must produce the same result regardless of which is folded into which.

Once merged, the accumulator’s value is written into the Iceberg snapshot summary under a namespaced key, sitting alongside the other metadata Iceberg already records per commit.

Commit processor: acting after each commit

A commit processor adds two hooks around the Iceberg commit: one that runs just before the snapshot is written, and one that runs just after it is durable. The pre-commit hook can set additional snapshot properties or abort the commit by throwing, which causes Flink to retry the checkpoint. The post-commit hook is where partition finalization lives: it reads the statistics just written to the snapshot summary, runs the watermark algorithm, and updates the table property if the watermark advances.

Both hooks receive the same context: a handle to the live table, the checkpoint identifier, and the merged accumulators from all writer subtasks.

Putting it together

The event-time accumulator collects the minimum, maximum, and a compact percentile sketch of event times across all records in a checkpoint window. At commit time those per-subtask observations are merged into a single global summary and written into the snapshot. The commit processor then reads that summary, together with recent snapshot history, runs the non-regressing watermark algorithm, and writes the result back to the table, all within the commit cycle without any external service.

Configurable Finalization Policies

Because each summary keeps the full distribution rather than a single number, the field the commit processor reads becomes a knob each table can turn. This is where the completeness-versus-latency trade-off is handled.

Most tables use the conservative default. Tables that value freshness and can accept a rare late record opt into a percentile instead, accepting that a small tail of records may arrive after their hour was marked safe to read.

Generalizing Beyond DB Ingestion

Nothing above is specific to database ingestion. The accumulator and commit processor only need a way to read an event time from each record; everything else, e.g. the sketch, the windowed lookback, the non-regressing watermark are generic. Any Flink-to-Iceberg pipeline that needs a completeness signal can adopt partition finalization by plugging in its own event-time extractor. And we are actively working on adopting this partition finalization mechanism into Pinterest’s next-generation Stream Ingestion framework.

Conclusion & What’s Next

Moving to streaming ingestion gives us fresh data in minutes, but it costs us the implicit completeness that batch loads provided for free. Partition finalization mechanism restores that guarantee: a cheap, mergeable event-time sketch on every commit, a non-regressing “safe to read” watermark derived from it, and a policy knob to trade-off completeness against freshness, and all of those are reusable beyond our DB ingestion framework.

In the following post, we will cover the incremental processing logic built on top of this ingestion framework and how we use it to support downstream use cases efficiently. Stay tuned for the future post: Incremental Processing on CDC Pipeline.

Acknowledgments

Huge thanks to my teammates Yisheng Zhou, Vi Nguyen, and Artem Tetenkin for building the Next Generation DB Ingestion at Pinterest together.

This project would not have been possible without the significant contributions and support of the following partners:

• Storage Services: Leonardo Marques Maciel Silva, Yuan Gao, Liqi Yi
• Storage Foundations: Tailin Lyu, Yu Su, Istvan Podor, John Grass
• Streaming Processing Platform: Kevin Browne
• Batch Processing Platform: Carlos Benavides
• Big Data Storage: Pucheng Yang, Mian Luo, Jenny Wang
• Data Warehouse: Bryant Xiao, Anita Narra, Pritam Pan
• Logging: Vahid Hashemian, Jeff Xiang, Jesus Zuniga

Special gratitude goes to Shardul Jewalikar, Ang Zhang, and Roger Wang for their continuous guidance, feedback, and support throughout the project.

Partition Finalization in Pinterest’s Next-Generation DB Ingestion Framework was originally published in Pinterest Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

Previously¹, we launched the next-generation serving stack for standard ads, which we call Nexus. Nexus decoupled candidate generation from scoring and moved us beyond the classic two-tower-only world, enabling richer model architectures while still meeting stringent latency and cost constraints.

Building on this system, we set out to design the first ads lightweight ranking model that goes beyond two towers. It jointly predicts three probabilities for each candidate ad: pCTR, the probability of a click; pGCTR30, the probability of a good click that lasts at least 30 seconds; and pOCTR, the probability of an outbound click to the advertiser’s destination. To support these objectives efficiently, we partition the query and Pin embeddings into task-specific CTR, gCTR30, and oCTR segments. For the CTR task, the fast two-tower prediction uses the first 64 dimensions of the CTR segment, while the three-tower prediction uses the full CTR segment together with richer cross features. For gCTR30 and oCTR tasks, full embeddings are shared between two-tower and three-tower predictions. This lets each task learn dedicated representations while sharing the overall model. In principle, Nexus places very few hard constraints on the architecture we can serve: cross-attention, sequence modeling, and more expressive interaction modules are all on the table.

However, in practice we quickly ran into the fundamental reality of ads lightweight ranking at Pinterest scale: for a typical request, we need to score on the order of hundreds of thousands of candidates (P99 post-targeting candidate counts can exceed 200K on some surfaces). We cannot simply keep increasing model complexity and expect to stay within our latency and cost budgets.

To strike a balance between latency and performance, we landed on a 3-tower co-train model design.

This design has a few key properties:

• We keep the query tower and Pin tower from the existing two-tower model, which lets us cache Pin embeddings offline and still obtain fast dot-product predictions for all candidates.
• We extend the architecture with a third cross tower that performs cross-attention between user sequences and candidate (Pin) features, plus an inter module that further mixes query, Pin, and cross embeddings.
• We co-train two-tower and three-tower predictions in a single model, giving us both fast but less accurate scores and slower but more accurate scores that we can deploy in different stages of the serving flow.

Put simply, the same model produces a fast two-tower score for every candidate and a richer three-tower score for a selected subset, so we can spend additional compute where it has the greatest impact.

Later in this blog, we will walk through the model architecture (cross tower and inter module), serving performance optimizations, and the two-stage scoring flow that leverages both two-tower and three-tower predictions.

By combining these changes, we maintained two-tower prediction quality while achieving around 30% reduction in offline loss for three-tower predictions across our engagement tasks, compared to the existing production model (details see below Offline Performance section). These offline gains translated into online lifts in CTR and gCTR30, reductions in cost per click (CPC), with a modest increase in infrastructure cost.

Model architecture

Our starting point was the existing two-tower engagement model, with separated query and Pin towers whose dot product feeds into task-specific heads. On top of this, we introduced two major components:

• A new cross tower that uses reduced-query cross-attention between user sequences and candidate features to capture high-order interactions.
• An inter module that jointly processes the query, Pin, and cross embeddings and produces a shared representation for all engagement tasks.

Below we describe the main design choices and trade-offs in each part.

Cross tower

The cross tower is responsible for modeling rich interactions between a user’s recent activity and a candidate ad. We use three on-site user sequence features (organic engagement, ads engagement, and search history) and four candidate features (advertiser ID, campaign ID, GraphSAGE embeddings, and Pin PinnerSAGE embeddings).

A natural first idea would be to build increasingly complex attention modules over these sequences and candidates. In practice, we explored several options:

• Merging full user sequences (across surfaces) and then running cross-attention with candidate features.
• Using shorter, truncated sequences to reduce compute and memory.
• Replacing attention with simpler interaction functions such as DIN-style pooling or average pooling.
• Crossing each sequence attribute with its corresponding candidate attribute individually, rather than merging first.

These variants exposed a clear trade-off: more expressive attention patterns (longer sequences, more attributes, per-attribute crossing) tended to improve offline loss but also increased latency, especially at high candidate counts. For example, using more complex cross architectures could reduce loss by several additional percentage points, but at the cost of tens of milliseconds of extra latency per request at 100K candidates.

We ultimately converged on an architecture that uses candidate side features to generate query tokens, which then interacts with user sequences to calculate attention. In this way, we can pick the most important candidate features and control the cost of transformer computation. This design gives us:

• Strong offline performance improvements versus production.
• A predictable compute profile that is easier to optimize and scale.
• A good balance between modeling capacity and serving latency.

The output of this cross tower is a cross embedding that summarizes how a user’s recent behavior interacts with a particular candidate ad.

Inter module

The inter module takes three inputs: the query embedding, the Pin embedding, and the cross embedding from the cross tower. Its goal is to produce a compact, shared representation that works well for all three engagement tasks (CTR, gCTR30, and oCTR), while keeping parameter count and serving latency under control.

Here as well, we evaluated multiple architectures:

• A deep MLP with DCN (Deep & Cross Network) layers.
• A standard MMoE (mixture-of-experts) with DCN.
• A top-K MMoE with DCN.
• A shared-bottom MLP with additive task-specific biases.

More complex structures such as MMoE with DCN achieved stronger loss reductions but also introduced noticeably higher latency compared to production. The shared-bottom MLP design provided a sweet spot: it delivered most of the performance gains while adding only modest latency, and it is architecturally simpler to optimize further.

In the final design, the inter module:

• Learns a shared logit for each task from the concatenated query, Pin, and cross embeddings.
• Adds task-specific biases computed from query and Pin embeddings for gCTR30 and oCTR.
• Outputs task logits that are then passed through sigmoid functions to produce probabilities.

This structure allows us to capture shared patterns across tasks while preserving enough task-specific flexibility.

Loss function

We train the model using a multi-task loss that combines main losses for the three-tower predictions with auxiliary losses for the two-tower co-train task.

The final loss takes the form of a weighted sum:

• Main three-tower losses for CTR, gCTR30, and oCTR.
• Co-train two-tower losses for CTR, gCTR30, and oCTR, computed from query–Pin dot products. For CTR, this fast auxiliary prediction uses the first 64 dimensions of the CTR embedding.

We tuned the task weights to balance learning stability and final performance, and landed on the following weighting scheme:

• Strong emphasis on main CTR loss.
• Moderate weight on main gCTR30 loss.
• Lower weight on main oCTR loss.
• Non-trivial but smaller weights on each of the co-train losses.

This configuration made the three-tower predictions the primary optimization target, while keeping the two-tower co-train task healthy enough to match or slightly improve on the production two-tower model.

The query tower produces a 192-dimensional task embedding: 144 dimensions for CTR, 32 for gCTR30, and 16 for oCTR. The Pin tower produces the corresponding task embedding and appends a 256-dimensional candidate-feature projection for the cross tower, producing a 448-dimensional Pin representation. For fast two-tower CTR scoring, we use only the first 64 dimensions of the 144-dimensional CTR segment. For three-tower CTR scoring, the inter module uses the full CTR segment together with the cross embedding. The additional Pin projection is computed in the Pin tower and cached offline, so the three-tower path can use these candidate features without per-candidate preprocessing at serving time.

Formally, our final loss is:

L = (100 * L_main_ctr + 5 * L_main_gctr30 + 1 * L_main_octr) + (10 * L_cotrain_ctr + 5 * L_cotrain_gctr30 + 1 * L_cotrain_octr)

Offline performance

We evaluated offline performance on held-out standard-ads data across three tasks (CTR, gCTR30, and oCTR), reporting relative loss reduction versus the production two-tower engagement model for both main (three-tower) and co-train (two-tower) predictions. The table reports ranges because we evaluated the model across multiple log sources; each endpoint is the result observed for a different source. For gCTR30, we observed a small degradation on the co-train loss which we deemed acceptable given the main task gains.

Across tasks, we observed:

Taken together, these results show that the co-train task maintains performance comparable to the existing production two-tower model, while the three-tower predictions deliver substantial improvements. This is important operationally: we can deprecate the standalone production two-tower engagement model and rely on the co-train head for fast scoring, without sacrificing quality.

Serving optimization

Serving a three-tower model over hundreds of thousands of candidates per request is expensive. To make the launch feasible, we invested heavily in model-level latency optimizations. Below are several techniques that had meaningful impact. Together, these changes reduced P99 model inference from over 200 ms to about 30 ms, leaving the final model only 1–2 ms slower than production.

Optimization 1: Move Pin pre-processing into the Pin tower

To perform cross-attention, sequence and candidate features must share the same dimensionality. In an initial design, we handled this with on-the-fly MLPs in the cross module, which added per-request compute proportional to the number of candidates.

Instead, we moved this preprocessing into the Pin tower. We append the processed Pin features to the original Pin embedding, increasing its dimension by an additional 256, and cache the resulting embedding offline. At serving time, the cross module can directly consume these enriched Pin embeddings with no additional per-request MLPs.

This change saved roughly 2 ms of latency at 100K candidates in our benchmarks.

Optimization 2: Lower precision

We also explored reduced-precision inference. By switching from FP32 to BF16 in the three-tower path, we significantly reduced model inference time while keeping model quality neutral.

On one representative benchmark, we observed:

• At 50K candidates, latency dropped from around 103 ms in FP32 to about 56 ms in BF16.
• At 20K candidates, latency dropped from around 45 ms to about 26 ms.

These gains played a key role in making the three-tower path practical at high candidate volumes.

Optimization 3: Late expansion of user features

In the three-tower engagement model, we compute predictions between one user and tens of thousands of candidate ads at once. User features are computed once and then expanded to match the batch size of candidates.

Earlier, this expansion happened just before the cross module to avoid duplicated computation. We realized we could delay expansion even further: instead of expanding before building the attention keys and values, we expand inside the cross module right before attention is computed.

This avoids redundant computation on large tensors and yields latency savings of around 15 ms at 100K candidates in our benchmarks.

Optimization 4: Pre-layer normalization in attention

Finally, we revisited how we apply LayerNorm inside the cross-attention module. Previously, we normalized the larger output sequence after attention, which has a batch size proportional to the number of candidates. We switched to normalizing the input sequence before attention instead; during serving this input has batch size 1, so the normalization work is much lower and independent of how many candidates we score.

During training, both options behave similarly. During serving, however, pre-layer normalization dramatically reduces the amount of work we do at large candidate counts. Flipping pre_lnorm from False to True reduced latency by about 3 ms at 100K candidates in our benchmarks.

Rethinking the serving flow

Even with model-level optimizations, running the full three-tower model on every candidate would still be too expensive. Post-targeting candidate counts can exceed 100K at P90 and reach up to over 200K at P99 on some surfaces. In early experiments where we scored all candidates with the three-tower model, model inference P99 latency exceeded 70 ms which is our timeout cutoff.

To tackle this, we redesigned the serving flow as a two-stage scoring pipeline that leverages both two-tower and three-tower predictions.

Stage 1: Fast scoring for all candidates

In the first stage, we use the two-tower head from the co-train model to score all candidates. These scores are combined into an initial utility: the overall ranking score that estimates a candidate ad’s value for the request and determines which candidates survive for later selection. This follows the existing production setup, but is powered by the new co-train model.

This stage is fast and inexpensive enough to run on the full candidate set.

Stage 2: Focused refinement with three-tower scoring

In the second stage, we identify the top-K candidates by utility and rescore only this subset with the three-tower model. Candidates outside this utility topK bypass the three-tower path and retain their two-tower scores.

We introduced a new hyperparameter, utility topK, which controls how many candidates the three-tower model sees. We tested several choices of the utility topK and measured both latency and downstream metrics such as clickthrough and web conversion impressions.

The trade-offs we observed:

• Smaller utility topK values reduce latency but can hurt web conversion impressions, because the final top-K selection stage needs a sufficiently large pool to satisfy different campaign groups and deduplication constraints.
• Very large utility topK values allow more candidates into the three-tower stage but directly increase latency, so we needed to pick a value that balanced candidate coverage and serving cost.
• Retaining candidates outside utility topK for later stages mitigates mixshifts with only a small additional latency cost.

We ultimately chose a utility topK of 40K. This threshold roughly corresponds to the 70th percentile for Home Feed and Related Pins and the 90th percentile for Search, which means the three-tower model scores the majority of candidates on most requests while keeping P99 latency within budget.

Final selection and bias considerations

After we obtain updated utility scores from the three-tower model for the utility topK subset, we blend them with the two-tower dot-product utilities. The final topK selection step then runs on this blended set of scores, selects pre-defined quotas from several candidate sources and performs deduplication to select the final set of ads shown to the user.

We made two design choices which may create concerns:

1. Heuristic selection of the utility topK subset for three-tower scoring.
2. Blending two-tower and three-tower utilities before final top-K selection.

Both heuristics are designed to favor high-utility candidates. In principle, they could bias the system toward items that already scored well in the two-tower stage, especially if three-tower predictions further amplify those scores.

To monitor this, we looked at calibration and mixshift. Encouragingly, we observed that the three-tower model actually reduces over-calibration for auction candidates, moving predicted CTR closer to realized CTR. We also did not observe a drop in standard web conversion impressions when using the blending option.

Online results and cost

In online A/B experiments on standard ads, the three-tower co-train model delivered around 1% gains in CTR, gCTR30, and oCTR, along with nearly 1% reductions in CPC, with only a modest increase in GPU spend.

Taken together, this represents a strong trade-off: meaningful engagement and efficiency gains for advertisers and users, with a small and well-understood increase in infra cost.

Conclusion

In this post, we walked through how we took Nexus beyond two towers by introducing a three-tower engagement co-train model for standard ads. On the modeling side, the cross tower and inter module allow us to capture richer interactions between user behavior and candidate ads. On the systems side, a combination of model-level optimizations and a two-stage scoring flow let us deploy this more powerful architecture while staying within tight latency and cost constraints.

Looking ahead, this work opens up several promising directions:

• Extending similar three-tower and co-train ideas to other objectives beyond engagement.
• Exploring even richer sequence modeling and attention patterns now that we have a scalable framework for late-stage scoring.
• Further tightening the feedback loop between offline architecture exploration, online performance, and infra-aware serving design.

Most importantly, it demonstrates that with the right system abstractions, we can continue to innovate on model architectures without losing sight of real-world constraints.

Acknowledgements

We thank Qingyu Zhou, Yuchen Shen, Li-Chien Lee, Qingmengting Wang, Zhixuan Shao, Tristan Nee, Sihan Wang, Lida Li, and Nuo Dou for their contributions to this project, and Renjun Zheng and Jamieson Kerns for their leadership support.

References

¹Yang, Xiao, et al. “Beyond Two Towers: Re-architecting the Serving Stack for Next-Gen Ads Lightweight Ranking Models (Part 1).” Pinterest Engineering Blog

Beyond Two Towers: Launching the 3-Tower Engagement Co-Train Model (Part 2) was originally published in Pinterest Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Billion-Embedding Challenge

At Pinterest, the “signal” is our lifeblood. Whether it’s a home decor enthusiast finding the perfect rug or a fashion seeker discovering a new aesthetic, our discovery engine relies on understanding deep semantic relationships to help our users find inspirations. Over the last few years, the explosive growth of embedding-based retrieval has fundamentally transformed how we surface these signals — and at the heart of that transformation is Manas, Pinterest’s in-house distributed search platform.

Embedding Retrieval is one of the core capabilities of Manas, supporting multiple approximate nearest neighbor search algorithms, hybrid queries with both token and embedding clauses, as well as real-time updates to ensure fresh contents become searchable within seconds. Deployed on over 80 clusters and serving billions of embeddings, Manas embedding retrieval powers all major product surfaces at Pinterest including Home Feed, Search, Related Pins, Ads, and Notifications.

However, as our corpus scales toward tens of billions of embeddings and our models capture increasingly complex interactions, we face mounting challenges around cost efficiency, scalability, and flexibility. On the infrastructure side, traditional ANN algorithms like HNSW are notoriously memory-hungry — they require the entire index to reside in RAM to maintain low query latency, making cost grow linearly with corpus size. On the modeling side, the classic two-tower retrieval paradigm is too restrictive: it reduces each candidate to a single embedding and scores relevance through a simple dot product, leaving little room to express richer, context-dependent notions of similarity.

To tackle these challenges, our team has been evolving Manas’s embedding retrieval stack across three fronts:

• Quantization. We reduce the memory footprint of embedding indices by compressing vectors into lower-bit representations with fewer effective dimensions. Quantization has been rolled out to all major use cases, delivering over 50% memory reduction in embedding indices and 20–30% cost savings in serving infrastructure.
• SSD-based Serving. Rather than holding entire indices in RAM, we serve ANN queries directly from SSD by carefully bounding the I/O per request — sustaining high throughput with low tail latencies at a fraction of the memory cost. Early experiments demonstrate a 10x reduction in memory usage and 40% CPU savings compared to in-memory serving
• Multi-embedding Retrieval. We move beyond the single-vector-per-candidate constraint of the two-tower model by supporting richer scoring functions that consider multiple embeddings per candidate. This unlocks more expressive ranking at the retrieval stage. We are currently partnering with a product team to launch a pilot use case.

In this blog post, we will delve into the technical details and results of each initiative, and discuss what’s next for embedding retrieval in Manas.

Quantization: Redefining the Footprint of High-Recall Search

For a long time, serving embeddings in 16-bit or 32-bit float precision was considered the common practice. But at Pinterest’s scale, raw precision is a luxury that often provides diminishing returns. We discovered that quantization — transforming these high-dimensional float vectors into compact integer representations — is one of our most effective ways to better cost efficiency.

Evaluating the Trade-offs: SQ vs. PQ

In the Manas stack, we focused our implementation on two primary quantization algorithms: Scalar Quantization (SQ) and Product Quantization (PQ). The core of these methodologies lies in partitioning the vector space into disjoint subspaces and mapping each vector into an integer representation: SQ applies uniform discretization from float to integer per dimension, while PQ performs a K-Means clustering in each subspace and maps a subvector to the ID of the closest cluster centroid.

Benchmarks on a 100-million-embedding GraphSage dataset confirmed this intuition. We evaluated both SQ and PQ across two ANN algorithms (HNSW and IVF) and observed a clear trade-off: PQ achieves higher compression but with a significant recall decrease, while SQ delivers strong compression with minimal loss on recall.

• PQ reduces the HNSW index by 74% and the IVF index by 93%, with a recall in the range of 70–80%
• SQ reduces the HNSW index by 59% and the IVF index by 75%, with a recall over 90% consistently

Given the trade-off demonstrated by the offline benchmarking exercise, we ran online A/B experiments in production to select the best performing quantizer for each use case, and ensure negligible impact on user engagement metrics when enabling quantization. We launched SQ and PQ across major product use cases, reducing the total memory allocation significantly and realizing 20–30% cost savings for serving.

Technical Deep Dive: SIMD and Linear Scaling

Shifting to 8-bit or 4-bit representations isn’t just a memory win; it’s a compute challenge. Usually, SQ requires a decoding step before distance computation, which can become a CPU bottleneck. To solve this, we implemented Linear Scaling SQ, which quantizes a vector by scaling only, and thus eliminates the decoding step before distance computation. The key enabler here is SIMD intrinsics, which allows the CPU to perform multiple 8-bit integer operations with each instruction, and hence reduces the total computing resources needed per query by 10–15% in our use cases.

SSD Serving: Moving Beyond the Constraints of RAM

Our journey of improving embedding retrieval cost efficiency led us to exploring alternative ANN algorithms that take advantage of recent NVMe SSD performance advancements. The current generation of SSD devices are roughly an order of magnitude cheaper per gigabyte, but with the latency increased from nanoseconds to microseconds, which can slow down queries if I/O operations are not carefully managed. To bridge this gap, we experimented with I/O-aware ANN algorithms that were designed to minimize the number of random reads issued per query while retaining a recall score as good as memory-based algorithms.

DiskANN vs. SPANN

In our experiments, we benchmarked two disk-based ANN algorithms, DiskANN and SPANN, with a 100M-embedding corpus collected from a Pinterest Search use case. While DiskANN is a robust graph-based approach, SPANN emerged as the better option for the Manas use cases.

Our team’s key observation was applying PQ quantization to the on-disk embedding store while retaining the full precision centroids helped the search accuracy and throughput significantly — a slight tweak from the original SPANN paper. This makes SPANN+PQ 4.5x faster than plain SPANN and achieves over 3x the QPS of DiskANN with 1/3 the latency, with a slight 5% recall drop.

Implementing SPANN in Manas

We implemented the SPANN algorithm in Manas, which stores the centroid index in the memory and the large posting lists in the disk, and guarantees both disk-access efficiency (low latency) and high recall by effectively reducing the disk access number per request. In the index-building stage, we adopt the hierarchical balanced clustering algorithm from SPANN for selecting the centroids, which ensures evenly distributed cluster sizes, and hence similar lengths of posting lists to keep the tail latency low. We build the centroid index using HNSW, which is well-suited for in-memory search over a relatively small set of centroids. In a preliminary evaluation with a Pin recommendation use case that indexes over 5 billion embeddings, our SPANN implementation saves over 40% of CPU time for production queries when compared with HNSW, with a <5% recall drop. Our next step is to adopt SPANN across all major use cases.

Multi-Embedding Retrieval: The Shift Toward Late Interaction

As we improve cost efficiency, we are also evolving the expressivity of the Manas embedding retrieval stack. The traditional “Two-Tower” model, while efficient, collapses an entire Pin or query into a single vector, often losing the nuanced, token-level semantics that define high-quality discovery.

Contrast in Paradigms: Sum of MaxSim

We are now moving toward Late Interaction models , such as ColBERT. Unlike the single dot product of two-tower models, late interaction represents documents and queries as lists of vectors. We use the “Sum of MaxSim” scoring logic to capture the maximum similarity between each query token and the document’s tokens.

Integrating this into Manas required comprehensive changes in our serving stack. We integrated the multi-embedding retrieval as a new query type and updated Manas to handle multiple query embeddings rather than a single vector per query. This required updating the Manas query parser to understand these complex queries, as well as running multiple ANN searches from a multi-embedding query simultaneously. Currently, we are working with a client team to launch a pilot use case for the multi-embedding query support in Manas. This represents the next frontier of Pinterest search: moving from “two tower” to true model-based retrieval.

Looking Ahead

The future of vector search at Pinterest lies at the intersection of infrastructure efficiency and retrieval model innovation. Over the next five years, our central goal is to evolve Manas embedding retrieval into an architecture that is scalable, cost-efficient, and open to new retrieval paradigms. We are pursuing this along three directions: adopting SPANN and SPFresh to push CPU and memory costs even lower for billion-scale indices; building first-class support for multi-embedding retrieval models like ColBERT that enable richer, interaction-based scoring at the retrieval stage; and embracing GPU-based retrieval systems like SilverTorch and TIGER to unlock new model capabilities.

Acknowledgements

Many people from Core and Ads Delivery Infra teams contributed to the projects discussed in this blog post. Special thanks to Ellie Madsen, Jennifer Kong, and Jiawei Kuang for working on various Manas embedding retrieval projects. Thanks to our collaborators from client teams, including Bowen Deng, Jiaxing Qu, Ryan Hou, Minhazul Islam SK, Wei-Ting Lin, J.J. Hu, Konik Kothari, Yujiao Guo, Hanlin Lu, Bella Huang, Ai Zhang, Janvi Palan. Last but not least, many thanks to Van Lam, Tao Yang, Deeksha Sharma, Kartik Paramasivam, Abhishek Tayal, Zheng Liu for leadership support.

Evolving Pinterest’s Embedding Retrieval Platform was originally published in Pinterest Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Why VLM Serving Matters at Pinterest

Pinterest is a visual search and discovery platform, so its AI systems must reason over both language and visual content. Vision-language models (VLMs), which can interpret images, compare visual candidates, and respond naturally to user intent, are becoming the foundation for the next generation of Pinterest experiences: Pinterest Assistant, hybrid search, multimodal reranking, content understanding, signal generation, content safety, and more.

This direction also reflects Pinterest’s broader strategy to customize open-source models to meet its product & scale needs. Pinterest Assistant is a standout example. This multi-turn conversational experience covers both user language and visual content. Serving it requires low-latency VLM inference over rich multimodal context as well as reworking Qwen3-VL with proprietary multimodal embeddings to cut runtime cost while improving performance.

Serving VLMs, however, introduces more challenges compared to text-only LLM workloads. Requests may carry multiple images, require extra vision encoder computation, incur larger and more variable prefill cost, and create higher KV cache pressure. To support this new class of models & product experiences, we built Pinterest’s VLM serving stack on top of NVIDIA Blackwell GPUs and NVIDIA Dynamo. Blackwell GPUs incorporate many architectural innovations that are uniquely positioned for today’s most demanding AI workloads — including higher BF16/FP8 compute throughput, increased memory bandwidth, and larger HBM memory capacity — that enable dramatically higher performance for inference. Dynamo provides a distributed inference orchestration layer that gives us the flexibility to optimize multimodal workloads across the full serving path, including disaggregated encoder/prefill/decode serving, multimodal support in the Dynamo frontend and vLLM, multimodal KV-aware routing with custom payloads, and KV cache offloading.

The Serving Challenge

As mentioned above, serving VLM for Pinterest use cases present several challenges:

Request payloads and preprocessing: For text-only serving, the request payload is just text (or tokens), so preprocessing is mostly tokenization plus applying a standard chat template; there are no external assets to fetch and no image-specific constraints. For VLM serving, the payload includes both text and images (URLs, base64, or precomputed embeddings), so the serving stack must download or load images, run image preprocessing (resize, enforce min/max pixels, normalization), map them into the model’s multimodal input format, and build prompts that mix both text and image content while inserting any required vision tokens or projectors, otherwise images get dropped or misinterpreted. Complexity is increased as requests include more images — Pinterest use cases sometimes require sending thousands of images per request.

Expensive prefill: In text-only serving, the expensive part is typically decode, with relatively modest and uniform prompt lengths, so KV cache pressure is more predictable. In VLM serving, requests often include many images or content items per query, which makes prefill dominant (encoding visual context is costly), drives much larger and more irregular KV caches, and requires our serving stack to support KV-aware routing, cache offloading, as well as careful prompt design to stay within latency and memory SLOs.

Multi-turn workloads: In text-only serving, multi-turn chat mainly increases prompt length and token costs but stays within a uniform text interface, so the serving logic is mostly about truncation and history management. In VLM serving, multi-turn workloads combine long dialog history with repeated or evolving visual context (e.g., revisiting or adding images/boards across turns), which makes prefill much heavier, complicates how visual state is represented across turns, and requires benchmarks and SLOs that reflect realistic multi-turn multimodal interaction patterns rather than single-shot prompts.

KV cache memory pressure: For text-only models, KV cache growth is driven by text token counts and is relatively predictable per request and per turn, so standard cache sizing and eviction often suffice. For VLM models, large visual contexts and long conversations can produce far bigger KV states per request, so serving must treat KV as a first-class constraint — using KV-aware routing, offloading, and disaggregated Encoder/Prefill/Decode designs — to avoid frequent evictions and maintain throughput under multimodal, prefill-heavy traffic.

Custom model and payload support: Text-only serving can often treat models as interchangeable behind a standard chat/completions API with generic JSON payloads and minimal per-model customization. VLM serving, by contrast, typically requires model-specific support for image fields, multimodal content arrays, projector layers (e.g., custom embedding projections), and custom routing or headers; the serving stack has to understand these payload shapes and model capabilities explicitly, and deployment artifacts and routers must be able to encode and route these richer, non-uniform multimodal requests correctly.

All of these VLM serving challenges required us to build a robust, flexible system that can meet our multimodal requirements. To build our serving stack, we relied on NVIDIA Dynamo’s multimodal serving capabilities.

Scaling with Blackwell and Building on NVIDIA Dynamo

Pinterest has been an early industry pioneer in adopting NVIDIA GPUs for online model serving at internet scale, starting with recommendation systems and expanding into LLM and VLM serving. Token costs and performance matters significantly for the viability of these products. Building on that foundation, we standardized on NVIDIA Blackwell GPUs B200 due to their market leading TCO for LLM/VLM inference. Blackwell also makes our LLM/VLM hardware stack future proof as our use cases and models continue to evolve. This cutting edge hardware enables Pinterest to be able to continue to get better TCO over time as we explore quantizations, improved kernels and multi-node inference.

Our in-house Gen AI Serving Solution is an end-to-end customizable stack centered around NVIDIA Dynamo as the serving orchestration framework. We use the OpenAI Chat Completions API, model-based Envoy routing and a model-aware gateway, Model Router, to provide a centralized way for all customers to call our system, ensuring a smooth client experience. Under the hood, we use vLLM as our inference engine and Weights and Biases for model management. Our compute infrastructure, PinCompute, is built on Pinterest’s internal centralized platform infrastructure that leverages AWS Elastic Kubernetes Service (EKS) along with NVIDIA GPUs. With the help of the Infra org, we manage dedicated EKS clusters that host all Gen AI Serving use cases. Notably, this is one of the first PinCompute EKS (PEKS) use cases at Pinterest. The Dynamo operator and components are installed through Helm charts, and our Dynamo workloads use the DynamoGraphDeployment CRD deployed as K8s manifests. To tailor the deployments to Pinterest’s requirements we inject additional sidecars and add custom containers to support functionality like Envoy (service mesh), model loading, and metrics scraping.

Our journey to Dynamo started with evaluating several Kubernetes-native frameworks that we found easy to start with but lacked flexibility in traffic management or forced reliance on a single inference ecosystem. We ultimately selected Dynamo as it is Kubernetes native, compatible with Pinterest Kubernetes and service discovery solution, inference engine agnostic, provides a flexible traffic solution, and uses a performant Rust-based router. During this process we developed a close relationship with the NVIDIA Dynamo team who have provided us with exceptional support. Pinterest utilizes many key features of Dynamo that provide flexibility and performance optimizations when powering our Gen AI Serving Stack.

P/D disaggregated serving

Our platform uses disaggregated prefilling and decoding inference to tailor serving to specific latency requirements (Time-to-First-Token (TTFT) or Inter-Token Latency (ITL)), optimizing hardware allocation by separating the distinct computational phases of LLM requests. This architecture is particularly effective for unblocking product launches with high traffic volume and tight latency requirements, especially for TTFT. Dynamo provides an easy to use solution to orchestrate distributed, disaggregated inference that allows us to explore the Pareto curve between the ratio of encoder (E), prefill (P), and decode (D) workers.

KV cache offloading

We leverage KV cache offloading, specifically via LMCache, for multi-tier offloading to CPU memory and disk, which is critical in high QPS, multi-turn scenarios. LMCache serves as a sophisticated extension for the inference engine, providing tiered storage across GPU, CPU DRAM, and local disk (NVMe) to preserve generation latency while managing high GPU memory pressure. This tiered approach includes asynchronous prefetching and compression, contributing to lower TTFT and increased throughput by effectively managing long-context scenarios where visual tokens would otherwise overwhelm available VRAM. LMCache fits seamlessly into Dynamo as one of the many KV cache integration option for KV cache offloading

Multimodal support in Dynamo frontend/vLLM

Pinterest’s image-based products have specific multi-modal serving requirements. We worked closely with the NVIDIA Dynamo team to develop corresponding multimodality features, including a frontend image decoder, multi-modal disaggregated serving, and multimodal KV router support to reduce recomputation for VLMs. These enhancements enable the serving stack to handle complex multimodal payloads, such as base64 encoded images or image URLs, and perform necessary image preprocessing directly in the Dynamo frontend. Furthermore, the implementation of multimodal KV-aware routing allows the system to track prefix cache overlap for visual content, which is essential for maintaining production latency in multi-turn interactions with high visual token counts.

Custom modality support: Projection Embeddings

Pinterest Assistant workloads often need to reason over large visual contexts: Pins, boards, products, and other image-heavy inputs that may appear across multi-turn interactions. Sending all of that context as raw image pixels is expensive for VLM serving because each request may require image loading, decoding, preprocessing, and online vision encoder computation before the language model can use the visual information. To reduce that cost, we added support for projection embeddings using PinCLIP, Pinterest’s internal image encoder, for generating Pin embeddings. Instead of sending raw images through the serving path, Assistant requests can send precomputed PinCLIP embeddings. Dynamo and the underlying inference engine then run a projector that maps those precomputed embeddings into the target VLM’s native visual token space, letting us reuse visual representations that already exist for many Pinterest entities while avoiding the most expensive parts of pixel-based image serving.

The performance impact of this approach has been significant. Compared with pixel-based image inputs in Dynamo, incorporating projection embeddings into Dynamo have yielded results that are substantially faster across our benchmarks: average gains are roughly 85x faster TTFT, 7.3x faster end-to-end latency, and 1.1x faster TPOT. Peak gains are even larger, reaching approximately 369x faster TTFT, 44x faster end-to-end latency, and 2.6x faster TPOT. Just as importantly, this makes much larger visual contexts practical: requests with 250 images represented as PinCLIP visual tokens reached latencies comparable to pixel-based requests with roughly 10 images, while carrying 25x more visual context. Even at that scale, the serving profile remained reasonable and production-ready.

Supporting this required changes across the API, artifact, serving, and engine layers. We introduced an updated ChatCompletions request format for projection embeddings, defined a model artifact contract so training and serving teams could package projector weights, model weights, and configs together into a single model artifact, added the new modality path in vLLM alongside image and video to decode embeddings, validate types and shapes, invoke the correct projector, and insert projected visual tokens into the model input sequence, and updated Dynamo to accept and route the new request format while preserving the multimodal contract. Adding multimodal KV-aware routing support for our modality delivered meaningful tail-latency gains: the strongest result improved TTFT p99 by 5.92x, and across the full benchmark matrix Multimodal(MM) KV-aware routing delivered about 1.42x average speedup on TTFT p99. Additionally, because embedding payloads are larger than ordinary text inputs, vLLM frontend processing latency became a bottleneck in some cases; Dynamo’s Rust frontend helps by providing an efficient path for receiving, parsing, and forwarding larger multimodal payloads.

Tool calling
Lastly, Dynamo fully supports tool calling with custom chat templates and multi-modal inputs, which is essential for providing flexibility for Pinterest’s agentic AI systems. This capability allows our agents to interact with internal tools and APIs in the Pinterest ecosystem, enabling more complex workflows that go beyond simple text for text and hybrid search, and other internal services. By leveraging custom chat templates, we can precisely define how the model should format its tool requests and handle the subsequent tool outputs, ensuring seamless integration with Pinterest’s internal services. Furthermore, the support for multi-modal inputs in tool calling means our agents can use visual information to inform their tool use, such as identifying an object in an image and then calling a specific search or recommendation tool to find similar products.

Benchmarking Real Multimodal Workloads with AIPerf

We use AIPerf, NVIDIA’s distributed benchmarking tool for standardizing our AI inference performance measurement, as the execution layer for our performance benchmarks. AIPerf is designed as a modular benchmarking framework, which makes it a better fit for complex generative AI workloads than tools focused mainly on single request/response patterns.

This is especially important as both Pinterest and the broader industry move toward agentic AI systems. These workloads are rarely a single model call. They often involve retrieval, routing, multiple model calls, tool use, multimodal inputs, and intermediate reasoning steps before producing a final response. This allows our optimizations to have grounding data and guardrails on whether we are improving or regressing.

AIPerf’s DAG support is a big part of why it works well for us. Instead of flattening an agentic workflow into one artificial request, we can model the actual execution graph: nodes represent meaningful stages in the system, and edges capture dependencies between steps. This lets us benchmark workflows that branch, fan out, join, or depend on earlier outputs, patterns that are increasingly common in real AI applications.

Just as importantly, AIPerf lets us shape the benchmark traffic to look more like real production usage. We can run benchmarks with configurable QPS, realistic Poisson request arrival patterns, multi-turn interactions, multimodal inputs, and configurable prompt characteristics such as system prompt size, prefix length, input token length, and number of visual/embedding items. This makes the benchmark less about testing an isolated model call and more about understanding how the full workload behaves under realistic load.

Internally, we pair AIPerf’s execution output with Pinterest-specific reporting. We use the results to power dashboards and summaries for latency, throughput, token usage, success rate, per-request details, and aggregate comparisons. That gives teams a practical way to compare runs, catch regressions, and understand whether a model or deployment can meet production SLOs under realistic multimodal and agentic workloads.

Product Use Cases Enabled

The VLM serving stack described above was built to support Pinterest Assistant, but the same architecture now serves as a reusable foundation for many GenAI and multimodal use cases across Pinterest. By standardizing on Dynamo for orchestration, vLLM for inference, and a common Chat Completions-compatible API, teams can launch new model-backed product experiences on top of this extensible serving platform.

Pinterest Assistant: Real-Time Multimodal Interactions

Pinterest Assistant is one of the first major product use cases enabled by this stack. As a conversational agent, Pinterest Assistant needs to support natural multi-turn interactions while reasoning over Pinterest’s visual content. A user may ask for help refining an idea, exploring a style, comparing products, or finding inspiration based on a set of Pins or images. Unlike a text-only assistant, this requires the serving system to handle both dialogue history and multimodal context in real time. Pinterest Assistant inference runs on NVIDIA B200 instances, which showed a greater than 2x latency improvement over Hopper during preliminary benchmarking.

Pinterest Assistant also benefits from custom modality support such as projection embeddings. Instead of always sending raw image pixels through the serving path, Assistant requests can use precomputed visual embeddings for Pinterest entities such as Pins, boards, and products. This allows the model to reason over much larger visual context while avoiding repeated image decoding and vision encoder computation, making richer real-time conversations practical.

A Shared Stack for Multimodal Product Patterns

As more Pinterest product surfaces adopt GenAI and multimodal models, this shared stack lets us support a growing range of patterns: conversational agents, re-rankers, OCR, safety systems, signal generation, and future VLM-powered experiences. The result is a serving platform that is not tied to a single product launch, but designed as a reusable foundation for multimodal AI at Pinterest.

Although Pinterest Assistant motivated many of the original requirements, the serving stack has grown to support a much broader set of use cases. Dynamo has become the out-of-the-box default for many GenAI serving workloads at Pinterest because it offers a flexible path for both text-only and multimodal deployment:

• Multimodal reranking uses VLMs to compare candidate content across textual and visual signals
• OCR workloads extract or reason over text in images
• Safety guardrails apply multimodal understanding to check whether responses or retrieved content meet product and policy requirements
• And more across signal generation, embedding-based workflows, and agentic systems

Dynamo’s LoRA hot loading has also accelerated experimentation under limited GPU capacity. Instead of standing up a separate full deployment per adapter — which increases GPU usage and operational overhead — client teams can load and evaluate multiple sets of LoRA weights dynamically against an existing base model. This shortens experimentation cycles and creates a smoother path from adapter training to production validation.

With a common serving foundation, teams reuse the same APIs, deployment patterns, routing layer, model management, observability, benchmarking, and GPU infrastructure rather than each building a custom solution. This gives product teams a paved path to focus on model behavior, integration, and evaluation. Dynamo is powering a reusable foundation for multimodal AI at Pinterest.

Lessons Learned and What’s Next

In building our Gen AI Serving platform, we’ve learned that VLM workloads are fundamentally prefill-heavy and cache-sensitive: encoding large visual contexts and long histories, not just decode, drives both latency and GPU memory utilization, so KV-aware routing, cache offload tiers, and disaggregated serving need to be designed explicitly. Dynamo’s multimodal KV-aware router and E/PD disaggregation and LMCache-based KV offloading turned out to be essential. We also found that payload design and routing are core serving problems, not just interface glue: the way we encode multimodal content arrays, choose image resolutions, and structure prompts directly determines whether Dynamo can reuse prefixes, route efficiently, and keep TTFT within product targets. On the evaluation side, we learned that benchmarks must reflect real multimodal product traffic — including multi-turn conversations, many images per request, and agentic DAGs — so we invested in AIPerf-based DAG benchmarks that mirror production QPS patterns instead of synthetic single-shot prompts. Finally, a shared serving platform built on Dynamo, vLLM, and EKS has significantly accelerated experimentation: once the stack supported multimodal routing, KV offload, and model management, new use cases like Pinterest Assistant and multimodal reranking could launch by reusing the same paved path instead of re-inventing infra per team.

Looking ahead, we’re investing in several new directions.

AI Configurator: Dynamo’s AI Configurator is a performance optimization tool that can simulate 10K+ deployment configurations in seconds, finding optimal prefill/decode worker counts, tensor/expert/data parallelism settings, and deployment parameters. It evaluates both aggregated and disaggregated serving architectures, and uses hardware-specific performance models to predict TTFT, ITL, and throughput across different GPUs. This tooling can help us create optimized deployments with lower lift, increasing performance and developer velocity across teams.

Dynamo Planner: As our workloads scale, so will the need to introduce autoscaling in order to maintain a highly available yet cost-efficient compute infrastructure. Dynamo’s Planner will provide a VLM/LLM-optimized autoscaler, which dynamically adjusts prefill and decode replica counts through four optimization targets: throughput (static queue/KV thresholds), latency (aggressive low-latency thresholds), load (user-defined prefill queue and decode KV utilization thresholds), and SLA (regression-based models targeting specific TTFT/ITL values)

Conclusion

NVIDIA Dynamo has given us a strong foundation for building Pinterest’s VLM serving stack and expanding it across emerging multimodal use cases. Its flexibility has been critical as we move from individual product launches toward a shared platform for production GenAI serving.

We’re excited to continue partnering with the NVIDIA Dynamo team and the broader community to push the limits of VLM and multimodal serving, and to make real-time multimodal AI systems faster, more efficient, and easier to deploy at scale.

Acknowledgements

This work would not be possible without the contributions from our partners and collaborators. Our thanks to:

Pinterest

AI Platform: Neha Upadhyay, Ananya Prabhu Angadi, Nazanin Farahpour, Howard Nguyen
Product ML Infra: Li Tang, Yayun Wang, Archer Liu
ATG: Yash Upadhyay, David Xue
Cloud Runtime Team: Vaibhav Shankar
CDP: Khoi Nguyen
Traffic: Peter Leng, James Fish, Scott Beardsley
Production Engineering: One Marino, Juan Pablo Daniel Borgna
Product Management: Colin Leatherbury
Leadership: Karthik Anantha Padmanabhan, Bo Liu, Roger Wang, Kartik Paramasivam, Matthias Zenger

NVIDIA
Elijah Soba, Qi Wang, Anthony Casagrande, Guan Luo, Kris Hung, Ryan McCormick, Harry Kim, Akshatha Kamath, Matthew Rawson

Building Pinterest’s VLM Serving Stack on NVIDIA Dynamo was originally published in Pinterest Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

A Fundamental Transformation

An AI team is fundamentally more than just a group whose members incorporate AI tools into their existing workflows. The journey to becoming an AI team necessitates a fundamental and comprehensive paradigm shift in how the team defines ownership, engages in strategic planning, and, most critically, executes on its core goals and objectives. This transformation is not merely an addition of new technology; it is a restructuring of the team’s operating model, philosophy, and individual roles.

In an AI-centric environment, every single member is significantly empowered, not only through access to cutting-edge AI tools and sophisticated models but through an expanded scope of responsibility and influence. These new capabilities allow individuals to automate routine tasks, accelerate data analysis, and rapidly prototype solutions, freeing up cognitive resources for higher-level, more strategic thinking. The expectation shifts from simply completing tasks to orchestrating intelligent systems and focusing on solving problems that were previously intractable.

Consequently, the very nature of the work for both individual contributors and managers will be fundamentally different from anything that has come before.

At Pinterest, this isn’t theoretical. Our infrastructure teams sit at the core of a product that serves billions of Pins, boards, ads, and real-time signals. That reality has forced us to treat “becoming an AI team” as an operational necessity, not a side project as we cannot keep scaling reliability, cost efficiency, and developer productivity using only traditional playbooks.

AI Team Capabilities

The advent of accessible Artificial Intelligence (AI) tools has fundamentally raised the performance ceiling for what development teams can accomplish. To illustrate this profound shift, consider a normally challenging project — for example, refactoring an entire legacy codebase to replace a widespread used process named “foo” with a more modern one named “bar.” This type of undertaking, which can be critical for long-term maintainability but often low on the priority list due to its sheer scale, would have previously required a dedicated team of several engineers and a timeline spanning many months, potentially a year or more.

With the seamless integration of modern AI tools, projects once considered infeasible or prohibitively expensive in terms of human capital and time can now be completed with incredible speed and accuracy. The very refactoring example mentioned above, utilizing sophisticated code-generation and transformation models, can now be executed and verified in a matter of a week or less.

This new capability changes far more than just project timelines; it fundamentally reshapes how organizations and teams must approach prioritization. The set of “possible projects” that are technically within reach has grown dramatically, multiplying the complexity of deciding where to focus energy. Consequently, the act of prioritization has become even more critical. Teams must internalize a new maxim: Just because something is feasible does not mean it is strategic. The ability to accomplish a task quickly with AI does not inherently align it with the core goals of the business or product.

By offloading repetitive, high-effort engineering tasks to AI, teams will fundamentally shift their focus. Engineers, product managers, and designers must dedicate more time and energy to high-level strategic thinking, in-depth problem definition, and validating genuine user needs, rather than getting caught up in execution details. Consequently, the core value of an engineering team moves from being expert coders to becoming expert strategists and problem-solvers, using AI as a powerful force multiplier for execution.

The Manager’s Evolving Role

As the daily work of individual contributors, particularly engineers, becomes increasingly elevated and augmented by AI, the role of the manager undergoes a profound transformation. Managers are systematically freed from the time-consuming, routine tasks traditionally associated with project management and oversight such as the meticulous tracking, status updates, resource allocation for predictable tasks, and micro-managing process adherence.

This liberation allows managers to pivot from being primarily process enforcers and administrative overseers to becoming true strategic guides and visionary leaders. Their focus shifts dramatically toward:

1. Strategic Direction and Vision Setting: The primary responsibility evolves to identifying which ambitious challenges and emerging technologies merit the team’s concentrated pursuit. This involves a higher-level view of the product roadmap, understanding market shifts, and translating the organization’s overarching goals into clear, actionable, and inspiring technical objectives for the team.
2. Maximizing Collective Impact: Instead of simply ensuring projects are completed on time, the manager’s role is to steer collective efforts toward the points of maximum strategic impact. This requires deep critical thinking, understanding the leverage points within the architecture or product, and making difficult trade-off decisions that align the team’s output directly with the most significant business outcomes.
3. Talent Development and Mentorship: With AI handling much of the repetitive code generation and debugging, the value of the human engineer shifts to creativity, complex problem-solving, and system-level architecture. Managers must therefore focus intensely on coaching and mentoring team members, helping them develop higher-order skills, navigate ambiguity, and define career paths that capitalize on advanced technical and strategic competencies.

AI work at Pinterest

Across Pinterest, we’re seeing similar patterns. Data and platform teams are building agents that write and validate Spark jobs, triage alerts, and help product teams explore data without deep query expertise. Our design, PM, and support functions are using AI for brainstorming, content drafting, and faster decision-making. Initiatives like our AI Acceleration Learning Sessions and GenAI productivity working groups are making these wins visible and repeatable, so teams don’t have to rediscover the same patterns in isolation.

For infrastructure teams, this has a very concrete flavor: agents that surface distressed databases before they page an engineer, tools that propose safer rollout plans, and coding assistants that make large-scale refactors on our storage platforms possible in days instead of quarters. Those are the kinds of capabilities that make an “AI team” feel less like a slogan and more like an everyday operating model.

Future Teams Are AI Teams

The integration of Artificial Intelligence into the modern workplace is not merely an incremental technological upgrade; it represents a fundamental shift in how work is conceived, executed, and managed. For every professional, from the newest individual contributor to the most seasoned manager, adapting to this new paradigm isn’t optional, it’s imperative for survival and success. The era of the “traditional” high-performing team is rapidly giving way to the AI-enabled team, and the difference in capability is staggering.

The core reality is that the capacity, efficiency, and innovative potential of AI-enabled teams will be an order of magnitude greater than their predecessors. AI tools will automate routine, repetitive tasks, freeing up human talent to focus on complex problem-solving, strategic thinking, and creative endeavors that require uniquely human judgment and emotional intelligence. This shift doesn’t eliminate roles; it elevates them, demanding a new set of skills centered around prompt engineering, data interpretation, critical thinking, and collaborative interaction with intelligent systems.

To remain effective, competitive, and truly innovative, teams and organizations must fully embrace this transformation. This involves more than just adopting new AI tools; it requires a deep cultural change.

The future of high-performance is intrinsically linked to this symbiotic relationship between human expertise and machine intelligence. Teams that hesitate will quickly find themselves outpaced by competitors who have successfully made the leap, highlighting that embracing this AI-driven evolution is not just a strategic advantage but is rapidly becoming the foundational requirement for sustainable innovation and effectiveness.

Previous Capacity & Commitments Model

Teams would assess available capacity for a work cycle and commit to specific deliverables. The expectation was to meet all commitments. In reality, factors like underestimated effort or reduced resource availability typically led to falling short (e.g., achieving 70% of commitments).

Initial Impact of AI Tools

AI tools (for documentation, coding, etc.) have been available and used by some individuals, leading to a modest increase in overall team efficiency. This early, non-uniform adoption has yielded minor capacity gains (e.g., an estimated 20% increase). Consequently, teams are performing better against commitments, but still slightly missing the mark (e.g., hitting 90%), meaning the team is still operating at 100% capacity (fully busy).

On our Storage Foundations team at Pinterest, we’ve already started to see this shift in very real terms. For example, work such as investigating distressed database clusters could previously require several days of effort is increasingly being front-loaded by AI agents that sift through logs, generate candidate hypotheses, and even draft remediation commands

We’re also starting to see this show up at the portfolio level. AI-assisted tooling around MySQL, TiDB and Cache Infrastructure has let us move from “only what’s absolutely urgent” to clearing whole classes of maintenance work that used to sit in the backlog indefinitely. That doesn’t mean we’ve “saved X engineers”; it means the same engineers are now spending much more of their time on roadmap-level questions instead of chronic firefighting and manual KTLO.

The Need to Pass the Tipping Point

To fully realize the benefits of AI, the team needs dedicated time for skill development and deeper integration of the technology. AI adoption increases team capacity. The crucial hurdle is reaching a point where the team’s increased capacity exceeds its commitments. This surplus time is essential for developing AI proficiency. The current challenge for most teams is this transitional phase: simultaneously learning and using new tools while struggling to meet existing deliverable expectations.

Navigating the Chaos: The Role of Leadership in the AI Transition

The transition to an AI-powered work environment is inherently complex and often turbulent. Teams are confronted with an overwhelming deluge of information, including a proliferation of AI tools, new academic research, industry blogs, online courses, and detailed demos. This sheer volume can quickly lead to paralysis and confusion: Which specific tools offer the best return on investment? What are the foundational concepts or practical skills I absolutely need to acquire? How do I strike a sustainable and productive balance between the necessary phase of exploration and real-world execution and delivery? The difficult truth is that no universal, one-size-fits-all answer exists to these critical questions.

In this phase of uncertainty, managers assume a role that is not merely supervisory, but truly critical. They must become the primary navigators, charting a course through the technological uncertainty. This guidance involves several key responsibilities:

1. Guiding Through Uncertainty: Leaders must acknowledge and validate the team’s confusion and anxiety, providing a clear vision of the destination even when the immediate path is obscured. They filter the noise, helping the team prioritize a manageable set of tools or learning tracks that align directly with organizational goals.
2. Encouraging Information Sharing and Experimentation: Managers should actively dismantle silos, creating structured and informal forums such as brown bag sessions, dedicated Slack channels, or internal demo days where team members can share their discoveries, successes, and, crucially, their failures. Furthermore, they must allocate dedicated time and resources for experimentation, framing it not as a diversion from work, but as an essential part of the new workflow.
3. Creating a Culture of Psychological Safety: Perhaps the most important leadership function is establishing an environment where it is genuinely “okay to learn and adapt.” This means normalizing the fact that competence in AI will be a journey, not an instant state. Leaders must replace the fear of failure with an appetite for fast, informed learning, celebrating the insights gained from an unsuccessful experiment as much as a successful deployment. This psychological safety accelerates the learning curve and transforms fear into proactive engagement with the new technology.

Guiding Your Team Through the AI Transition

The integration of Artificial Intelligence represents not just an incremental technological update, but a fundamental shift in the landscape of daily work. While it may unfold more gradually than a traditional corporate reorganization, the transition to an AI-augmented environment will prove to be an even more profound, systemic transformation, altering core processes, roles, and skill requirements. It is crucial for leadership to acknowledge that change, even when promising vast positive outcomes, is an inherently stressful process for employees. Proactive and empathetic guidance is essential to navigate this period successfully.

For individual contributors, the core of their professional identity and daily tasks will evolve significantly, shifting the focus from execution to strategy, guidance, and unique human insights.

Prior professional experience remains absolutely essential. However, the application of that experience will fundamentally change. Instead of primarily using experience to execute tasks manually, team members will leverage it for guiding, critiquing, and collaborating with advanced AI agents and tools. Their role shifts to one of a skilled orchestrator, using their deep understanding of the problem space, customer needs, and business context to ensure the AI’s output is accurate, relevant, and fully aligned with strategic goals.

As AI takes on an increasing share of repetitive, data-intensive, and complex implementation work (“the how”), the demand for a strategic mindset within individual roles will escalate. Team members must become much more strategic partners, actively participating in and driving decisions about what work is most valuable to pursue, why it matters, and how to define success. This requires moving beyond merely fulfilling assigned tasks to critically analyzing the business problem, prioritizing initiatives, and defining clear, high-leverage prompts and guardrails for the AI to follow. This strategic contribution elevates the value and scope of every role.

Conclusion

Embracing this holistic shift thoughtfully, strategically, and together is not optional. It is essential for engineering teams aiming to thrive, innovate, and lead in this new era where intelligence is the core layer of the product. The successful team will be one that is centered around the continuous optimization of the AI-driven system.

Becoming an AI Team was originally published in Pinterest Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Pinterest home feed candidate generation is a large-scale User-to-Pin retrieval problem. A common approach is a two-tower model: a user tower encodes the user, an item tower encodes candidate Pins, and approximate nearest neighbor search retrieves Pins close to the user embedding. But Pinterest users often have multiple intentions at once — planning a renovation, saving recipes, exploring fashion, or organizing travel ideas. A single retrieval embedding can struggle to capture this diversity.

Conditional Learned Retrieval, or CLR, extends the two-tower setup by conditioning the user tower on an explicit retrieval context. Instead of producing only one user embedding, CLR can generate condition-aware embeddings that reflect different aspects of a user’s interests while still grounding retrieval in the user’s overall behavior.

Prior Pinterest work studied this formulation in two settings. The RecSys’24 paper: Bootstrapping Conditional Retrieval for User-to-Item Recommendations described how to bootstrap conditional retrieval by constructing training data for (user, condition) -> item retrieval from existing user-item and condition signals, and applied it to interest-based notifications. The KDD’25 paper: Synergizing Implicit and Explicit User Interests: A Multi-Embedding Retrieval Framework at Pinterest placed Conditional Retrieval within a broader multi-embedding retrieval framework for home feed, where explicit interest conditions complement implicit interests extracted from user behavior.

In this blog, we describe how CLR evolved from early interest-conditioned retrieval into a broader retrieval system for Pinterest home feed. We focus on three areas: expanding CLR to support more retrieval use cases, scaling the model foundations through sequence modeling and more general condition representations, and redesigning the serving infrastructure to make multi-condition retrieval efficient at production scale.

Expanding CLR Across Use Cases

Increasing diversity of the retrieval candidates is a reliable source to drive engagement impact, because they provide a broad range of content for ranking and blending to work with. With CLR, this becomes scalable by either expanding to new types of conditions or providing new sources of conditions given a user.

To support each condition type, we need to train CLR models using (user, condition, engaged Pin) triplets. Fortunately we already have a few heuristic-based candidate generators in home feed that can help bootstrap these use cases.

Interest Conditions
At Pinterest, we have a predefined interest taxonomy to categorize Pins. We initially launched CLR in home feed by sampling a few interests from user-to-interest signals. Later we leveraged a new user interest signal generated from LLMs as conditions.

Pin Conditions
We later introduced Pins as conditions, by clustering users’ recently engaged Pins, selecting medoids of those clusters and leveraging pre-training embeddings to represent them in the model. In doing so, we were able to deprecate legacy heuristic CGs in favour of CLR; yielding impressive metric wins while simplifying our serving stack.

Board Conditions
At Pinterest, users interact with Pins which are distributed across various Boards on the platform. We can think of Pins and Boards as forming a bipartite graph with Pins on one side, linked to Boards on another. By leveraging random walks, we can traverse this graph to recommend not only Pins, but also Boards to users. Similarly to the above, we then constructed Board conditions using pre-training embeddings in the same space as Pins to represent them, yielding large metric wins and simplified serving stack once more.

Agentic Condition Budget Tuning
Following the migration to GPU-based serving, which significantly reduced latency and freed capacity, the team leveraged Claude Code to automate experimentation by continually reading experiment feedback to optimize the number of conditions inferred per request for CLR across Interest, Pin and Board conditions. By utilizing automated tuning policies, we successfully improved key engagement metrics including with low cost increase.

Pin tower improvement: large id embedding table and semantic ids

ID embedding is an important feature in our recommendation systems. We began with using image signatures to represent Pin ids (ref), which behave like random hash numbers. Since this id space is huge (billion-scale), we ended up pretraining an id embedding table of size 20GB for sufficient memorization and tolerable collisions. We used TorchRec to shard this embedding table across multiple GPUs to support efficient training.

We later introduced semantic ids to complement image signatures. We built semantic ids by quantizing static content embeddings (PinClip fusion ref) using residual-quantized VAEs. These semantic ids are hierarchical codes that cluster similar contents together, thus visually and semantically similar Pins could share nearby codes. Long-tail Pins suffer from cold start issues and limited training data. Semantic ids enable long-tail Pins to share the same id embedding space with popular Pins, thus can borrow their collaborative signal. Our semantic ids have five hierarchical layers, each layer has 2048 possible codes. Instead of using another large id embedding table, we found using five small embedding tables (one per layer) for these codes to be sufficient. After semantic id embeddings are looked up from these tables, we use a stack of five MLP layers to fuse these id embeddings sequentially. Finally, the image signature embedding and semantic id embedding are fed into the feature cross layer on the Pin tower.

Scaling CLR Model Foundations

Scaling User Understanding: From User Sequences to Foundation Models

Conditional user sequence transformer
Early CLR models could condition retrieval on an explicit context, but the user representation was still largely built from static or aggregated features. This made it harder to decide which parts of a user’s recent history mattered for a given Interest, Board, or Pin condition.

To make CLR sequence-aware, we introduced a Conditioned User Sequence Transformer. The model converts raw condition features into condition tokens, appends them to the user sequence, and encodes the combined sequence with a Transformer. The condition token acts like a query over recent user actions, helping the model focus on the sequence signals most relevant to the retrieval context.

The Transformer outputs encoded condition tokens, encoded recent user sequence tokens, and raw condition features through a residual path, which are passed into the User Tower DHEN layer. This established the core abstraction for later model scale-ups: represent the condition as tokens, append them to user history, and encode the extended sequence.

The next step was to make this encoder more powerful. As CLR became more central to home feed retrieval, we wanted to capture richer action context, longer-term behavior patterns, and stronger alignment between user history and candidate Pins. This motivated the move toward a Foundation Model based CLR architecture.

Foundation model in CLR
To capture deeper user sequence understanding, we upgraded the sequence encoder by integrating the PinFM (ref) into the CLR user tower. The Foundation Model is a large-scale transformer trained on global user action sequences across multiple Pinterest surfaces before being fine-tuned within the Unified CLR framework.

The foundation model uses a large ID embedding table in addition to OmniSage embeddings to memorize the engagement-oriented representations of Pins. This ID embedding table is reused even at the Pin tower to encode the candidate Pin. We believe that using this consistent representation of Pins on both sides of the tower will make it easier for them to align. To reinforce the pretraining objective during finetuning, we also use the next token loss during fine tuning as this helps the model adapt to the user sequence in the CLR dataset. We use positive actions such as saves, repins, sends and downloads to construct this loss. In addition to pooling the user sequence and condition outputs, we also apply an attention pooling to get a weighted average of the user sequence transformer outputs. This would give the feature cross an holistic view of the entire sequence.

We also introduce a contrastive alignment loss between the condition token representation and the candidate Pin representation. This encourages the condition token to be closely aligned with Pins embedding space, which makes it easier to rationalize a Query-key similarity in cross-attention.

Scaling Condition Representation: Unified CLR and Router Simplification

As CLR expanded beyond its first use cases, the condition interface became an important scaling challenge. Supporting each condition type with its own model, feature, and serving setup would make every new condition more expensive to launch and maintain. To scale CLR, we needed the model to support heterogeneous conditions through a shared architecture.

Unified CLR
The first step was Unified CLR. Before unification, home feed served separate CLR models for different condition types: Interest CLR used an interest ID in the user tower, while Board CLR used a board embedding. Both were effective, but maintaining separate models created duplicated work across training pipelines, feature upgrades, experiments, serving configs, and indexing.

Unified CLR consolidated these use cases into a single model trained on both interest and board conditions. The model was modified to accept multiple condition types, with missing condition features imputed by default values, and training used conditional filtering to focus on examples with valid conditions. This gave the team one shared foundation for future condition types, surfaces, feature upgrades, and model architecture improvements.

Router Simplification
While Unified CLR brought our models under a single roof, the interface for defining conditions still suffered from an underlying scaling bottleneck. Historically, our routing logic was highly condition-specific. Introducing a new condition type (e.g., Board, Interest, or Pin) required adding bespoke features to the model and heavily zero-padding the master feature container. This legacy approach led to feature explosion, siloed learning, and high engineering maintenance overhead. To break this bottleneck, we refactored the routing logic into a condition-agnostic Slot Architecture. Instead of creating custom pipelines for every new condition, we bucketed incoming features into three predefined, shared slots:

1. condition_os slot: Houses Omnisage-compatible embeddings (e.g., Pin OS, Board OS, or interest text embeddings).
2. condition_id slot: Houses specific identifier embeddings (e.g., Interest IDs or Semantic IDs).
3. condition_type_id slot: Encodes the condition type into a 32-dimensional learned embedding to preserve candidate generator identities.

Scaling CLR Training Efficiency

Integrating a large-scale foundation model into the user tower significantly increased computational complexity. To maintain high developer velocity and support larger batch training, we introduced two key lossless infrastructure optimizations that increased our training throughput by 2x.

Request-level-training
Within a single training batch, user features are often repeated across different candidate conditions. We deduplicate these identical user features using a unique combination of user-id and condition identifiers. This drastically reduces the effective batch size passing through the user tower. Post-forward pass, the generated embeddings are re-duplicated to match the original batch size for contrastive loss calculation.

M-Falcon Optimization
Since we use causal attention, the representation of the user sequence is always independent of the condition tokens. So within a batch, we append all conditions for a user to a single user sequence and ensure the same user sequence is not redundantly computed in the batch. We use a custom block attention mask to ensure that condition tokens cannot attend to each other, while still allowing them to fully attend to the underlying user sequence. This drastically cuts the effective batch size moving through the transformer layers, lowering the GPU memory footprint for both ID embeddings and activations.

Attention mask

Scaling CLR Serving Infrastructure

During initial CLR work, we treated each condition as an additional user-level feature — resulting in one model request per condition. As we began scaling to more and more conditions, the cost on our compute clusters was growing rapidly, and we achieved great cost savings and scale unlocking with the following methods:

Single Model Request
Noting that each model request contains duplicate features, we optimized CLR serving by consolidating multiple separate CLR model requests per backend request into a single one with deduplicated request-level features. To achieve this, we modified the model’s TorchScript to batch these features internally and used Torch Jit to parallelize forward passes, shifting the bottleneck from network to CPU while keeping latency neutral.

GPU Serving + NVEmbed
We optimized our logic further by shifting to a paradigm of CLR being a ranker of conditions, leveraging improved internal compute capabilities to form batches; where conditions are treated as items and user level features get broadcasted across these items. This allowed us to significantly simplify our models logic, align CLR inference with the “1 query — N doc” paradigm, mirroring our L2 ranker’s structure. We exported a CUDA-compatible model, deployed on g6e.4xlarge machines, and refactored the serving path — deprecating earlier model-side batch formation logic and improving output parsing.

Furthermore, to help serve and experiment with multiple large ID embedding tables post Foundation Model launch, our team has adopted NVEmbed; a framework for unifying embedding tables and dense models in a single servable torchscript artifact.

Key Wins & Impact:

• Financial & Performance: 7 figure cost savings and reduced p90 model latency by 85% (80ms to 12ms)
• Scalability: Unblocked major initiatives, including CLR FM and Search CLR, while enabling more conditions per user.1
• Dev Velocity: The simplified infrastructure reduces technical debt and lowers the barrier for scaling future conditions.

Future work

Looking ahead, we plan to continue scaling CLR beyond its current home feed use cases. CLR is designed to support multiple condition types through a shared model architecture; it can also serve as a foundation for retrieval across multiple surfaces. Expanding to new surfaces will also enable training on broader and more diverse training data. As we scale the training data, condition coverage, and model capacity, CLR can become a more general retrieval model for personalized learned retrieval across Pinterest.

Conclusion

Conditional Learned Retrieval started as a way to add explicit retrieval context to Pinterest’s two-tower candidate generation system. By conditioning the user tower on signals, CLR helps home feed retrieve candidates that better reflect different aspects of a user’s intent.

Scaling CLR required us to make condition retrieval both more expressive and more efficient. On the modeling side, we improved how the system understands user behavior and represents different retrieval contexts. On the infrastructure side, we made it practical to evaluate many conditions per request to production scale. Together, these changes helped CLR grow from an early interest condition model to a broader retrieval framework for home feed, with a path toward more surfaces, condition types in the future.

Acknowledgements

Matthew Lawhon, Zili Li, Matt Chun, James Li, Dylan Wang, Bowen Deng, Tao Mo, Nezanin Farahpour

Scaling Conditional Learned Retrieval for Pinterest Home Feed was originally published in Pinterest Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.