AI Engineering Glossary

Precise, practical definitions for the systems you actually build.

Adil Shamim

65 min read

·

Sep 11, 2026

--

Listen

Share

Press enter or click to view image in full size

Each entry follows the same structure: what the term means, why it actually matters, how it shows up in real systems, and what people commonly get wrong about it. Read it top to bottom once, then use it as a lookup. Organized into twelve learning areas: Math & Training, Models & Inference, Data & Representations, Retrieval & Generation, Prompting & Context, Agents & Tools, Evaluation & Safety, AI-Native Development, Infrastructure & Serving, Reliability & Operations, Security & Governance, and Multimodal Systems.

Table of Contents

1. Math & Training
2. Models & Inference
3. Data & Representations
4. Retrieval & Generation
5. Prompting & Context
6. Agents & Tools
7. Evaluation & Safety
8. AI-Native Development
9. Infrastructure & Serving
10. Reliability & Operations
11. Security & Governance
12. Multimodal Systems

1. Math & Training

Gradient Descent

Definition: An optimization method that repeatedly adjusts a model’s parameters in the direction that reduces a loss function, using small steps sized by the learning rate. Why it matters: It’s the mechanism underneath essentially every trained model — nothing “learns” without some version of this loop running millions of times. In practice: You rarely implement it by hand; a framework like PyTorch runs it via an optimizer object, and your job is choosing the learning rate, schedule, and batch size around it. Common confusion: Gradient descent isn’t one algorithm — Adam, SGD, and RMSProp are all different ways of using the gradient to decide the actual step. Related terms: Gradient, Backpropagation, Learning Rate, Optimizer

Backpropagation

Definition: The algorithm that computes how much each parameter contributed to the final error, by applying the chain rule backward through the network’s computation graph. Why it matters: It’s what makes training deep networks computationally feasible — without it, computing gradients for millions of parameters individually would be intractable. In practice: Every modern framework handles this automatically through autograd; you write the forward pass, and backpropagation is derived for you. Common confusion: Backpropagation computes gradients — it doesn’t decide how to use them. That’s the optimizer’s job. Related terms: Autograd, Gradient, Optimizer, Gradient Descent

Loss Function

Definition: The single scalar value a model’s training process is trying to minimize, computed by comparing the model’s prediction against the actual target. Why it matters: Choosing the wrong loss function trains a model to optimize the wrong thing — it can score well on its loss and still fail the actual task. In practice: Regression tasks typically use mean squared error; classification tasks typically use cross-entropy; custom tasks often need a custom or combined loss. Common confusion: A falling loss curve doesn’t guarantee a useful model — it only guarantees the model is getting better at whatever the loss measures. Related terms: Cross-Entropy, Gradient, Optimizer, Overfitting

Cross-Entropy

Definition: A loss function based on the negative log-probability a model assigns to the correct class, heavily penalizing confident wrong predictions. Why it matters: It’s the standard loss for next-token prediction, which means it’s effectively the objective behind almost every language model’s pretraining. In practice: Lower cross-entropy loss during training generally correlates with better next-token prediction, though it doesn’t directly measure downstream task quality. Common confusion: Perplexity is derived from cross-entropy, but the two aren’t interchangeable unless the averaging and log base are handled consistently. Related terms: Loss Function, Softmax, Perplexity

Gradient

Definition: A vector of partial derivatives showing how a small change in each parameter would affect the loss, pointing in the direction of steepest increase. Why it matters: It’s the raw signal every training update is built from — no gradient, no learning. In practice: Gradients get computed per batch, then transformed, clipped, or adapted by an optimizer before actually being applied to the weights. Common confusion: A model doesn’t necessarily move in the exact negative-gradient direction — modern optimizers modify that direction substantially. Related terms: Gradient Descent, Backpropagation, Optimizer

Learning Rate

Definition: A hyperparameter that scales how large each parameter update is during training. Why it matters: It’s often the single most consequential setting in a training run — too high and training diverges into noise, too low and it barely progresses. In practice: Most training runs use a schedule — often a warmup period followed by decay — rather than a single fixed value throughout. Common confusion: There’s no universal “correct” learning rate; it depends on batch size, model size, and optimizer, and usually needs tuning per setup. Related terms: Gradient Descent, Batch Size, Hyperparameter

Batch Size

Definition: The number of training examples averaged together into a single gradient update. Why it matters: Larger batches produce smoother gradient estimates and better hardware utilization, but demand more memory and often require adjusting other hyperparameters. In practice: Teams frequently use gradient accumulation to simulate a larger effective batch size than fits in memory at once. Common confusion: Doubling batch size doesn’t mean you should automatically double the learning rate — the right scaling rule depends on the optimizer and setup. Related terms: Gradient Accumulation, Learning Rate, Gradient Descent

Epoch

Definition: One complete pass through the entire training dataset. Why it matters: It’s a common unit for tracking training progress, though its precise meaning varies with data sampling and distributed training setups. In practice: Training runs typically monitor validation performance across epochs and stop early once it stops improving, rather than fixing an epoch count in advance. Common confusion: More epochs isn’t automatically better — beyond a point, additional epochs tend to increase overfitting rather than genuine skill. Related terms: Overfitting, Batch Size, Eval Set

Overfitting

Definition: A state where a model performs well on training data but poorly on unseen data, because it has learned the specific examples rather than the underlying pattern. Why it matters: It’s the single most common reason a model looks great in development and disappoints in production. In practice: You detect it by watching training and validation loss diverge — training loss keeps dropping while validation loss flattens or rises. Common confusion: A model can overfit even with a huge dataset if the task is narrow enough or training runs long enough — dataset size reduces the risk, it doesn’t eliminate it. Related terms: Regularization, Dropout, Dataset Split, Eval Set

Regularization

Definition: Any technique that deliberately constrains a model during training to reduce its tendency to overfit. Why it matters: Left unconstrained, a sufficiently large model can simply memorize training data rather than learning generalizable patterns. In practice: Dropout, weight decay, data augmentation, and early stopping are all regularization techniques applied for different reasons and at different stages. Common confusion: Regularization typically makes training loss look slightly worse — that’s expected, and the point is better validation and real-world performance, not a lower training number. Related terms: Dropout, Weight Decay, Overfitting

Dropout

Definition: A regularization technique that randomly zeroes out a fraction of activations during training, forcing the network not to rely too heavily on any single path. Why it matters: It’s one of the simplest, most effective ways to reduce overfitting in large networks. In practice: Dropout is active only during training and gets switched off for inference, so the network runs at full capacity when actually being used. Common confusion: Monte Carlo dropout is a deliberate exception — it keeps dropout active at inference time specifically to estimate a model’s uncertainty. Related terms: Regularization, Overfitting, Activation Function

Weight Decay

Definition: A regularization technique that adds a penalty proportional to the size of the model’s weights, discouraging unnecessarily large parameter values. Why it matters: It tends to produce models that generalize better by discouraging weights from growing to fit noise in the training data. In practice: AdamW applies weight decay in a way that’s decoupled from the gradient update, which behaves more predictably than folding it into the loss directly. Common confusion: Weight decay and L2 regularization are mathematically related but not identical once you’re using an adaptive optimizer like Adam. Related terms: Regularization, AdamW, Optimizer

Adam / AdamW

Definition: Adaptive optimizers that maintain a running estimate of each parameter’s gradient and squared gradient, then use those estimates to scale each parameter’s update individually. AdamW decouples weight decay from that gradient-based update. Why it matters: Adam is the default starting point for training most modern deep learning models because it tends to converge reliably with minimal manual tuning. In practice: AdamW has largely replaced plain Adam for training large models because of its more predictable weight decay behavior. Common confusion: Adam being a strong default doesn’t make it the best optimizer for every task — some setups genuinely benefit from SGD with momentum or other alternatives. Related terms: Optimizer, Weight Decay, Gradient Descent, Learning Rate

Gradient Clipping

Definition: A technique that limits the magnitude of gradients before they’re applied to the model, capping either individual values or the overall gradient norm. Why it matters: It prevents a single unusually large gradient — from a bad batch or numerical instability — from destabilizing an entire training run. In practice: It’s especially common in training recurrent models and large transformers, where gradient explosions are a known risk. Common confusion: Gradient clipping treats a symptom, not a cause — if clipping triggers constantly, that usually points to a deeper problem with the data, loss, or learning rate. Related terms: Gradient, Gradient Descent, Learning Rate

Mixed Precision

Definition: A training technique that performs most computation using lower-precision numbers (like 16-bit floats) while keeping a few precision-sensitive operations at higher precision. Why it matters: It roughly doubles training speed and halves memory use on modern accelerators, with minimal impact on final model quality. In practice: Frameworks like PyTorch handle this largely automatically through built-in mixed-precision training utilities. Common confusion: Mixed precision is a training-speed technique, not the same thing as quantization, which shrinks an already-trained model’s weights. Related terms: Quantization, Tensor, CUDA

Fine-Tuning

Definition: Continuing to train an already-pretrained model on a narrower dataset or objective, adapting its behavior without starting from scratch. Why it matters: It’s the standard way to specialize a general-purpose model for a specific tone, format, domain, or task. In practice: Teams fine-tune for consistent output style or structured formats far more often than for injecting new factual knowledge. Common confusion: Fine-tuning is not a substitute for retrieval when facts need to stay current or traceable — it shapes behavior, it doesn’t append a searchable knowledge base. Related terms: LoRA, RLHF, RAG

LoRA (Low-Rank Adaptation)

Definition: A fine-tuning method that freezes a pretrained model’s original weights and trains a small set of additional low-rank matrices instead. Why it matters: It delivers most of the benefit of full fine-tuning at a fraction of the memory and compute cost, making customization accessible without massive infrastructure. In practice: LoRA adapters can be swapped in and out at inference time, letting one base model serve multiple specialized behaviors. Common confusion: LoRA doesn’t change the base model’s weights at all — it’s an additive layer, which is exactly why it’s so cheap to train and store. Related terms: Fine-Tuning, Quantization, QLoRA

RLHF (Reinforcement Learning from Human Feedback)

Definition: A training stage that uses human preference data — comparisons between candidate responses — to train a reward signal, which then shapes a model’s behavior through reinforcement learning. Why it matters: It’s the main technique behind turning a raw, next-token-predicting base model into an assistant that reliably follows instructions and produces preferred responses. In practice: RLHF typically follows an initial supervised fine-tuning stage, refining behavior further based on comparative human judgments. Common confusion: RLHF shapes preferred behavior — it doesn’t guarantee factual accuracy or eliminate the need for evaluation and safety testing. Related terms: DPO, SFT, Alignment

DPO (Direct Preference Optimization)

Definition: A preference-training method that optimizes a model directly on pairs of preferred and rejected responses, without training a separate reward model or running a full reinforcement learning loop. Why it matters: It achieves similar goals to RLHF with a simpler, often more stable training setup. In practice: DPO has become a common lightweight alternative to full RLHF pipelines, especially for teams without infrastructure for reward-model training. Common confusion: DPO doesn’t eliminate the need for good preference data — its results are only as good as the comparisons it’s trained on. Related terms: RLHF, SFT, Alignment

Knowledge Distillation

Definition: Training a smaller “student” model to reproduce the behavior or output distribution of a larger, more capable “teacher” model. Why it matters: It’s a practical way to get much of a large model’s capability into something cheaper and faster to actually run in production. In practice: Distillation often combines the teacher’s soft output probabilities with the original hard labels during student training. Common confusion: A distilled model doesn’t inherit every property of its teacher — only the behavior actually reflected in the training signal transfers. Related terms: Fine-Tuning, Quantization, Loss Function

Quantization

Definition: Reducing the numerical precision of a trained model’s weights — from 32-bit or 16-bit down to 8-bit or even 4-bit — to shrink its memory footprint and increase inference speed. Why it matters: It’s often the difference between a model that requires enterprise-grade hardware and one that runs on a single consumer GPU. In practice: Quality loss from quantization is usually small but not zero, and it should be measured against your actual eval set rather than assumed. Common confusion: Quantization happens after training and affects inference; mixed precision happens during training and affects the training process itself. Related terms: Mixed Precision, LoRA, Inference

2. Models & Inference

Transformer

Definition: A neural network architecture built around the attention mechanism, allowing every position in a sequence to directly relate to every other position rather than processing step by step. Why it matters: It’s the architecture underneath essentially every major language, vision, and multimodal model in production today. In practice: Variants differ mainly in whether they use an encoder, a decoder, or both — decoder-only transformers now dominate large language models. Common confusion: “Transformer” describes an architecture family, not a specific product — GPT, Claude, and BERT are all transformers with very different training and structure. Related terms: Attention, Self-Attention, Encoder, Decoder

Attention

Definition: A mechanism that computes how much each element in a sequence should influence the representation of another, by comparing query and key vectors and using the result to weight value vectors. Why it matters: It lets a model route information between any two positions in a sequence directly, regardless of distance — the core capability that made transformers so effective. In practice: Real implementations use multiple parallel “heads,” each learning to attend to different kinds of relationships in the data. Common confusion: Attention weights show where the model’s computation drew information from — they’re not a reliable, faithful explanation of the model’s actual reasoning. Related terms: Self-Attention, Cross-Attention, Transformer, KV Cache

Self-Attention

Definition: Attention where the queries, keys, and values all come from the same sequence, letting every token in an input relate to every other token in that same input. Why it matters: It’s what allows a model to resolve context within a single passage — like understanding what a pronoun refers to several words earlier. In practice: Self-attention is typically masked in decoder-only models so a token can only attend to earlier positions, not future ones. Common confusion: Self-attention and cross-attention use the same underlying math — the only difference is whether queries and keys/values come from the same sequence or different ones. Related terms: Attention, Cross-Attention, Transformer

Cross-Attention

Definition: Attention where the query representations come from one sequence while the key and value representations come from a different sequence or modality. Why it matters: It’s the mechanism that lets one stream of information — like text — pull relevant information from a completely different stream, like an image. In practice: Encoder-decoder translation models use cross-attention so the decoder’s output tokens can attend back to the encoded source sentence. Common confusion: Cross-attention isn’t inherently multimodal — it can connect two text sequences just as easily as text and images. Related terms: Attention, Self-Attention, Multimodal Fusion

Encoder

Definition: The half of a model architecture that transforms an input into an internal representation, typically able to see the entire input at once. Why it matters: Encoder representations are the foundation for tasks like classification, embedding generation, and providing context to a decoder. In practice: Models built purely on encoders — like BERT — are used for understanding tasks rather than open-ended text generation. Common confusion: An encoder can still produce useful outputs through task-specific heads even though it’s not designed for autoregressive generation. Related terms: Decoder, Transformer, Embedding

Decoder

Definition: The half of a model architecture that generates output, typically one token at a time, restricted to only seeing what’s already been produced. Why it matters: Decoder-only architectures are the basis of nearly every large language model used for open-ended generation today. In practice: A decoder-only model still uses self-attention internally — it just applies a causal mask so each position can only look backward. Common confusion: “Decoder” describing a model architecture and “decode phase” describing a stage of the serving pipeline are related but distinct concepts. Related terms: Encoder, Transformer, Autoregressive

Autoregressive

Definition: A generation approach where each output token is predicted based on all the tokens that came before it, and the newly generated token is appended before predicting the next one. Why it matters: It explains both why language models can generate coherent, context-aware text and why generation time scales with output length. In practice: This step-by-step dependency is exactly what the KV cache exists to speed up, avoiding redundant recomputation at every step. Common confusion: The unit being predicted is a token, not necessarily a whole word — and the model doesn’t have to always pick the single most likely token at each step. Related terms: KV Cache, Decoding Strategy, Temperature

KV Cache

Definition: Stored key and value tensors from previously processed tokens during generation, reused so the model doesn’t have to recompute attention over the full sequence at every new step. Why it matters: It’s one of the biggest practical reasons modern chat models feel fast rather than painfully slow as conversations get longer. In practice: KV cache memory usage grows with sequence length, batch size, and model size — it’s a major factor in how much concurrent traffic a server can actually handle. Common confusion: The KV cache holds runtime state for one active sequence — it’s a different concept from prompt caching, which reuses computation across separate requests. Related terms: Attention, Autoregressive, Prompt Cache

Inductive Bias

Definition: The built-in structural assumptions an architecture carries before it ever sees training data, which shape what kinds of patterns it can learn efficiently. Why it matters: Choosing an architecture is partly choosing which assumptions about the data you want baked in from the start. In practice: A CNN’s inductive bias toward locality makes it efficient for images; a transformer’s causal masking biases it toward predicting the future from the past. Common confusion: Transformers are sometimes described as having minimal inductive bias, but tokenization, position encoding, and masking all still encode real assumptions. Related terms: Transformer, CNN, Feature

Inference

Definition: Running a trained model to produce an output — a prediction, an embedding, or generated tokens — without updating its parameters. Why it matters: It’s the actual moment of value delivery for any deployed model — everything else exists to make this step accurate, fast, and affordable. In practice: Inference costs and latency, not training costs, dominate the long-term economics of most deployed AI products. Common confusion: Model weights stay fixed during inference, but the surrounding application can still update caches, conversation state, or external memory. Related terms: Decoding Strategy, KV Cache, Model Serving

Decoding Strategy

Definition: The algorithm that converts a model’s raw next-token probability scores into an actual sequence of generated tokens. Why it matters: The same model with the same prompt can produce noticeably different quality, diversity, and consistency purely based on how decoding is configured. In practice: Chat applications typically use some form of sampling with temperature and nucleus (top-p) constraints rather than always picking the single most likely token. Common confusion: Decoding strategy changes how outputs are selected — it has no effect on the model’s trained parameters or knowledge. Related terms: Temperature, Top-k Sampling, Nucleus Sampling, Autoregressive

Temperature

Definition: A parameter that scales the sharpness of a model’s output probability distribution before sampling, controlling how “confident” or “random” generation feels. Why it matters: It’s one of the most direct levers for tuning output behavior — low for consistent, focused answers, higher for varied, creative ones. In practice: Factual and structured-output tasks typically use low temperature; creative writing tasks often benefit from a higher setting. Common confusion: Temperature at zero doesn’t guarantee fully deterministic output across every system — some implementation details can still introduce minor variation. Related terms: Decoding Strategy, Top-k Sampling, Nucleus Sampling

Top-k / Nucleus (Top-p) Sampling

Definition: Sampling constraints that limit which tokens are eligible for selection — top-k restricts to the k most probable tokens; nucleus sampling restricts to the smallest set of tokens whose combined probability crosses a threshold p. Why it matters: Both prevent a model from occasionally sampling a wildly unlikely token, keeping generation coherent while still allowing variation. In practice: Nucleus sampling has become the more common default because it adapts to how “peaked” or “flat” the probability distribution is at each step, unlike top-k’s fixed cutoff. Common confusion: These constraints work alongside temperature, not instead of it — they’re usually applied together. Related terms: Temperature, Decoding Strategy

Logits

Definition: The raw, unnormalized scores a model outputs for each possible next token, before any probability conversion. Why it matters: They’re the direct output of the model’s final layer, and many decoding and calibration techniques operate on them before softmax is applied. In practice: Some APIs expose raw or log-probabilities of the top tokens, which is useful for debugging model confidence or building custom decoding logic. Common confusion: Logits aren’t probabilities themselves — they need to pass through softmax to become a valid probability distribution. Related terms: Softmax, Decoding Strategy, Calibration

Softmax

Definition: A function that converts a vector of raw scores into a probability distribution, ensuring all values are positive and sum to one. Why it matters: It’s the final step that turns a model’s internal scores into something that can actually be sampled from as next-token probabilities. In practice: Softmax is applied after logits and before any temperature or sampling constraint takes effect. Common confusion: Softmax doesn’t add any new information — it’s a normalization step, not a learned transformation. Related terms: Logits, Cross-Entropy, Decoding Strategy

Mixture of Experts (MoE)

Definition: An architecture where a model contains multiple specialized subnetworks (“experts”), and a learned router selects only a subset of them to process each token. Why it matters: It lets a model have a very large total parameter count while keeping the actual compute cost per token much lower than a dense model of the same size. In practice: MoE models require careful load balancing so tokens aren’t disproportionately routed to a small number of overloaded experts. Common confusion: MoE isn’t a way to combine several unrelated models — it’s one model with internal routing, trained end-to-end as a single system. Related terms: Expert Parallelism, Transformer, Tensor Parallelism

CUDA

Definition: NVIDIA’s platform and programming model for running general-purpose parallel computation on its GPUs. Why it matters: Nearly every major deep learning framework relies on CUDA under the hood to execute the massive amounts of matrix math training and inference require. In practice: CUDA version compatibility between your framework, drivers, and hardware is a common, recurring source of environment setup headaches. Common confusion: GPU acceleration isn’t synonymous with CUDA specifically — other hardware and software stacks exist, though CUDA remains the dominant one. Related terms: Tensor, Mixed Precision, Inference

Tensor

Definition: A generalized multi-dimensional array — scalars are 0-dimensional tensors, vectors are 1D, matrices are 2D, and deep learning routinely works with higher dimensions still. Why it matters: It’s the fundamental data structure every deep learning computation is built from, whether that’s a single weight or an entire batch of images. In practice: Understanding a tensor’s shape — its dimensions — is often the fastest way to debug why a model’s forward pass is failing. Common confusion: A tensor isn’t just “a matrix with a fancier name” — matrices are a specific 2D case within the broader, more general tensor concept. Related terms: CUDA, Mixed Precision, Embedding

Parameter

Definition: A single learnable numeric value within a model — one weight or one bias — adjusted during training to reduce the loss. Why it matters: The total parameter count is the most common (if imperfect) shorthand for describing a model’s scale and capacity. In practice: When someone says a model “has 70 billion parameters,” this is literally the count of these individual learnable numbers. Common confusion: A higher parameter count doesn’t automatically mean better performance — training data quality, architecture, and technique all matter just as much. Related terms: Hyperparameter, Tensor, Fine-Tuning

3. Data & Representations

Tokenization

Definition: The process of splitting raw text into the discrete units — tokens — that a model actually processes as input. Why it matters: Everything downstream depends on it — context window limits, cost, and even a model’s ability to handle unusual words all trace back to how tokenization splits text. In practice: Tokenizers are trained separately from the model itself and stay fixed once deployed; changing tokenizers effectively requires retraining. Common confusion: A token isn’t the same as a word — common words are often single tokens while rare words get split into multiple subword pieces. Related terms: Byte Pair Encoding, Vocabulary, Embedding

Byte Pair Encoding (BPE)

Definition: A tokenization algorithm that builds a vocabulary by starting from individual characters and repeatedly merging the most frequent adjacent pair. Why it matters: It balances vocabulary size against the ability to represent rare or unseen words as smaller, known pieces rather than failing entirely. In practice: How a BPE tokenizer segments code, whitespace, and multilingual text can noticeably affect a model’s efficiency and behavior on those inputs. Common confusion: BPE is one specific tokenization family, not a universal description of how every model builds its vocabulary. Related terms: Tokenization, Vocabulary, Token

Vocabulary

Definition: The complete, fixed set of tokens a model was trained to recognize and generate. Why it matters: Any input has to be broken into pieces from exactly this set — anything outside it simply doesn’t exist as far as the model’s concerned. In practice: Vocabulary size is a real design tradeoff — larger vocabularies mean shorter sequences per input but a bigger embedding table to train and store. Common confusion: A larger vocabulary doesn’t automatically make a model smarter — it changes how text is segmented, not how well the model reasons. Related terms: Tokenization, Byte Pair Encoding, Embedding

Embedding

Definition: A learned dense vector representation of a discrete item — a word, sentence, image, or user — positioned in a continuous space so that similar items end up close together. Why it matters: Embeddings are the bridge between raw, symbolic data and the continuous math every neural network actually operates on. In practice: Embedding models are trained (or fine-tuned) for specific tasks — a general text embedding and one tuned for code search behave meaningfully differently. Common confusion: Similarity in one embedding space doesn’t transfer to another — distance only means something relative to the specific model and metric used to compute it. Related terms: Cosine Similarity, Vector Database, Latent Space

Cosine Similarity

Definition: A measure of how similar two vectors are, computed as the normalized dot product between them, capturing direction rather than magnitude. Why it matters: It’s the standard scoring function for comparing embeddings in search, recommendation, and retrieval systems. In practice: Cosine similarity scores range from -1 to 1, but in practice, most real-world embedding comparisons cluster in a narrower, less intuitive band. Common confusion: A high cosine similarity score doesn’t prove factual equivalence — it reflects the embedding model’s learned notion of relatedness, which isn’t infallible. Related terms: Embedding, Semantic Search, Reranker

Feature

Definition: An individual measurable input variable used by a model to make a prediction. Why it matters: In classical machine learning, feature quality often matters more than model choice — a well-engineered feature set can outperform a fancier model on raw data. In practice: Deep learning largely automates feature extraction from raw data, but tabular and structured-data problems still frequently benefit from manual feature engineering. Common confusion: A single stored column can actually represent multiple distinct features once transformed — and a learned representation can contain useful features with no clean human-readable label at all. Related terms: Embedding, Latent Space, Inductive Bias

Latent Space

Definition: The internal, compressed representation space a model organizes its learned concepts in, where geometric proximity tends to reflect conceptual similarity. Why it matters: It’s where a model’s actual “understanding” lives — the structure that lets it generalize to inputs it’s never seen before. In practice: Generative models like diffusion models and VAEs often operate directly in a compressed latent space rather than in raw pixel space, for efficiency. Common confusion: A latent space’s dimensions rarely correspond to clean, individually interpretable human concepts — meaning is usually distributed across many dimensions at once. Related terms: Embedding, VAE, Diffusion Model

Data Provenance

Definition: Traceable information about where a dataset originated, how it was transformed, and how derived artifacts — embeddings, eval cases, fine-tuned checkpoints — relate back to it. Why it matters: Without it, you can’t reliably investigate contamination, honor licensing constraints, or remove data after a source is found to be problematic. In practice: Mature teams version their datasets explicitly and carry that version metadata forward into everything trained or evaluated on them. Common confusion: A source URL alone isn’t provenance — real provenance also covers collection time, licensing, filtering, and every transformation applied afterward. Related terms: Data Deduplication, Dataset Split, Data Leakage

Data Deduplication

Definition: The process of detecting and removing exact or near-duplicate examples within or across datasets. Why it matters: Repetition in training data can distort the learned distribution, increase memorization, and make evaluation results look artificially strong. In practice: Deduplication typically combines exact hashing for identical content with similarity-based methods to catch near-duplicates and paraphrases. Common confusion: Deduplication isn’t the same as general data cleaning — two genuinely distinct, legitimate records can still share substantial text without being duplicates in the problematic sense. Related terms: Data Provenance, Benchmark Contamination, Dataset Split

Dataset Split

Definition: A documented division of data into separate subsets — typically training, validation, and test — used for fitting a model, making development decisions, and final evaluation. Why it matters: Without a clean split, the evidence you use to choose a model can’t also serve as honest proof that it actually generalizes. In practice: Splitting by a real deployment unit — user, time period, or organization — is often far more meaningful than a purely random split. Common confusion: A random split doesn’t guarantee independence — near-duplicate records or correlated entries can still cross the boundary between splits. Related terms: Eval Set, Data Leakage, Distribution Shift

Data Leakage

Definition: The unintended presence of information during training that wouldn’t actually be available at real prediction time, or that crosses a held-out evaluation boundary. Why it matters: It produces evaluation metrics that look great in development and then collapse the moment the system meets genuinely unseen data. In practice: Common sources include normalizing data before splitting, using future information in historical features, and repeatedly tuning prompts against the same held-out set. Common confusion: Leakage isn’t limited to duplicate rows — subtler sources, like global statistics computed across the whole dataset, can leak just as badly. Related terms: Dataset Split, Benchmark Contamination, Distribution Shift

Distribution Shift

Definition: A meaningful difference between the data distribution a system was built or tested on and the distribution it actually encounters after deployment. Why it matters: A model can pass every offline evaluation and still fail once real users, real inputs, or real operating conditions look different from what it was tested on. In practice: Teams monitor for this by tracking performance and input characteristics by segment over time, not just aggregate accuracy. Common confusion: Distribution shift doesn’t require the model itself to have changed — the world around a completely static model can shift and cause the same failure. Related terms: Dataset Split, Eval Set, Overfitting

4. Retrieval & Generation

RAG (Retrieval-Augmented Generation)

Definition: A pattern where relevant documents are retrieved at query time and included in a model’s prompt, so it generates an answer grounded in that retrieved content rather than relying purely on what it memorized during training. Why it matters: It lets a model answer questions using information that’s current, private, or simply too specific to have been part of its training data. In practice: A production RAG system is really a pipeline — chunking, embedding, retrieval, and often reranking — not just “add documents to the prompt.” Common confusion: Adding retrieved documents to a prompt creates the opportunity for a grounded answer — it doesn’t guarantee the model will actually use them correctly. Related terms: Chunking, Vector Database, Grounding, Reranker

Chunking

Definition: Splitting source documents into smaller, individually retrievable pieces before indexing. Why it matters: Chunk size and boundaries directly determine whether retrieval returns enough context to be useful without flooding the prompt with noise. In practice: Preserving natural structure — keeping headings, code blocks, and tables intact — usually produces better retrieval than a purely fixed-length split. Common confusion: There’s no universal ideal chunk size — the right choice depends on document structure, query type, and the embedding model in use, and should be validated on real queries. Related terms: RAG, Vector Database, Reranker

Vector Database

Definition: A database purpose-built to store embeddings and efficiently find the vectors closest to a given query vector, even across millions of entries. Why it matters: It’s the piece of infrastructure that makes semantic search at real scale computationally feasible. In practice: Most vector databases use approximate nearest neighbor indexing under the hood, trading a small amount of recall for dramatically faster search. Common confusion: A vector database on its own doesn’t handle chunking, embedding generation, or reranking — it’s one component in a larger retrieval pipeline. Related terms: Embedding, HNSW, Approximate Nearest Neighbor

Dense Retrieval

Definition: Retrieval that ranks candidates by embedding similarity to a query, rather than by exact keyword overlap. Why it matters: It can surface paraphrases and semantically related content that share few or no exact words with the query — something keyword search fundamentally can’t do. In practice: Dense retrieval typically works best combined with a lexical method like BM25, since each catches different failure cases the other misses. Common confusion: Dense retrieval and reranking are different stages — retrieval searches the whole collection, while reranking rescores an already-narrowed candidate set. Related terms: Embedding, BM25, Hybrid Retrieval, Reranker

BM25

Definition: A lexical ranking algorithm that scores documents based on query term matches, accounting for term rarity, frequency, and document length. Why it matters: It remains a genuinely strong baseline for exact terms, identifiers, and rare vocabulary that dense embeddings frequently blur together. In practice: Many production retrieval systems still run BM25 alongside dense retrieval rather than replacing it outright. Common confusion: BM25 has no real understanding of semantic meaning — its strength is precise lexical matching, not conceptual similarity. Related terms: Dense Retrieval, Hybrid Retrieval, RAG

Hybrid Retrieval

Definition: Retrieval that combines signals from multiple methods — commonly lexical search like BM25 and dense embedding search — merging or reranking the combined results. Why it matters: Exact identifiers and semantic paraphrases behave completely differently, so relying on a single retrieval signal reliably misses relevant results. In practice: A common pattern is retrieving candidates from both methods independently, merging their rankings, and then applying a reranker to the combined set. Common confusion: Hybrid retrieval merges candidate lists from multiple search methods — that’s a distinct step from reranking, which rescores candidates that have already been retrieved. Related terms: BM25, Dense Retrieval, Reranker

Reranker

Definition: A second-stage model that takes an already-retrieved set of candidates and rescores them more carefully than the initial retrieval pass could afford to. Why it matters: First-pass retrieval optimizes for speed across a huge candidate pool; a reranker can afford to be slower and more accurate over a much smaller shortlist. In practice: Rerankers commonly lift the truly best few results into the top positions even when the initial retrieval pass ranked them lower. Common confusion: A reranker doesn’t search the full collection — it only reorders candidates that retrieval already surfaced. Related terms: Dense Retrieval, Hybrid Retrieval, Cosine Similarity

Approximate Nearest Neighbor (ANN)

Definition: A search approach that returns vectors likely to be among the closest matches to a query, without exhaustively comparing against every vector in the collection. Why it matters: Exhaustive search doesn’t scale — ANN is what makes searching millions or billions of vectors practical in real time. In practice: ANN search involves a genuine tradeoff between speed, memory, and recall, and that tradeoff is usually tunable through index parameters. Common confusion: ANN describes a search objective and tradeoff — HNSW is a specific algorithm that implements it, not a synonym for it. Related terms: HNSW, Vector Database, Embedding

HNSW (Hierarchical Navigable Small World)

Definition: An approximate nearest neighbor algorithm that organizes vectors into layered proximity graphs, searching from coarse upper layers down to detailed lower layers. Why it matters: It’s one of the most widely used indexing methods for making high-recall vector search fast at genuinely large scale. In practice: HNSW indexes typically need to be rebuilt when the underlying embedding model changes, since the geometry the index relies on changes too. Common confusion: HNSW is an index structure, not a similarity metric or an embedding model — those are separate pieces of the retrieval stack. Related terms: Approximate Nearest Neighbor, Vector Database, Embedding

Grounding

Definition: Connecting a generated answer back to identifiable evidence, state, or observations that can actually be checked. Why it matters: It gives a generated response something to be verified against, rather than leaving every claim resting purely on the model’s unchecked output. In practice: Requiring citations for factual claims, and then checking whether the cited passage actually supports the claim, is a common and effective grounding check. Common confusion: Having retrieved documents available in the prompt creates an opportunity for grounding — it’s not a guarantee the model will use them faithfully. Related terms: RAG, Hallucination, Verification Gate

Semantic Search

Definition: Search that matches based on meaning rather than exact keyword overlap, typically implemented by comparing embeddings. Why it matters: It surfaces relevant results that traditional keyword search would completely miss due to phrasing differences. In practice: Semantic search quality depends heavily on the embedding model’s training — a model tuned for one domain can perform noticeably worse in another. Common confusion: Semantic search and dense retrieval describe largely the same underlying technique from different angles — one names the goal, the other names the method. Related terms: Embedding, Dense Retrieval, Cosine Similarity

5. Prompting & Context

Prompt Engineering

Definition: The practice of deliberately crafting an input’s wording, structure, and examples to reliably improve a model’s output quality. Why it matters: Small, careful changes in phrasing and structure can move task accuracy substantially without touching the model itself. In practice: Effective prompt engineering usually means testing variations against a real eval set, not just trusting instinct about what “sounds” clearer. Common confusion: Prompt engineering is about instruction wording specifically — it’s a subset of the broader discipline of context engineering, not a replacement for it. Related terms: Context Engineering, Chain of Thought, Few-Shot

Context Window

Definition: The maximum number of tokens a model can process in a single call, encompassing instructions, conversation history, retrieved content, and the generated output together. Why it matters: Everything a model can consider at once is bounded by this limit — it’s a hard constraint on how much information a single request can actually use. In practice: Conversation history is only available to a model if the application explicitly resends or reconstructs it — the model has no memory between separate calls on its own. Common confusion: A larger context window doesn’t guarantee every included detail gets used reliably — more tokens available isn’t the same as more attention paid to each one. Related terms: Token Budget, Context Engineering, Prompt Cache

Context Engineering

Definition: The broader discipline of designing everything supplied to a model at each step — instructions, retrieved evidence, tool results, examples, and prior state — not just the instruction wording itself. Why it matters: Model failures are often failures of missing, stale, or poorly ordered context rather than failures of the model’s underlying capability. In practice: A well-engineered context packet includes the task’s goal, relevant constraints, and the most recent relevant state — updated as that state changes. Common confusion: Context engineering decides what enters the model’s working context; prompt engineering focuses more narrowly on how the instructions themselves are worded. Related terms: Prompt Engineering, Context Window, Agent State

Token Budget

Definition: How the limited space in a context window is actually allocated — across instructions, conversation history, retrieved content, and room for the output. Why it matters: Careless allocation means important content gets crowded out by less useful content, degrading quality without any obvious error message. In practice: Systems working with long documents or long conversations often need explicit strategies — summarization, truncation, or compression — to stay within budget. Common confusion: A generous token budget doesn’t solve poor context prioritization — what you include still matters more than how much room you have. Related terms: Context Window, Context Compression, Context Engineering

Context Compression

Definition: Reducing the token footprint of source material while trying to preserve the information a model actually needs for the task at hand. Why it matters: It lets long tasks fit within a limited context window, but every piece of information left out carries real risk of losing something the model needed. In practice: Effective compression tends to preserve specific facts and identifiers verbatim while summarizing more redundant surrounding text. Common confusion: Compression is inherently lossy unless it retains the complete original — a shorter summary isn’t automatically an equivalent substitute for the full context. Related terms: Token Budget, Context Engineering, Context Window

System Prompt

Definition: Instructions configured in advance, typically invisible to the end user, that establish a model’s role, tone, and behavioral constraints before a conversation begins. Why it matters: It’s usually the highest-authority instruction layer in an application, setting the baseline behavior everything else operates within. In practice: Well-designed system prompts are specific about role, constraints, and edge cases rather than vague aspirational statements. Common confusion: A system prompt sets intended behavior — it isn’t a hard security boundary on its own, and shouldn’t be relied on as the only defense against prompt injection. Related terms: Instruction Hierarchy, Prompt Injection, Context Engineering

Chain of Thought (CoT)

Definition: Prompting a model to work through intermediate reasoning steps before producing a final answer. Why it matters: Decomposing a problem into steps genuinely helps on multi-step tasks that a single direct answer tends to get wrong. In practice: Asking for a concise plan, then independently verifying the result, tends to work better than relying purely on a long, unchecked reasoning transcript. Common confusion: A fluent, plausible-sounding chain of reasoning is not proof the final answer is correct — it’s not a substitute for actual verification. Related terms: Prompt Engineering, Verification Gate, In-Context Learning

Few-Shot Prompting

Definition: Including a small number of example input-output pairs directly in a prompt so the model can infer the intended task, format, or pattern. Why it matters: It lets a general-purpose model adapt to a specific format or task without any actual training or fine-tuning. In practice: Example quality and diversity matter far more than sheer example count — a handful of well-chosen examples beats a dozen redundant ones. Common confusion: Poor or contradictory examples can actively hurt reliability rather than help it — few-shot prompting isn’t automatically an improvement over zero-shot. Related terms: Zero-Shot, In-Context Learning, Context Window

Zero-Shot Prompting

Definition: Asking a model to perform a task using only an instruction, with no examples included in the prompt. Why it matters: Modern large models handle a striking range of tasks zero-shot, which significantly simplifies prompt design compared to earlier generations of models. In practice: Clear, specific instructions matter even more in zero-shot prompting, since there’s no example to implicitly clarify ambiguity. Common confusion: Strong zero-shot performance on one task doesn’t guarantee equally strong zero-shot performance on a different, unrelated task. Related terms: Few-Shot, In-Context Learning, Prompt Engineering

In-Context Learning

Definition: A model’s ability to adapt its behavior based on instructions or examples present in the current input, without any actual update to its trained parameters. Why it matters: It’s the underlying capability that makes both zero-shot and few-shot prompting work at all — one pretrained model handling entirely new tasks on the fly. In practice: This adaptation only lasts for the current context — nothing about it persists once that conversation or request ends. Common confusion: In-context learning is temporary conditioning, not memory, fine-tuning, or genuine proof the model has inferred the intended rule correctly. Related terms: Few-Shot, Zero-Shot, Context Window

Instruction Following

Definition: How reliably a model’s output actually satisfies the specific constraints in an instruction — format, length, tone, and boundaries — rather than just producing fluent, plausible text. Why it matters: Fluent output that ignores half the stated requirements is a real, common failure mode distinct from simply being wrong. In practice: Testing instruction following explicitly — with conflicting constraints or strict format requirements — catches failures that general quality checks tend to miss. Common confusion: Instruction following measures obedience to the stated task, not factual correctness or alignment more broadly — a model can follow instructions perfectly and still be wrong. Related terms: System Prompt, Alignment, Prompt Engineering

Instruction Hierarchy

Definition: The rules that determine which instruction wins when sources with different authority — system prompts, users, and untrusted retrieved content — conflict with each other. Why it matters: Agent systems constantly mix trusted goals with external, sometimes adversarial text, and need a defined way to resolve that conflict. In practice: A common pattern is treating retrieved or tool-provided content strictly as data to reference, never as a source of new instructions to obey. Common confusion: An instruction hierarchy can meaningfully improve behavior, but it isn’t a hard security boundary on its own — permissions and approval gates still matter. Related terms: System Prompt, Prompt Injection, Least Privilege

6. Agents & Tools

Agent

Definition: A system in which a model selects actions toward a goal, observes the results of those actions, and continues under some form of orchestration. Why it matters: Reliability in an agentic system comes overwhelmingly from the surrounding harness, permissions, and verification — not from the model alone. In practice: A simple coding agent might read a repository, propose a change, run tests in a sandbox, and pause for approval before anything reaches production. Common confusion: Full autonomy isn’t a requirement for something to count as an agent — a system with human approval steps built in is still an agent. Related terms: Agent Harness, Agent State, Tool Contract, Human-in-the-Loop

Agent Harness

Definition: The runtime infrastructure surrounding a model that assembles context, exposes tools, tracks state, enforces limits, and decides when the agent should continue, retry, or stop. Why it matters: Two systems built on the identical underlying model can behave very differently depending entirely on the quality of the harness around it. In practice: A harness might cap an agent at a fixed number of tool calls per task and require a passing test before it considers a step complete. Common confusion: A harness is broader than a prompt template but narrower than an entire finished product — it’s specifically the operational scaffolding in between. Related terms: Agent, Tool Contract, Agent State, Sandbox

Agent State

Definition: The explicit, structured data an agent maintains across steps — its current objective, completed actions, open questions, and relevant artifacts. Why it matters: Explicit state is what makes long-running tasks resumable and inspectable, instead of relying on the model reconstructing progress from raw conversation history each time. In practice: A coding agent’s state might track the selected issue, files changed so far, and the latest test result, updated after every action. Common confusion: State is not the same thing as conversation history — a transcript is raw evidence, while state is the compact, operational summary actually used to decide what happens next. Related terms: Checkpoint, Context Engineering, Handoff

Agent Memory

Definition: Information stored outside a model and selectively pulled back into context for future runs — prior decisions, learned preferences, or verified facts. Why it matters: It gives an agent continuity across sessions without forcing every past interaction into every single prompt. In practice: Effective agent memory retrieves only what’s relevant to the current task rather than dumping an entire interaction history in indiscriminately. Common confusion: Agent memory and agent state serve different jobs — state tracks the currently active task, memory preserves information for possible future tasks. Related terms: Agent State, Context Engineering, Semantic Search

Function Calling

Definition: A mechanism where a model outputs a structured request naming a specific tool and its arguments, which application code then validates and actually executes. Why it matters: It’s the foundational bridge that lets a model trigger real-world actions instead of only producing text. In practice: The model only ever requests an action — trusted application code decides whether and how that request actually gets executed. Common confusion: Function calling alone doesn’t make a system an agent — it’s one capability an agent uses, not the entire architecture. Related terms: Tool Contract, Agent, Structured Output

Tool Contract

Definition: The explicit specification of what a tool expects as input and what it guarantees as output. Why it matters: Without a clear contract, neither an agent nor the humans debugging it can reliably predict what a given tool call will actually do. In practice: Well-designed tool contracts include validation, clear error behavior, and documented side effects, not just an input schema. Common confusion: A tool contract describes the interface — it says nothing on its own about what permissions the calling agent actually has to invoke it. Related terms: Function Calling, MCP, Least Privilege

MCP (Model Context Protocol)

Definition: An open standard that defines a consistent interface for connecting AI models to external tools, data sources, and systems. Why it matters: It replaces the need for every application to build its own one-off integration for every tool a model might need to use. In practice: An MCP server exposes a set of tools and resources that any compatible client can discover and call using a shared protocol. Common confusion: MCP standardizes the interface between models and tools — it doesn’t itself grant permissions or provide safety guarantees around what those tools do. Related terms: Tool Contract, Function Calling, Agent Skill

Human-in-the-Loop (HITL)

Definition: A workflow design where a person supplies judgment, correction, or approval at explicitly defined points within an AI-driven process. Why it matters: Human involvement delivers the most value at genuinely high-stakes, ambiguous, or irreversible decision points — not scattered everywhere as an undefined safety net. In practice: A system might let routine, low-stakes cases proceed automatically while routing anything uncertain or high-value to a human reviewer with full context. Common confusion: Including a human reviewer doesn’t automatically make a system safe — that reviewer still needs sufficient time, context, and authority to make a real decision. Related terms: Approval Gate, Agent, Guardrails

Approval Gate

Definition: A control point that blocks a consequential action until an authorized person or policy explicitly grants permission. Why it matters: It limits the potential damage of an uncertain agent decision while still preserving automation for lower-risk, reversible work. In practice: An agent might freely draft and test a database migration in a disposable environment, but require explicit sign-off before running it against production. Common confusion: An approval gate asks whether an action is authorized; a verification gate separately asks whether evidence shows the action was actually correct. Related terms: Human-in-the-Loop, Least Privilege, Verification Gate

Orchestration

Definition: The logic that coordinates multiple steps, tools, or agents toward a shared objective, deciding what runs when and in what sequence. Why it matters: As systems grow beyond a single model call, something has to actually manage the sequencing, retries, and handoffs between the pieces. In practice: Orchestration can be as simple as a fixed sequence of steps or as complex as a dynamic, model-driven decision about what to do next. Common confusion: Orchestration coordinates the overall flow — it’s a distinct concern from the individual tool contracts and permissions governing each step within that flow. Related terms: Agent, Delegation, Durable Execution

Delegation

Definition: Assigning a clearly bounded subtask to another agent or person, along with the context, authority, and expected output needed to actually complete it. Why it matters: It enables real specialization and parallel work without losing ownership of the overall task or the ability to properly integrate the results. In practice: Effective delegation gives the receiving agent the exact files, criteria, and deadline it needs, and requires it to report findings back rather than silently modifying shared work. Common confusion: Sending a vague instruction to another agent isn’t reliable delegation — the receiver genuinely needs a defined scope and a clear path back. Related terms: Scope Contract, Handoff, Orchestration

Handoff

Definition: A structured transfer of a task between people or agents that preserves the objective, current state, evidence, and remaining work. Why it matters: A good handoff means the next worker doesn’t have to reconstruct the entire task from a long, unstructured transcript. In practice: An effective handoff includes the accepted plan, work already completed, and the exact next action — not just a general summary of what happened. Common confusion: A plain summary describes what happened; a handoff also explicitly states what’s authoritative now and what should happen next. Related terms: Agent State, Checkpoint, Delegation

Checkpoint

Definition: A saved snapshot of a workflow’s progress that allows it to resume from a known point rather than restarting from scratch. Why it matters: Long-running agent workflows and training runs can recover from an interruption without losing completed work or expensive progress. In practice: A workflow checkpoint typically stores operational state and artifact references; a training checkpoint stores weights, optimizer state, and training position. Common confusion: A checkpoint is meant to support recovery — it’s not just a raw transcript, and a weights file without resume metadata isn’t a complete checkpoint on its own. Related terms: Agent State, Durable Execution, Handoff

Durable Execution

Definition: Running a workflow so that its state and completed steps survive crashes, restarts, or long waits without redoing work that already succeeded. Why it matters: Agent tasks frequently span multiple model calls, tool invocations, and approvals — a single crashed process shouldn’t be the only record of that progress. In practice: Durable execution typically persists state after each meaningful transition and uses idempotency to make retries of external actions safe. Common confusion: Durable execution doesn’t automatically make every action safe — individual side effects still need idempotency and, where relevant, compensation logic. Related terms: Checkpoint, Idempotency, Agent State

7. Evaluation & Safety

Evaluation (Eval)

Definition: A defined, repeatable process for measuring a model or system’s behavior on representative tasks using explicit success criteria. Why it matters: Without it, “quality” stays a subjective impression from a handful of demos rather than something you can actually track and improve over time. In practice: Mature teams run the same representative test cases before and after every meaningful change, comparing results directly rather than relying on intuition. Common confusion: A single benchmark score is one data point about a system’s performance — it’s not a complete account of how it will behave in production. Related terms: Eval Set, LLM-as-a-Judge, Regression Test

Eval Set

Definition: A versioned collection of test inputs, expected properties, and scoring rules used to consistently measure a system against a defined capability or risk. Why it matters: A repeatable eval set turns vague quality impressions into comparable evidence and catches regressions the moment prompts, models, or retrieval change. In practice: Well-maintained eval sets keep adversarial and edge cases alongside typical ones, and stay separate from whatever examples were used during development. Common confusion: Repeatedly tuning a system directly against the same held-out eval set eventually leaks information from that set and inflates how good the results actually look. Related terms: Evaluation, Regression Test, LLM-as-a-Judge

LLM-as-a-Judge

Definition: Using one language model to evaluate or score another model’s outputs at scale, typically against defined criteria. Why it matters: Human review genuinely doesn’t scale to every iteration of a fast-moving system, and LLM judges can offer a practical, faster substitute. In practice: Judge prompts benefit from the same rigor as any other prompt — clear criteria, examples of good and bad outputs, and periodic spot-checking against human judgment. Common confusion: An LLM judge inherits its own blind spots and biases — it’s a useful scaling tool, not an infallible ground truth. Related terms: Evaluation, Eval Set, Calibration

Hallucination

Definition: Generated output that’s false, unsupported by available evidence, or inconsistent with the task’s actual source of truth — regardless of how confident or fluent it sounds. Why it matters: You generally can’t inspect whether a specific claim actually existed in a model’s training data, so production systems need to focus on checking support and correctness directly. In practice: Requiring citations for factual claims — and then genuinely checking whether the citation supports the claim — is one of the most reliable mitigations available. Common confusion: A hallucination describes an output-quality failure — it’s not a claim about the model’s intent, since the model isn’t deliberately attempting to deceive anyone. Related terms: Grounding, RAG, Verification Gate

Guardrails

Definition: Layered system controls — including schemas, policy checks, classifiers, allowlists, sandboxing, and post-action verification — that constrain a model’s inputs, tool use, and outputs. Why it matters: No single filter catches every failure mode, which is exactly why real guardrail systems stack multiple, independent layers according to actual risk. In practice: Effective guardrails combine input validation, output checking, and permission limits rather than relying on any single layer to do all the work. Common confusion: Guardrails genuinely reduce risk, but they don’t constitute proof that a system is fully safe — they’re mitigation, not a guarantee. Related terms: Least Privilege, Approval Gate, Defense in Depth

Alignment

Definition: The ongoing effort to make a model or AI system behave in ways that match intended goals and human preferences, including in adversarial or unexpected situations. Why it matters: A system can technically optimize its stated metric while still violating what a user actually wanted — alignment work exists specifically to close that gap. In practice: Alignment isn’t purely a training-time concern — evaluation, oversight, and system-level controls all contribute to it as well. Common confusion: Alignment is broader than simply “making AI safe” in the abstract — it’s specifically about behavior matching intended goals, which includes but isn’t limited to safety. Related terms: RLHF, DPO, Guardrails

Calibration

Definition: How closely a system’s stated confidence matches the actual observed frequency of it being correct at that confidence level. Why it matters: A system can be accurate on average and still dangerously overconfident on exactly the cases where people most rely on its stated confidence. In practice: Checking calibration means bucketing predictions by confidence level and comparing each bucket’s stated confidence against its actual accuracy. Common confusion: Calibration measures how trustworthy a confidence score is — it says nothing directly about overall accuracy, factuality, or reasoning quality. Related terms: Softmax, Precision & Recall, Evaluation

Benchmark Contamination

Definition: Overlap or information leakage between evaluation examples and the data used to train, tune, or otherwise improve the system being evaluated. Why it matters: Contamination can make a benchmark score reflect prior exposure to the exact answers rather than genuine ability to generalize. In practice: Detecting it involves tracking dataset provenance and checking training sources for exact and near-duplicate overlap with benchmark content. Common confusion: Contamination is broader than literal exact copying — paraphrased content, leaked answer keys, and repeated prompt tuning against the same set can all leak evaluation information. Related terms: Data Deduplication, Eval Set, Data Leakage

Precision & Recall

Definition: Precision measures how many of the items flagged as positive were actually positive; recall measures how many of the actually positive items were successfully flagged. Why it matters: The two metrics trade off against each other constantly, and which one matters more depends entirely on whether false positives or false negatives are more costly for the task. In practice: A spam filter typically prioritizes precision to avoid burying real email, while a fraud detector often prioritizes recall to avoid missing genuine fraud. Common confusion: A high overall accuracy can hide a serious precision or recall problem, especially on datasets where one class is far rarer than the other. Related terms: Evaluation, Calibration, Exact Match

Exact Match (EM)

Definition: A metric that counts an output as correct only if its normalized form exactly equals an accepted reference answer. Why it matters: It’s deterministic and easy to audit for tasks with one clearly correct answer, though it offers no partial credit. In practice: Exact match is typically paired with other checks on tasks where multiple valid phrasings exist, since it’s blind to harmless formatting differences. Common confusion: A low exact-match score doesn’t necessarily mean the answer was wrong — it might just be phrased differently from the reference in a way that’s still correct. Related terms: Precision & Recall, Eval Set, Structured Output

Red Teaming

Definition: Deliberately attempting to break a system through adversarial prompts, jailbreaks, and edge cases before real attackers or real users find the same gaps. Why it matters: It surfaces failure modes that standard evaluation, built around expected, well-behaved inputs, tends to miss entirely. In practice: Effective red teaming derives its test cases from concrete, defined prohibited behaviors rather than open-ended, unstructured probing. Common confusion: A single round of red teaming isn’t a one-time certification — confirmed failures should become part of an ongoing, versioned adversarial eval set. Related terms: Jailbreak, Guardrails, Eval Set

Jailbreak

Definition: An adversarial input or interaction strategy designed to get a model to produce output that its training or system-level controls are meant to prevent. Why it matters: A successful jailbreak exposes a real gap between stated policy and actual behavior — one that becomes more consequential once a model can also control tools or protected data. In practice: Jailbreak testing typically varies phrasing, format, and interaction length to probe whether a restriction holds up under genuine pressure. Common confusion: A jailbreak specifically targets a model’s behavioral restrictions; prompt injection is a related but distinct attack aimed at hijacking instruction following, though the two can overlap in a single attack. Related terms: Red Teaming, Prompt Injection, Guardrails

Regression Test

Definition: An automated test that catches when previously working behavior silently breaks after a change to a prompt, model, or pipeline. Why it matters: Systems built around models can regress in subtle, non-obvious ways that a casual manual check will often miss. In practice: Regression tests are most effective when they run automatically on every meaningful change, not just occasionally when someone remembers to check. Common confusion: A regression test verifies that something that used to work still works — it’s not the same as broader evaluation aimed at measuring overall capability. Related terms: Eval Set, Evaluation, Flaky Test

8. AI-Native Development

Coding Agent

Definition: An agent specialized for software engineering work — capable of reading a repository, editing files, running development tools, and using the results to keep progressing on a task. Why it matters: Its practical value depends heavily on repository context, tool permissions, and verification — not purely on raw code generation quality. In practice: A well-scoped coding agent might work from a defined issue, propose a patch, run the test suite, and pause for human review before anything merges. Common confusion: A tool that only suggests code snippets isn’t necessarily an agent — the defining feature is actually acting through tools and observing the results. Related terms: Agent Harness, Scope Contract, Verification Gate

Idempotency

Definition: The property that repeating an operation with the same identity doesn’t produce additional side effects beyond the first successful execution. Why it matters: Retries are a routine, expected part of distributed agent systems — without idempotency, one uncertain response can duplicate a payment, a comment, or a deployment. In practice: A common implementation attaches an idempotency key to a request and persists the completed result, so a retry returns that stored result instead of re-executing the action. Common confusion: Idempotency doesn’t require every response to be byte-for-byte identical on retry — it requires that the resulting state change isn’t duplicated. Related terms: Retry with Backoff, Durable Execution, Checkpoint

Circuit Breaker

Definition: A reliability control that temporarily stops calls to a struggling dependency once failures cross a defined threshold, then periodically probes whether it has recovered. Why it matters: It prevents repeated failures in one dependency from consuming latency and capacity across the rest of a system. In practice: A circuit breaker typically opens after a run of consecutive failures, fails fast or falls back during that open period, then allows a limited health probe after a cooldown. Common confusion: A circuit breaker reacts to a dependency’s actual health; a rate limit instead controls the allowed volume of requests, regardless of how healthy the dependency currently is. Related terms: Retry with Backoff, Backpressure, Rate Limit

Backpressure

Definition: A flow-control mechanism that slows or rejects new incoming work when a downstream component can’t safely keep up with the current rate. Why it matters: Without it, queued work can silently grow until it exhausts memory, blows through rate limits, and drags the whole system down. In practice: A system might pause accepting new agent jobs, or return an explicit retryable response, once a downstream queue hits its defined capacity. Common confusion: Backpressure protects capacity proactively, before failure — a circuit breaker instead reacts after failures have already shown a dependency is unhealthy. Related terms: Circuit Breaker, Rate Limit, Retry with Backoff

Retry with Backoff

Definition: Retrying a failed operation, but with a progressively increasing delay between each attempt. Why it matters: It gives a struggling system genuine room to recover, instead of a flood of immediate retries making an already-degraded situation worse. In practice: Backoff is frequently combined with jitter — small random variation in delay — to prevent many clients from retrying in synchronized, load-spiking waves. Common confusion: Retries have to be paired with idempotency to be safe — otherwise a retried action can duplicate a side effect that already actually succeeded. Related terms: Idempotency, Circuit Breaker, Deadline Propagation

Flaky Test

Definition: A test that passes and fails inconsistently across equivalent runs, with no actual relevant change in the underlying code. Why it matters: Flaky tests quietly erode trust in verification — people and agents alike learn to ignore failures, or simply retry until they get a passing result. In practice: The right fix is identifying and eliminating the actual source of nondeterminism — uncontrolled timing, concurrency, or shared state — not just re-running the test until it passes. Common confusion: A test that consistently catches a genuinely intermittent product bug is valuable evidence of a real issue, not a flaky test that should be dismissed. Related terms: Regression Test, Retry with Backoff, Verification Gate

Cost per Successful Task

Definition: Total system cost divided by the number of tasks that actually satisfy a defined success criterion, including retries and failed attempts. Why it matters: A cheap-looking individual model call can still produce an expensive overall workflow if it fails often enough to require repeated correction. In practice: Measuring this means tracking cost across many real tasks end-to-end and dividing by how many actually passed evaluation — not just averaging per-token pricing. Common confusion: Cost per token measures raw usage; cost per successful task measures actual useful outcomes, and the two numbers can tell very different stories. Related terms: Evaluation, Retry with Backoff, Verification Gate

Scope Contract

Definition: An explicit boundary defining exactly what an agent is authorized to access, change, or decide for a specific task. Why it matters: It’s what turns “let the agent handle it” from an open-ended risk into a genuinely controlled delegation with clear limits. In practice: A scope contract typically names the files or systems in bounds, the actions allowed, and the conditions under which the agent must stop and escalate. Common confusion: A scope contract defines what an agent may do — it’s distinct from the tool contract, which defines how a specific tool actually behaves once called. Related terms: Delegation, Least Privilege, Agent Harness

9. Infrastructure & Serving

Model Serving

Definition: The infrastructure layer that exposes a trained model so applications can actually call it, handling batching, scaling, and request routing. Why it matters: It’s the difference between a model that works in a notebook and one that reliably answers real traffic at real scale. In practice: Serving systems have to balance throughput, latency, and cost simultaneously — optimizing purely for one tends to hurt the other two. Common confusion: Model serving is an infrastructure concern distinct from model quality — a great model can still deliver a poor experience if served badly. Related terms: Autoscaling, Continuous Batching, Goodput

Prefill

Definition: The first stage of generating a response, where a model processes the entire input prompt in one parallel computation pass. Why it matters: Prefill cost scales with prompt length, and for very long prompts it can dominate the time before any output even begins streaming. In practice: Systems handling long prompts alongside many concurrent requests often need dedicated scheduling strategies specifically for prefill work. Common confusion: Prefill and decode stress hardware in genuinely different ways — a serving system tuned well for one can still perform poorly on the other. Related terms: Decode Phase, Time to First Token, Chunked Prefill

Decode Phase

Definition: The stage after prefill where a model generates new tokens one at a time, each depending on everything generated before it. Why it matters: Decode is inherently sequential, which is exactly why longer outputs take proportionally longer to produce than shorter ones. In practice: Decode latency and throughput are typically measured separately from prefill, since they behave so differently under load. Common confusion: “Decode phase” names a stage in the generation pipeline — it’s related to, but distinct from, “decoder” as an architectural term. Related terms: Prefill, KV Cache, Time per Output Token

Continuous Batching

Definition: A serving technique that adds and removes individual requests from an active batch at each generation step, rather than waiting for every request in a fixed batch to finish together. Why it matters: Since different requests naturally produce different output lengths, this keeps hardware utilized without forcing short requests to wait on the longest one in their batch. In practice: It’s become close to a standard feature in modern high-throughput LLM serving frameworks. Common confusion: Continuous batching is a serving-time scheduling technique — it has nothing to do with gradient accumulation or training batch size. Related terms: Dynamic Batching, Decode Phase, Goodput

Dynamic Batching

Definition: Grouping incoming requests into batches based on compatible shapes, a maximum batch size, and an allowed wait time. Why it matters: Batching multiple requests together substantially improves hardware utilization, but waiting to form a batch can hurt latency when traffic is sparse. In practice: Tuning the max wait time is a direct throughput-versus-latency tradeoff that needs to match actual traffic patterns. Common confusion: Dynamic batching assembles a batch from queued requests up front; continuous batching changes a batch’s membership while generation is already actively running. Related terms: Continuous Batching, Admission Control, Model Serving

FlashAttention

Definition: An attention algorithm that computes the exact same mathematical result as standard attention, but restructures the computation to minimize slow memory transfers. Why it matters: Attention is frequently limited by memory movement rather than raw arithmetic, especially for long sequences — an IO-aware algorithm meaningfully improves real speed and memory efficiency. In practice: It’s supported by most modern serving frameworks and typically enabled by default when the hardware and configuration support it. Common confusion: FlashAttention changes how attention is computed, not the actual attention result it targets — it’s an exact algorithm, not an approximation. Related terms: Attention, KV Cache, Mixed Precision

Autoscaling

Definition: Automatically adjusting the number or capacity of serving workers based on observed demand, resource use, or application-level metrics. Why it matters: AI workloads can shift demand faster than manual provisioning can react, but scaling decisions also have to account for real model load times and accelerator availability. In practice: Effective autoscaling maintains a minimum warm capacity so the very first requests during a scale-up event don’t hit an unacceptably slow cold start. Common confusion: Autoscaling adds or removes capacity — it doesn’t make an already-overloaded downstream dependency any faster on its own. Related terms: Model Serving, Saturation, Load Shedding

Chunked Prefill

Definition: Dividing a long prompt’s prefill work into smaller, schedulable pieces so it can interleave with decode steps from other active requests. Why it matters: A single very long prompt can otherwise monopolize the accelerator and stall every other in-flight generation, hurting tail latency even when total throughput still looks healthy. In practice: Chunk size is typically tuned against real measured workloads, balancing prefill completion time against decode latency for concurrent requests. Common confusion: Chunked prefill changes how prompt computation gets scheduled — it doesn’t split a user’s actual context into independent semantic pieces. Related terms: Prefill, Decode Phase, Continuous Batching

Disaggregated Serving

Definition: A serving architecture that runs prefill and decode work on separately provisioned worker pools, transferring the necessary state between them. Why it matters: Since prefill and decode have very different hardware demands, independent pools let each be sized and scheduled for its own actual bottleneck. In practice: This design requires efficiently transferring KV-cache state between the prefill and decode pools, which introduces its own real cost worth measuring. Common confusion: Disaggregation separates runtime stages of serving — it’s a different concept from tensor or pipeline parallelism, which split the model itself across devices. Related terms: Prefill, Decode Phase, Tensor Parallelism

Tensor Parallelism

Definition: Splitting the matrix operations inside a single model layer across multiple devices, so one layer’s computation runs in parallel. Why it matters: It’s essential for running models too large to fit — or run efficiently — on a single accelerator. In practice: Tensor parallelism requires fast interconnects between devices, since the split computation has to communicate intermediate results constantly. Common confusion: Tensor parallelism splits operations within a layer, while pipeline parallelism splits entire layers across devices in sequence — the two are complementary, not interchangeable. Related terms: Pipeline Parallelism, Expert Parallelism, Mixture of Experts

Pipeline Parallelism

Definition: Splitting a model’s layers across multiple devices in sequence, so different stages of the forward pass execute on different hardware simultaneously. Why it matters: It’s another common way to fit and run very large models across multiple devices, often combined with tensor parallelism rather than used alone. In practice: Pipeline parallelism introduces “bubbles” — idle time while later stages wait on earlier ones — which scheduling techniques try to minimize. Common confusion: Pipeline parallelism splits a model’s layers in sequence across devices, distinct from tensor parallelism, which splits computation within a single layer. Related terms: Tensor Parallelism, Expert Parallelism, Model Serving

Goodput

Definition: The rate of completed requests that actually satisfy defined service constraints — like both time-to-first-token and per-token latency targets — under a stated workload. Why it matters: Raw throughput can rise while the actual user experience gets worse, since goodput specifically counts only the work that meets the real service contract. In practice: Reporting goodput requires clearly declaring the latency thresholds being measured against, since it depends entirely on that definition. Common confusion: Goodput isn’t the same as total completed throughput, and it isn’t a fixed, universal property of a given model — it depends heavily on workload and the success thresholds chosen. Related terms: Time to First Token, Time per Output Token, Model Serving

Time to First Token (TTFT)

Definition: How long a user waits after sending a request before the very first output token arrives. Why it matters: It’s often the single biggest factor in whether an interface feels instant or sluggish, independent of how fast the rest of the response streams. In practice: TTFT is heavily influenced by prefill time, which scales with prompt length — long prompts can noticeably hurt this metric even with a fast model. Common confusion: A fast TTFT doesn’t guarantee a fast overall response — the pacing of the remaining tokens is a separate metric entirely. Related terms: Prefill, Time per Output Token, Goodput

Time per Output Token (TPOT)

Definition: The average time between generated tokens once decoding is actually underway. Why it matters: It determines whether a streaming response feels like it’s flowing smoothly or stuttering noticeably as it’s read. In practice: TPOT is affected by batching, concurrent load, and KV-cache pressure — it’s rarely a fixed, constant number in a real production system. Common confusion: TPOT is an average across a request; a single gap between two consecutive tokens is a distinct, more granular measurement. Related terms: Decode Phase, Time to First Token, Continuous Batching

10. Reliability & Operations

Service Level Indicator (SLI)

Definition: The actual, concretely measured metric used to track a service’s performance — such as the percentage of requests completed within a target latency. Why it matters: It’s the raw, objective number every reliability conversation and decision ultimately has to be grounded in. In practice: Good SLIs are tied directly to user-visible outcomes, not just internal system health signals that don’t reflect real user experience. Common confusion: An SLI is the measurement itself — a service level objective is the separate target set against that measurement. Related terms: Service Level Objective, Error Budget, Availability

Service Level Objective (SLO)

Definition: The specific target committed to for a given service level indicator — for example, “99.9% of requests succeed within two seconds.” Why it matters: It turns a vague goal like “be reliable” into something concrete that can actually be tracked, alerted on, and held accountable to. In practice: SLOs are typically set slightly below what’s technically achievable, deliberately leaving room to absorb normal operational variance. Common confusion: An SLO is a target you’re committing to hit — it’s distinct from an SLA, which is typically a contractual commitment with real consequences attached for missing it. Related terms: Service Level Indicator, Error Budget, Availability

Error Budget

Definition: The amount of allowed failure a service level objective permits over its measurement window before that objective is considered exhausted. Why it matters: It gives reliability and product decisions a shared, objective basis: spend remaining budget on shipping fast, slow down and prioritize fixes once it’s running low. In practice: Teams typically track error budget burn by cause and by user segment, not just as a single flattened aggregate number. Common confusion: An error budget isn’t a license to deliberately cause incidents — it’s an operating policy derived directly from a real, user-facing reliability target. Related terms: Service Level Objective, Availability, Incident Response

Availability

Definition: The proportion of eligible service interactions or time windows in which users could actually obtain acceptable service. Why it matters: A service can technically be running while users still can’t complete anything useful — availability specifically has to be tied to user-visible success, not raw process uptime. In practice: Measuring availability well means clearly defining what counts as eligible and what counts as acceptable, and calculating consistently over a fixed window. Common confusion: Availability is just one reliability outcome among several — it doesn’t, on its own, describe latency, correctness, or safety. Related terms: Service Level Indicator, Error Budget, Incident Response

Incident Response

Definition: The coordinated process for detecting, analyzing, containing, recovering from, and communicating about an event that threatens a service. Why it matters: During an actual incident, clear roles and solid evidence matter far more than improvised heroics — especially when model behavior obscures where the real failure is happening. In practice: Effective incident response defines severity levels and command roles in advance, well before any actual incident occurs. Common confusion: Incident response manages the event itself and its immediate consequences — deeper root-cause analysis and long-term prevention continue well after service is restored. Related terms: Postmortem, Availability, Error Budget

Canary Release

Definition: A deployment strategy that exposes a new version to a small, limited slice of traffic before expanding the rollout to everyone. Why it matters: It limits the blast radius of any defect and produces real production evidence before a new model, prompt, or service reaches every user. In practice: A canary’s metrics get directly compared against the existing control version, with predefined failure conditions that trigger an automatic rollback. Common confusion: A canary release limits exposure to a new version — it doesn’t replace the need for pre-deployment testing or a solid rollback plan. Related terms: Evaluation, Graceful Degradation, Incident Response

Graceful Degradation

Definition: Preserving a bounded, functional core of a service when a dependency is impaired, by reducing optional features rather than failing every request outright. Why it matters: AI systems typically depend on several fallible components at once, so a deliberate reduced mode protects the essential user outcome during partial failure. In practice: Systems designed for this predefine exactly which capabilities may be disabled in advance, rather than improvising a fallback in the middle of an actual incident. Common confusion: Graceful degradation shouldn’t mean silently returning a worse answer as if nothing changed — users generally need some visibility when a promised guarantee has materially shifted. Related terms: Circuit Breaker, Load Shedding, Availability

Admission Control

Definition: A gate that decides whether a request may even enter a bounded queue or service, based on the system’s current capacity and policy. Why it matters: Rejecting excess work at a controlled boundary protects requests that have already been admitted from queue growth and cascading timeouts. In practice: Admission control typically estimates a request’s cost up front and checks available capacity before ever accepting it into the system. Common confusion: Admission control specifically acts before acceptance; load shedding can reject or remove already-accepted work at several different points in the system. Related terms: Load Shedding, Backpressure, Rate Limit

Load Shedding

Definition: Deliberately dropping or rejecting lower-priority work during overload conditions, in order to protect capacity for the requests that actually matter most. Why it matters: Without it, an overloaded system tends to serve everything slowly and badly rather than serving the most important work well. In practice: Shedding decisions typically factor in request priority, not just simple arrival order, so the most important traffic gets protected first. Common confusion: Load shedding is a reactive response to overload that’s already happening — admission control instead acts proactively at the door, before work is even accepted. Related terms: Admission Control, Graceful Degradation, Backpressure

Deadline Propagation

Definition: Passing the remaining end-to-end time budget down through every downstream call, so each dependency knows how long the original request can still usefully wait. Why it matters: Without it, independent timeouts at each layer can collectively exceed the user’s actual deadline, leaving abandoned work still consuming real capacity after the answer’s no longer useful. In practice: Deadline propagation sets one overall deadline at the entry point and subtracts elapsed time before every subsequent downstream call. Common confusion: A deadline marks an absolute completion boundary — a retry delay is a separate setting that still needs to fit inside that same overall budget. Related terms: Retry with Backoff, Service Level Objective, Tail Latency

Postmortem

Definition: A written analysis conducted after an incident, documenting what happened, why it happened, and what’s actually changing as a result. Why it matters: Without one, teams tend to fix the immediate symptom and quietly leave the actual underlying cause in place for the next incident. In practice: The most useful postmortems focus on concrete, specific system and process changes rather than vague resolutions like “be more careful.” Common confusion: A postmortem is a genuine learning document, not a tool for assigning individual blame — that framing is exactly what makes people honest in writing them. Related terms: Incident Response, Availability, Error Budget

11. Security & Governance

Prompt Injection

Definition: An attack where malicious instructions are embedded in a model’s input, attempting to override its intended behavior. Why it matters: It’s frequently described as the SQL injection of the LLM era — a fundamental, structural risk of any system that follows instructions written in natural language. In practice: Mitigations include clearly separating trusted instructions from untrusted content and enforcing least privilege on anything the model can actually act on. Common confusion: Prompt injection targets a system’s instruction-following behavior specifically — it’s related to, but not identical with, a jailbreak, which targets broader behavioral restrictions. Related terms: Indirect Prompt Injection, Instruction Hierarchy, Jailbreak

Indirect Prompt Injection

Definition: Prompt injection delivered through content a system retrieves or observes — a webpage, document, or tool output — rather than typed directly by the user. Why it matters: An agent has no easy, reliable way to distinguish “an instruction from my actual user” from “an instruction hidden inside this document I just happened to read.” In practice: Labeling all external, retrieved content strictly as untrusted data — never as authoritative instructions — is a core defense against this specific attack. Common confusion: “Indirect” describes the delivery path, not a weaker version of the attack — a hidden instruction buried in retrieved content can be exactly as consequential as a direct one. Related terms: Prompt Injection, Trust Boundary, Data Exfiltration

Data Exfiltration

Definition: Unauthorized transfer of protected data out of a system or trust boundary to a destination not permitted to receive it. Why it matters: An agent can leak sensitive data through generated text, a tool call’s arguments, or even an innocuous-looking URL — not only through an obvious database breach. In practice: Defenses include minimizing what data is readable in the first place, allowlisting legitimate destinations, and actively inspecting outbound tool calls. Common confusion: Ordinary retrieval of data by an authorized component isn’t exfiltration on its own — the risk is specifically in unauthorized movement or later disclosure of that data. Related terms: Trust Boundary, Least Privilege, Indirect Prompt Injection

Least Privilege

Definition: Giving any user, agent, or system only the minimum level of access it genuinely needs to complete its actual job. Why it matters: It directly limits the real-world damage a compromised, manipulated, or simply buggy agent can actually do. In practice: Applying this well means scoping an agent’s tool access, file access, and permissions narrowly and specifically per task, rather than granting broad standing access by default. Common confusion: Least privilege limits what’s possible for an agent to do — it doesn’t by itself guarantee that whatever it does within those limits is correct. Related terms: Scope Contract, Approval Gate, Defense in Depth

Trust Boundary

Definition: The line where data or instructions cross from a trusted source into a system, or from that system back out. Why it matters: It’s specifically the point where content needs to actually be validated, rather than assumed safe by default. In practice: Clearly identifying trust boundaries in a system’s design makes it much easier to decide exactly where validation and sanitization genuinely need to happen. Common confusion: A trust boundary is a structural, architectural concept — it’s distinct from any single specific control, like a guardrail, that gets applied at that boundary. Related terms: Defense in Depth, Indirect Prompt Injection, Data Classification

Defense in Depth

Definition: Using multiple independent, layered controls at several different system boundaries so that a single failed control doesn’t determine the overall outcome. Why it matters: AI systems combine probabilistic model behavior with untrusted content and real external tool access, which makes any single filter an inherently inadequate security boundary on its own. In practice: A defense-in-depth approach pairs instruction-level controls with narrow permissions, sandboxing, and human approval for genuinely consequential actions. Common confusion: Adding more layers isn’t automatically better — each one should address a genuinely distinct failure mode rather than just repeating the same underlying assumption. Related terms: Guardrails, Least Privilege, Trust Boundary

Data Classification

Definition: Assigning data to documented sensitivity levels so that handling, access, and retention rules can scale with the real consequences of it leaking. Why it matters: A pipeline can’t apply genuinely proportionate controls if source documents, prompts, and generated artifacts are all treated as equally sensitive by default. In practice: Classification is typically applied at ingestion and carried forward through every derived artifact that’s built from that original data. Common confusion: Data classification is about protection requirements — it’s a completely different concept from a machine-learning classification task, despite the shared word. Related terms: Data Minimization, Trust Boundary, Audit Log

Data Minimization

Definition: Limiting what data is collected, processed, exposed, and retained to only what’s genuinely necessary for a specific, defined purpose. Why it matters: Every unnecessary field sitting in a prompt, trace, or log is one more thing that can leak — reducing that surface reduces real risk directly. In practice: Effective minimization means defining exactly what fields are actually required before collection even begins, then setting clear retention limits. Common confusion: Minimization doesn’t mean collecting zero data — it means being able to genuinely justify every field, its use, and how long it’s kept, against the stated purpose. Related terms: Data Classification, Least Privilege, Context Engineering

Audit Log

Definition: A durable, access-controlled record of security- or accountability-relevant events, including who or what acted, what changed, and when. Why it matters: Consequential agent actions need real evidence that supports investigation and accountability — well beyond just ordinary performance debugging. In practice: A solid audit log records tool authorizations, approval decisions, and external writes, while carefully redacting secrets and restricting who can access the log itself. Common confusion: A trace typically helps diagnose one specific execution path; an audit log preserves events specifically required for accountability across executions and over real time. Related terms: Data Classification, Approval Gate, AI Risk Assessment

AI Risk Assessment

Definition: A documented analysis of how an AI system could realistically affect people and organizations — covering hazards, likelihood, impact, and the controls actually in place. Why it matters: A model’s raw capability alone doesn’t determine risk — deployment context, who’s affected, and human oversight all meaningfully change both the real harms and the controls actually needed. In practice: A useful assessment identifies concrete, credible failure and misuse scenarios rather than staying at a purely abstract, generic level. Common confusion: A risk assessment supports a decision made under stated assumptions at a point in time — it isn’t a permanent, one-time safety certificate. Related terms: Guardrails, Data Classification, Model Card

Model Card

Definition: Structured public documentation of a model’s intended use, known limitations, and evaluation results. Why it matters: It gives anyone adopting a model the information they actually need to use it appropriately, rather than discovering its real limitations the hard way in production. In practice: A useful model card covers evaluation results across relevant, meaningful subgroups, not just one aggregate accuracy number. Common confusion: A model card documents evidence and intended use — it’s not a legal guarantee or a substitute for your own deployment-specific evaluation. Related terms: Datasheet for Datasets, AI Risk Assessment, Evaluation

Datasheet for Datasets

Definition: Structured documentation of a dataset’s motivation, collection process, composition, and known limitations. Why it matters: A dataset being publicly available doesn’t make it safe or well-suited for every purpose — downstream teams need real evidence about how it was actually built. In practice: A datasheet is typically published and versioned alongside the dataset itself, and updated whenever the dataset materially changes. Common confusion: A datasheet documents evidence — it isn’t a license, a quality guarantee, or a substitute for your own task-specific evaluation of the data. Related terms: Data Provenance, Model Card, Data Deduplication

12. Multimodal Systems

Vision-Language Model (VLM)

Definition: A model trained to jointly understand images and text together — describing a photo, answering a question about a chart, reading text embedded in a screenshot. Why it matters: It extends the same reasoning capability that made large language models useful into genuinely visual tasks, rather than requiring an entirely separate system. In practice: A VLM typically combines a vision encoder with a language model, connected through cross-attention or a shared token space. Common confusion: A strong VLM doesn’t guarantee reliable, pixel-precise visual detail — the specific level of grounding depends heavily on the model’s actual architecture and training. Related terms: Cross-Attention, Vision Transformer, Multimodal Fusion

Vision Transformer (ViT)

Definition: A transformer architecture applied to images by dividing them into fixed-size patches and treating each patch as a token. Why it matters: It let the same core attention mechanism that revolutionized language modeling be applied directly to visual data, rather than relying purely on convolutional architectures. In practice: ViT-based models frequently need substantially more training data than CNNs to reach comparable performance, since they carry a weaker built-in assumption about image locality. Common confusion: A ViT patch isn’t the same thing as a single pixel or one specific object — its exact scope depends entirely on the chosen patch size. Related terms: Transformer, Image Token, Inductive Bias

Cross-Attention (in multimodal systems)

Definition: Attention where queries from one modality — like text — attend to keys and values from a different modality, like image features. Why it matters: It’s the specific mechanism that lets one modality learnably pull relevant information directly out of another. In practice: A captioning model’s text-generation side might use cross-attention to pull in relevant visual features while it writes each successive word. Common confusion: Cross-attention on its own doesn’t guarantee genuine cross-modal understanding — it provides the mechanism, but real alignment still depends on the training. Related terms: Attention, Multimodal Fusion, Vision-Language Model

Early Fusion

Definition: Combining raw or low-level representations from multiple modalities before most of the actual task-specific modeling occurs. Why it matters: It can expose genuinely fine-grained cross-modal relationships, but it also requires compatible representations and careful handling of missing input from either side. In practice: Early fusion typically converts each modality into a shared token representation and fuses them before the main shared backbone even begins processing. Common confusion: Early fusion describes where in the architecture streams get combined — it doesn’t automatically guarantee the model actually learns useful alignment between them. Related terms: Late Fusion, Multimodal Fusion, Image Token

Late Fusion

Definition: Processing each modality mostly independently and combining the results only near the end of the model. Why it matters: It’s simpler to build and train than early fusion, but it can miss finer-grained relationships between modalities that only become visible earlier in processing. In practice: Late fusion is a common, practical choice when strong single-modality models already exist and full end-to-end joint training isn’t feasible. Common confusion: Late fusion isn’t strictly “worse” than early fusion in every case — the better choice genuinely depends on the task and how tightly correlated the modalities actually need to be. Related terms: Early Fusion, Multimodal Fusion, Cross-Attention

Image Token

Definition: A discrete unit representing part of an image — typically derived from a patch, region, or a learned visual codebook entry. Why it matters: Turning visual input into token-like units lets transformer-based models process images using largely the same machinery built for text. In practice: Image tokens count against a model’s overall input token budget, exactly the same way text tokens do. Common confusion: An image token isn’t necessarily one pixel or one distinct object — its actual scope depends entirely on the specific visual encoder or tokenizer used. Related terms: Vision Transformer, Patch Embedding, Token Budget

Audio Token

Definition: A discrete identifier produced by an audio codec or tokenizer representing a short segment or feature of an audio signal. Why it matters: Discrete audio representations let sequence models process, predict, and generate sound using the same token-oriented architecture built for text. In practice: Audio tokenization needs careful versioning alongside the model, since a mismatched codec can silently corrupt generation quality. Common confusion: An audio token doesn’t correspond to a fixed duration or a specific phoneme — its actual meaning depends entirely on the tokenizer and codebook design. Related terms: Automatic Speech Recognition, Token, Embedding

Automatic Speech Recognition (ASR)

Definition: The task and pipeline that maps spoken audio into text transcription. Why it matters: It’s the foundational technology behind every voice assistant’s “listening” step, and speech interfaces genuinely depend on far more than language modeling alone. In practice: Production ASR systems get evaluated by word or character error rate, broken down across languages, accents, and background noise conditions. Common confusion: ASR transcribes what was said — identifying who said it is a separate task called speaker diarization, and translating it is separate still. Related terms: Audio Token, Encoder, Multimodal Model

Diffusion Model

Definition: A generative model trained around a progressive noising process and a learned reverse process, typically starting generation from pure noise and denoising it step by step. Why it matters: It’s the architecture behind most of today’s leading image, audio, and video generation systems. In practice: Many diffusion models operate in a compressed latent space rather than directly on raw pixels, which substantially speeds up both training and generation. Common confusion: Diffusion is a general-purpose generative framework — it’s not an image-only technique, despite images being its most visible application. Related terms: Latent Space, VAE, Inference

GAN (Generative Adversarial Network)

Definition: A generative setup with two networks trained together — a generator producing synthetic data and a discriminator trying to distinguish real data from generated fakes. Why it matters: It was one of the earliest architectures capable of producing genuinely convincing synthetic images, and it established the whole idea of adversarial training. In practice: GANs can be notoriously unstable to train, which is part of why diffusion models have become more widely favored for many generation tasks in recent years. Common confusion: A GAN’s generator and discriminator are trained together, adversarially — neither one is trained to completion independently before the other joins in. Related terms: Diffusion Model, Loss Function, Latent Space

VAE (Variational Autoencoder)

Definition: A generative model that learns to compress data into a structured latent space and then decode it back into the original data space. Why it matters: It’s useful both for generating new data and for learning genuinely meaningful compressed representations that other systems can build on. In practice: VAEs are frequently used as the compression step underneath a larger diffusion pipeline, so the actual diffusion process operates in a smaller latent space rather than raw pixels. Common confusion: A VAE’s latent space is deliberately structured to support generation — that’s a different design goal from a general-purpose embedding aimed purely at retrieval or similarity search. Related terms: Latent Space, Diffusion Model, Embedding

Multimodal Fusion

Definition: The general umbrella term for any technique that combines information from multiple modalities into a shared representation a model can reason over. Why it matters: It’s the connecting concept behind every technique in this section — early fusion, late fusion, and cross-attention are all specific approaches to solving this same underlying problem. In practice: The right fusion approach for a given system depends on the specific task, the available compute, and how tightly the modalities genuinely need to interact. Common confusion: Multimodal fusion isn’t one single technique — it’s a category encompassing several genuinely different architectural strategies with real tradeoffs between them. Related terms: Early Fusion, Late Fusion, Cross-Attention

The TL;DR

Twelve layers, over 150 terms, one continuous reference: the math that trains a model, the architecture that runs it, the data that feeds it, the retrieval that grounds it, the prompting that steers it, the agents that act with it, the evaluation that checks it, the development practices that keep AI-native software sane, the infrastructure that serves it, the reliability work that keeps it up, the security that protects it, and the multimodal layer extending it past text.

Bookmark this. Search it when a term comes up mid-conversation. That’s genuinely what it’s for.

Contact & Connect

Website: https://www.adilshamim.me/

LinkedIn: https://www.linkedin.com/in/adilshamim8

GitHub: https://github.com/AdilShamim8

Kaggle: https://www.kaggle.com/adilshamim8

Twitter/X: https://x.com/adil_shamim8

Medium: https://adilshamim8.medium.com/