A model advertising a 1 million-token context window does not sell one million tokens for a fixed price. It sells the ability to process up to that much active context in a request. You are still billed according to the provider's rate card, and a multi-turn conversation can process the same history many times.
That distinction explains the “why did this cost so much?” question behind long-document chats and agents. This guide treats context-window pricing as an accounting problem: what enters each request, what gets repeated, which tokens receive a cache discount, and when a large-window premium changes the rate.
TL;DR: capacity is not price
| Question | Direct answer |
|---|---|
| Is 1M context a prepaid bucket? | No. It is a maximum capacity; actual processed tokens are billed |
| Do earlier messages count again? | Usually yes, because relevant history is processed on later turns |
| Can 10 × 10K turns bill 100K total? | Usually more—cumulative history can make it roughly 550K input tokens |
| Does caching make repeats free? | No. Cache reads are discounted, and writes/storage/expiry rules apply |
| Can long context have a higher rate? | Yes. Some models apply premium rates above a threshold |
| Best cost control? | Retrieval plus compaction: send the state needed for the next decision, not the whole archive |
Context window, input tokens, and output tokens are different meters
The context window is the combined space available to the model for instructions, conversation history, documents, images converted to tokens, tool results, reasoning state where applicable, and generated output. Providers define details differently, but the practical constraint is:
input tokens + reserved/generated output tokens <= context capacity
The rate card usually meters input and output separately. Output often costs several times more because generation is sequential and computationally expensive. Some providers separately identify cached input, cache writes, storage duration, search calls, or other tools.
OpenAI's GPT-5.5 model page is a clean July 2026 example: a 1.05M-token window, $5/M input, $0.50/M cached input, and $30/M output. It also states that prompts above 272K input tokens move to 2× input and 1.5× output pricing for the full session. The big window is a capability and a pricing threshold at the same time.
Worked math: ten turns that add 10K tokens each
Imagine a conversation where each turn adds 10,000 tokens of new user material and model output combined. To keep the example simple, assume the application sends all prior state on the next request.
| Turn | Active prior context sent | New material | Approximate billed input |
|---|---|---|---|
| 1 | 0K | 10K | 10K |
| 2 | 10K | 10K | 20K |
| 3 | 20K | 10K | 30K |
| 4 | 30K | 10K | 40K |
| 5 | 40K | 10K | 50K |
| 6 | 50K | 10K | 60K |
| 7 | 60K | 10K | 70K |
| 8 | 70K | 10K | 80K |
| 9 | 80K | 10K | 90K |
| 10 | 90K | 10K | 100K |
| Total processed input | — | 100K unique | 550K cumulative |
The user introduced 100K tokens, but the model processed roughly 550K input tokens across the sequence. The exact split depends on when outputs are included and how the API manages state, yet the triangular growth is the important idea:
10K × (1 + 2 + ... + 10) = 10K × 55 = 550K
At $5 per million input tokens, that input costs $2.75 before output. At $0.50/M cached input, a perfect cache hit on every repeated prefix would lower the repeated portion dramatically—but not all 550K is automatically cached, and new suffixes still use the base rate.
Why 1M vs 128K is not a fair sticker-price comparison
Suppose Model A supports 1M tokens at $5/M input. Model B supports 128K at $2/M. A shallow comparison says Model A costs 2.5 times more. Real workloads create three different comparisons.
Scenario 1: the task is 40K tokens
Both models fit it. Model A input costs $0.20; Model B costs $0.08. Unless Model A's quality saves a retry or more expensive output, B is cheaper.
Scenario 2: the task is a 300K corpus, but only 30K is relevant
Model A can accept everything. The base-rate input would be $1.50, though GPT-5.5's published premium threshold would make a comparable over-threshold request more expensive. Model B cannot accept everything, but a retrieval system can select the relevant 30K for $0.06 plus retrieval overhead.
The smaller context did not lose. Better context selection won.
Scenario 3: the answer requires relationships across all 300K tokens
Chunking may miss cross-document contradictions or dependencies. A large-window model can be worth the premium because it reduces orchestration, repeated summaries, and reconciliation failures. The proper metric is cost per correct completion, not cost per input token.
This is why AI benchmark scores and context size alone are poor procurement tools. They omit your corpus structure, required recall, latency, and cost of a wrong answer.
The four ways repeated context gets billed
1. Uncached input
New or non-matching prompt content receives the standard input rate. Reordering instructions, changing early text, injecting timestamps at the top, or varying tool definitions can prevent a stable prefix from matching a cache.
2. Cache writes
Some providers charge a premium to create a reusable cache entry. Anthropic's pricing documentation describes different multipliers for five-minute and one-hour cache writes. A write only pays back when the prefix is reused enough before expiry.
3. Cache reads
Cache hits are heavily discounted—often around one-tenth of base input in representative rate cards—but not free. They also do not discount newly appended conversation state or a prefix that changed.
4. Provider-managed conversation state
An API may let you reference a prior response rather than manually resending the transcript. That simplifies application code; it does not imply zero-cost memory. Inspect the usage object returned by the API and the provider's stateful-conversation rules.
A cache break-even example
Take a stable 200K-token policy manual on a model with a $3/M base input rate, a 1.25× five-minute cache-write multiplier, and 0.1× cache-read rate.
Without caching, five uses cost:
5 × 0.2M × $3.00 = $3.00
With one cache write and four hits:
write: 0.2M × $3.75 = $0.75
reads: 4 × 0.2M × $0.30 = $0.24
total: $0.99
The cache saves $2.01 across five uses. If you use the manual once, the premium write costs more than ordinary input. Caching is an optimization for repeated stable prefixes, not a checkbox to turn on everywhere.
Why agents amplify context costs
A human chat may have ten turns. An agent can make ten calls before the user sees the first result: plan, search, open, parse, rank, search again, call a database, draft, critique, revise. Tool outputs are inserted into later calls, making the active state grow.
Our real monthly agent budget models one 20-call research run at 450K cumulative input tokens. The same mechanism scales to the $600/day case in our Varick agent cost teardown: spend is often a loop-shape problem, not merely an expensive-model problem.
Three design mistakes make it worse:
- Appending raw pages forever. Search snippets, HTML navigation, repeated disclaimers, and duplicate passages stay in state after their useful facts were extracted.
- Regenerating whole artifacts. Asking for a complete 4,000-word rewrite after one factual correction pays output cost again and adds the new draft to future input.
- Parallel duplication. Five subagents each receive the same 100K reference pack, creating 500K input before any of them act.
Context engineering that reduces cost without making the model blind
Retrieve evidence, do not preload the library
Index the corpus, search for likely relevant passages, then let the model request more when evidence is insufficient. Retrieval can fail, so preserve citations and run recall tests on known questions. The choice is not “everything” versus “nothing”; it is controlled disclosure.
Compact completed work into verified state
After a stage finishes, replace the full transcript with a structured checkpoint:
objective: Compare three supplier proposals
verified_facts:
- Supplier A total: $184,000; source: proposal-a.pdf p.12
- Supplier B excludes migration; source: proposal-b.pdf p.7
open_questions:
- Supplier C data residency language conflicts across pages 4 and 18
decisions:
- Normalize all totals to a 24-month term
discarded:
- Marketing claims without contractual support
This preserves provenance and unresolved risk while deleting discussion that no longer affects the next action.
Keep the prefix stable
Put system instructions, tool schemas, static examples, and long reference documents in a consistent order. Append variable user content later. Avoid timestamps or random identifiers near the beginning unless they genuinely affect behavior.
Bound tool output
Ask browsers and databases for selected fields, relevant passages, or a byte/token cap. Store the complete artifact outside the context and pass a handle plus excerpts. A model does not need 80KB of HTML to know a page title, publication date, and three relevant paragraphs.
Patch instead of regenerate
For documents and code, request a targeted edit or diff. This reduces expensive output, avoids feeding multiple near-identical drafts into state, and makes human review easier.
What people are asking about long context
Does the model “remember” everything inside the window equally?
No. Capacity is not guaranteed retrieval quality. Models can miss details in the middle of long inputs, confuse near-duplicate facts, or over-weight recent instructions. Test the exact document shape and ask for citations or evidence spans.
Is a 1M window better than RAG?
They solve different failure modes. Long context preserves global relationships; retrieval controls relevance and cost. Strong systems combine them: retrieve a high-recall working set, then use enough context to compare it properly.
Why did one long request cost more than my arithmetic?
Check whether you crossed a premium threshold, whether reasoning tokens were billed as output, whether images or PDFs were tokenized, whether tools created separate fees, and whether the provider counted the whole session at the higher rate. The returned usage fields are more authoritative than the visible text box.
A preflight checklist before sending 500K tokens
- Can retrieval reduce the corpus without losing cross-document relationships?
- Will the request cross a premium long-context threshold?
- Is the stable prefix eligible for caching, and will it be reused before expiry?
- How much output and reasoning should be reserved?
- Are duplicate pages, boilerplate, or old drafts present?
- Can tool output stay external until explicitly requested?
- What is the maximum dollar cost of this call and one retry?
- What evidence will prove that the large context improved the result?
A million-token window is valuable when the job needs it. It is expensive when an application uses capacity as a substitute for deciding what matters. Price the whole trajectory, not the advertised window.
Related on explainx.ai
- What running an AI agent costs per month
- AI token costs surge: enterprise finance and governance
- Context, prompt, loop, and harness engineering
- Context-mode MCP and sandboxed agent context
- SubQ sparse attention and 12M context
- What is an agent harness?
Pricing examples and thresholds were checked July 26, 2026. Provider tokenization, caching, state management, and long-context rules differ and change; recalculate with the live documentation and usage fields for your exact model.