The Context Window Is Not a Clipboard
Bigger context windows didn't solve the problem of what goes in them. Most production LLM failures aren't model failures — they're context failures.
The models keep getting bigger context windows. 128k. 200k. 1M tokens. Teams see those numbers and do the same thing every time: dump more in.
The whole document. The full conversation history. Every tool output. Every intermediate result. All of it, concatenated, pasted in before the instruction.
Then they wonder why quality degrades.
Context is not neutral.
Every token you include shifts the model’s attention. Relevant signal competes with irrelevant noise. The model isn’t reading your context the way you read a document — it’s weighing all of it simultaneously. When you include 50 pages of background material and then ask a specific question at the end, you’ve created an attention problem, not a convenience.
This is why “just give it more context” is bad engineering advice. More context doesn’t help if it dilutes the signal. In many cases, it actively hurts.
There’s a name for this in the research: lost-in-the-middle. Critical information buried in the center of a long context window is reliably underweighted compared to information at the edges. The model isn’t ignoring it intentionally — the architecture just doesn’t weight position uniformly. That’s a known property of transformers. If you’re not designing around it, you’re not designing.
The failures look like model failures. They’re not.
“The model missed the key constraint.” → You buried it in paragraph eight.
“The model gave a generic answer.” → You gave it generic context.
“The model hallucinated a detail that was in the document.” → You gave it a 40-page document when you needed three paragraphs.
Before you conclude the model is wrong, audit what you gave it. The context window is an input, and you’re responsible for its quality.
What context engineering actually looks like:
Select, don’t aggregate. Retrieve what’s relevant to the specific query. Not the whole knowledge base — the relevant chunks, re-ranked by relevance, trimmed to what’s needed.
Put important things at the edges. If something must not be ignored, it belongs at the start of the context or immediately before the instruction. Not in the middle.
Compress intermediate results. If your agent loop is passing full tool outputs back into context, start summarizing them. The model doesn’t need the raw JSON — it needs the meaning.
Version your context templates. The shape of your context is part of your prompt. Treat it like code. Track what changed, measure the impact.
Set a context budget. Maximum tokens per section, enforced. Not a soft guideline — a hard limit that forces you to prioritize.
The 1M token context window is useful.
It’s useful for ingesting large documents when you need to reason across the whole thing. It’s not a license to stop thinking about what you’re sending.
A bigger clipboard doesn’t make you a better writer. A bigger context window doesn’t make your system smarter. It just means you have more room to make the same mistake at greater scale.
Design the context. Measure its quality. Treat what goes in as carefully as you treat what comes out.
The model’s performance is bounded by your inputs. Most teams are leaving headroom on the table.