Context engineering: the 2026 skill every AI builder needs

If you asked teams in 2024 what separated good AI apps from bad ones, most would have said model choice or prompt wording. Ask in 2026 and the answer has shifted: it is what you put in front of the model. Andrej Karpathy put it pithily in June 2025 when he described context engineering as "the delicate art and science of filling the context window with just the right information for the next step" (Towards AI, 2026). A year on, the phrase has hardened from a catchy framing into a genuine engineering discipline — and it is the one that separates demos from production systems.

Here is the uncomfortable math. In a typical agentic run, the user's message is a tiny fraction of what reaches the model — the system prompt, tool schemas, retrieved documents, memory, conversation history, and prior tool outputs make up nearly all of it (Dev.to, 2026). You are not really "prompting" a model. You are assembling an input package, and the quality of that package is your quality ceiling.

This guide gives you a systematic approach to context design: why it matters, what can go wrong, and the concrete techniques production teams use to get it right.

Why context quality beats model choice#

Three lines of evidence make the case.

First, context rot is real and universal. Chroma Research tested 18 frontier models across nearly 195,000 calls in July 2025 and found that every model degrades as input length grows — well before the advertised window fills, and even on intentionally simple tasks like fact retrieval. Distractors compound the damage, and models with near-perfect "needle in a haystack" scores still fail on realistic long-context tasks. Practical rule of thumb from the research: the effective context — the part the model actually uses reliably — is roughly 30–60% of the advertised window.

Second, position matters as much as content. The "Lost in the Middle" finding from 2023 (Liu et al.) showed a U-shaped attention curve: models recall best what's at the beginning and end of a context, and can lose 20+ percentage points of accuracy when critical facts sit in the middle. Stuffing more documents into the window doesn't make the model smarter; it buries the signal in low-attention zones.

Third, attention is zero-sum. Every token you add draws from a finite attention budget. Irrelevant passages steal weight from relevant ones, and semantically similar but wrong content ("distractors") confuse the model's relevance judgment. More context can actively hurt — a finding Anthropic formalized in its September 2025 engineering post on effective context engineering for agents, which frames the context window as a finite budget to curate, with the target being "the smallest possible set of high-signal tokens."

The takeaway: you cannot buy your way out of bad context with a bigger model or a bigger window. Context is a scarce resource. Budget it.

The anatomy of a context package#

Before optimizing, map what actually flows into your model on a typical call. In a 2026-era agent, the layers are roughly:

Two properties make this tractable: each layer has its own cost model, cache behavior, and update cadence — optimizable independently. And a feature that exceeds its budget is a defect, not a request for a bigger window.

Technique 1: Budget tokens like money#

The single highest-leverage habit is giving every layer an explicit token budget and enforcing it in code. Hard-cap RAG chunks. Cap conversation history. Cap tool schemas by scoping them to the current turn. When a layer overflows its budget, you compress or drop — you never silently let it grow.

Pair budgets with prompt caching (supported by major providers): put your stable system prefix first, exactly identical across calls, and let the provider cache it. Then inject dynamic content after the cached prefix, never before — reordering breaks cache hits.

Technique 2: Compress, don't stuff#

When context must shrink, you have a ladder of options, from cheapest to most expensive:

• Summarization checkpoints. After N turns of tool use, summarize the trajectory into a compact state: decisions made, TODOs, constraints learned. Anthropic's engineering guidance specifically recommends tuning the compaction prompt toward recall — it is better for the summary to include something irrelevant than to lose a critical constraint.
• Structured note-taking. Keep durable state outside the window — a NOTES.md or memory tool the agent explicitly reads and writes — rather than relying on raw history. This is cheaper than carrying full transcripts and far more reliable than hoping the model remembers.
• Reference over inclusion. Pass document IDs and let a tool fetch the full text on demand, instead of dumping every retrieved document into the window up front.
• Token compression. Techniques like LLMLingua-2 can reportedly compress long documents by large factors with small quality loss — useful for the retrieved-evidence layer, though results vary by content type, so measure on your own data.

The common failure mode to avoid: compressing the instructions while leaving the noise. Keep rules and constraints intact; compress history and evidence.

Technique 3: Scope tools and retrieve just in time#

Two of the biggest context hogs are tool schemas and retrieved documents, and both suffer from the same disease: loading everything "just in case."

• Scope tool schemas per turn. An agent with 40 tools sees 40 schemas on every call. If only 3 are relevant to the current step, scope the rest out. Anthropic's guidance on writing tools for agents also argues for fewer, consolidated, workflow-shaped tools — each one you remove permanently shrinks every future context.
• Just-in-time retrieval. Fetch context when the agent needs it, not preemptively. RAG hygiene matters enormously here: Chroma's context-rot work shows that even a single semantically-similar-but-wrong passage degrades accuracy. So retrieve narrowly, rank aggressively, and cap the number of chunks. Five excellent chunks beat twenty mediocre ones.
• Isolate sub-agent contexts. For deep dives, spawn a sub-agent with its own fresh context and require only a distilled summary back. This keeps the coordinator's window clean — at the cost of more total tokens, so reserve it for genuinely parallel, high-value work (Anthropic reported multi-agent runs using roughly 15× more tokens than chat-style runs, so the bar is high).

Technique 4: Defend the trust boundary#

Context engineering is also a security discipline. Retrieved documents, tool outputs, and web pages are data, not instructions — but models can treat injected text in those sources as commands. Separate untrusted content clearly, keep system rules authoritative over anything retrieved, and record provenance: where did each important fact come from, and how fresh is it? A stale policy document overriding a newer one is a context bug with security implications.

A checklist you can run this week#

If you maintain any LLM app or agent, here is the practical pass:

1. Inventory the window. Log, for one real session, every layer of a typical call and its token count. You will be surprised.
2. Set budgets. Give each layer a cap; make overflows a logged defect, not a silent event.
3. Cache the stable prefix. Order matters: static first, dynamic after.
4. Cap and rank RAG. Fewer, better chunks. Measure retrieval precision, not just recall.
5. Scope tools. Remove or hide tools the current turn can't plausibly need.
6. Add compaction. Summarize long trajectories; keep decisions and constraints, drop play-by-play.
7. Move durable state out of the window. Memory tools and notes files beat raw history.
8. Measure context quality, not just answers. Track per-layer token usage, cache hit rates, and how accuracy changes as context length grows. If you can't see context rot in your own system, you can't manage it.

The takeaway#

Prompt engineering was about wording a single message. Context engineering is about designing the entire information environment the model operates in — what enters the window, in what order, at what cost, and with what authority. The research is unambiguous that bigger windows don't solve this: every frontier model tested degrades with length, attention is zero-sum, and position shapes recall. The builders winning in 2026 aren't the ones with the cleverest prompts or the newest model. They're the ones treating context as an engineered, budgeted, measured system — and giving the model the smallest possible set of high-signal tokens for the next step.