Why AI Agents Get Dumber the Longer They Run: Context Rot and Context Engineering Explained

Run an AI agent for twenty turns and it starts making mistakes it wouldn't have made on turn two: forgetting an instruction you gave early on, contradicting a fact it already established, or fixating on an irrelevant detail from a tool call six steps back. The instinctive explanation is "the model got dumber." It didn't. What actually happened has a name now: context rot, and it's a property of how transformer attention behaves as input length grows, not a bug you can patch by switching models.

Chroma, the open-source vector database, ran the study that gave the phenomenon its name. Their July 2025 technical report tested 18 frontier models, including GPT-4.1, Claude Opus 4, and Gemini 2.5, on controlled tasks where input length was the only variable and task difficulty stayed constant. Every single model showed the same pattern: performance degrades non-uniformly as context grows, and a 200K-token context window can already show meaningful accuracy loss well before you've filled even a quarter of it. The full methodology and code is on GitHub if you want to reproduce it against your own model choice.

Why "just use a bigger context window" doesn't work

The instinct to fix this by upgrading to a model with a larger window misunderstands the mechanism. Context rot isn't a capacity problem, it's an attention problem. The original evidence for this goes back to a 2023 Stanford, UC Berkeley, and Samaya AI paper, Lost in the Middle, which found that models retrieve information reliably from the start and end of a long context but get measurably worse at pulling facts from the middle, a U-shaped accuracy curve that shows up across model families and hasn't gone away as context windows have grown into the millions of tokens. Chroma's follow-up study found the same shape plus two compounding effects: attention dilution, where relevance gets spread thinner as more tokens compete for it, and distractor interference, where content that's topically similar but actually irrelevant actively pulls the model's attention away from the right answer.

Put plainly: stuffing more tokens into the prompt to "give the model more to work with" adds noise faster than it adds signal. A model with a 1M-token window doesn't reliably reason across 1M tokens, it reliably reasons across the tokens closest to the edges of what you gave it. If you've read our piece on why RAG beats naive context stuffing for grounding an agent in your own data, this is the deeper reason that approach works: retrieval keeps the relevant tokens near the edges of a short context instead of buried in the middle of a long one.

The four moves: what Anthropic actually recommends

Anthropic's own engineering team published a detailed breakdown of how they build agents around this constraint, and it's a useful mental model precisely because it comes from a team that operates agents at scale, not a vendor selling a fix. They frame context as "a finite resource with diminishing marginal returns," and describe four concrete techniques:

• Compaction — when a conversation approaches its context limit, summarize the history into a high-fidelity distillation and restart from that summary instead of truncating or letting the window overflow. Anthropic notes compaction should preserve enough fidelity that the agent "continues with minimal performance degradation," not just enough to fit.
• Structured note-taking — the agent writes persistent notes (a task list, a scratchpad file, a memory block) outside the active context window and reads them back in only when relevant, rather than keeping the entire task history live in every call.
• Sub-agent architectures — a coordinating agent delegates focused work to sub-agents that each get a clean, narrow context window, then return a condensed result. This is the same separation-of-concerns idea we covered in our LangGraph vs CrewAI vs AutoGen comparison for multi-agent orchestration, applied specifically to keeping any one agent's context small.
• Just-in-time retrieval — instead of pre-loading every document an agent might need, keep lightweight references (file paths, IDs, search queries) in context and fetch the actual content only when a step needs it, mirroring how a person doesn't memorize a library before answering a question about one book in it.

What this looks like in a real API call

This isn't just a design philosophy, Anthropic ships it as an actual API feature. Context editing on the Claude API includes a `clear_tool_uses_20250919` strategy that automatically clears the oldest tool call results once a conversation crosses a token threshold you configure, on the reasoning that a file you read three tool calls ago is usually dead weight once Claude has already extracted what it needed from it. A separate `clear_thinking_20251015` strategy does the same for extended-thinking blocks. The key distinction from compaction: compaction summarizes and replaces the whole conversation, while context editing selectively prunes specific content types and leaves everything else intact, so you can combine both depending on what's filling up the window. Anthropic's own context engineering cookbook walks through wiring memory, compaction, and tool-result clearing together for a genuinely long-running agent.

The free, open-source route: compressing the prompt itself

Not every stack has access to managed context editing, and not every model provider offers it. If you're running an open-weight model or want to cut token cost regardless of provider, LLMLingua from Microsoft Research is the open-source tool built for exactly this. It's Apache-licensed, available on GitHub and Hugging Face, and works by removing low-information tokens from a prompt while preserving the ones that carry meaning, claiming up to 20x compression with minimal quality loss on the original version. Its follow-up, LongLLMLingua, specifically targets the lost-in-the-middle problem and reports up to 21.4% better RAG performance using a quarter of the original tokens, and LLMLingua-2 trades some compression ratio for 3-6x faster inference. If your pipeline already does retrieval, dropping a compression pass between retrieval and generation is a low-effort way to fight context rot without touching your model provider at all — and it stacks with prompt caching, since a shorter, well-compressed static prefix caches just as well as a long one while costing less per call when it misses cache.

Context engineering vs. agent memory: not the same problem

It's worth being precise about the boundary here, because it's easy to conflate this with memory systems. Context engineering is about managing what's in the window for a single, bounded task. Agent memory, the kind Mem0, Zep, and Letta provide, is about persisting facts across sessions that would otherwise disappear entirely once a conversation ends. You need both for a production agent: memory decides what's worth keeping at all, context engineering decides what's worth keeping in front of the model right now. An agent with perfect long-term memory and no context discipline will still rot mid-task; an agent with great context hygiene and no memory will forget everything the moment the session resets.

Building this into your own agent

The practical checklist, in order of effort: measure before you optimize (Chroma's methodology gives you a repeatable way to test your own model and task against increasing input length), retrieve narrowly instead of pre-loading broadly, clear or compact tool results once they've served their purpose, and split any task that's growing a single agent's context past a few thousand tokens of history into sub-agents with their own clean windows. None of this is exotic engineering, it's closer to memory management in any other software system: finite resource, real cost to holding onto stale state, and a real cost to losing state you actually needed. We build exactly this discipline into the agent projects inside AI Product Development Bootcamp, where "why is my agent getting worse on long tasks" is one of the first production bugs students learn to diagnose instead of papering over with a bigger model.