An agent with a 1-million token window that still fails to retrieve a single critical fact hidden in the middle of its memory is a useless agent.
This is “context rot”. Pure noise. It proves that window size is a vanity metric. The real bottleneck in agentic workflows isn’t how much an agent can “see”, but how much it can actually attend to. We’ve spent the last two years obsessed with prompt engineering, tweaking the wording of a request to get a better answer. But as we move toward autonomous agents that run for hours or days, the problem shifts from how we ask to what the agent knows at any given millisecond.
We’re moving from Prompt Engineering to Context Engineering.
The Shift from Prompt Engineering to Context Engineering
For a long time, the industry treated the LLM as a stateless function:
f(prompt) = response
If the response was bad, you changed the prompt. But agents aren’t functions; they’re processes. They require a persistent information environment.
The technical reason for this shift is the transformer constraint. In a standard transformer architecture, every token attends to every other token. As you increase the context window, the computational cost grows quadratically (and usually costs a fortune in API credits), but more importantly, the “attention budget” gets diluted.
Think of it like a spotlight. In a 2k token window, the spotlight is tight and bright. In a 1M token window, the light is spread so thin that the model starts missing the “needle” in the haystack. This creates a performance gradient: the more noise you feed the model, the higher the probability it will hallucinate or ignore a critical constraint.
This is why I argue that “more context” is often a bug, not a feature.
Prompting is about instructions; contexting is about the information environment. If you’re just stuffing a PDF into a prompt and hoping for the best, you’re not building an agent, you’re building a very expensive search bar. Real context engineering is the art of high-signal token curation. It’s about aggressively pruning the input so the model only sees the absolute minimum required to make the next decision.
The Anatomy of a Context Platform: Memory, State, and Recall
If the LLM is the CPU, the context platform is the RAM and the Disk. You can’t rely on the model’s internal weights to store user preferences or task progress. You need an external architecture that manages how information flows into that narrow attention window.
Short-term vs. Long-term Memory: Implementing Biological Decay
Most developers implement memory as a simple list of the last messages. This is a recipe for failure. It leads to “context drift”, where the agent forgets the original goal because it’s too focused on the last three turns of conversation.
I prefer a tiered memory approach that mimics biological decay.
Working Memory (L1): The immediate conversation turn. High precision, zero latency.
Episodic Memory (L2): Summary-based snapshots of previous turns.
Semantic Memory (L3): Long-term facts stored in a vector database or graph.
The trick is “biological decay”. Not all information is equal. A user’s preference for “dark mode” should persist forever; the fact that they were annoyed by a specific API error three minutes ago should decay. I’ve seen teams implement a “decay score” where tokens are pruned or summarized based on their utility over time. If a piece of context hasn’t been accessed in five turns, it gets compressed into a summary or evicted entirely.
Beyond Chat History: Graph State Machines and Actor Models
Chat history is a linear lie. Real tasks are non-linear. If an agent is researching a company, then writing a report, then correcting a typo in that report, a linear history just looks like a mess of text.
To get actual traceability, you need a Graph State Machine. Instead of a list of messages, the agent’s state is a node in a graph.
By tracking the state as a graph, you can provide the agent with “state-aware context”. When the agent is in the “Drafting Phase”, it doesn’t need the raw search results from the “Research Phase”, it only needs the synthesized analysis. This saves your attention budget for the task at hand.
For high-reliability agents, I’d even suggest an Actor Model. Treat each sub-task as an independent actor with its own private memory. This prevents “cross-contamination” where a failure in one part of the loop poisons the context for the rest of the agent’s reasoning.
Checkpointing: Preventing Agent Amnesia
I once worked with a team building an autonomous coding agent that could run for an hour. They hit a wall where, halfway through a complex refactor, the agent would “forget” why it had changed a specific variable in the first file, leading to a cascade of breaking changes.
The fix was checkpointing.
Just like a database transaction, an agent should create a “context snapshot” before entering a high-risk loop. If the loop fails or the agent gets lost in a reasoning spiral, you don’t just restart, you roll back to the last known good state. This prevents the “amnesia” that occurs when an agent’s context window is filled with 50 turns of failed attempts to fix a bug.
Architectural Patterns for Context Delivery
How do you actually get this data into the model without creating a monolithic, brittle mess of glue code?
Context-as-a-Service (CaaS)
Stop building your context retrieval logic directly into your agent’s main loop. Decouple the “World Model” from the “Reasoning Engine”.
In a CaaS architecture, the agent doesn’t “search a database”. Instead, it sends a request to a Context Service: “I’m currently in State X, trying to achieve Goal Y. Give me the relevant context”.
The CaaS layer handles the heavy lifting, vector search, SQL queries, API calls, and pruning, and returns a curated package of tokens. This allows you to swap out your retrieval strategy (e.g., moving from Pinecone to a GraphDB) without touching your agent’s core logic.
The Model Context Protocol (MCP)
We’ve spent years building bespoke connectors for every single data source. It’s a nightmare. The Model Context Protocol (MCP) is the move toward a plug-and-play ecosystem.
Instead of writing a get_jira_ticket() function for every agent, MCP standardizes how a server exposes tools and resources to a model. It shifts the burden of connectivity from the agent to the provider. If every enterprise tool (Salesforce, GitHub, Slack) implemented an MCP server, your agent would instantly have a standardized way to “plug in” to the company’s context without you writing a single line of integration code.
Agentic RAG vs. Text-to-SQL
You’ll hear a lot of hype about “Agentic RAG”, where the agent iteratively searches, reads, and searches again. While powerful, it’s often overkill and slow.
Here’s my rule of thumb:
Use Agentic RAG when the answer requires synthesis across multiple unstructured documents (e.g., “Compare the Q3 strategy of our top three competitors”).
Use Text-to-SQL when you need high-precision facts from structured data (e.g., “How many users in Germany churned in October?”).
Trying to use RAG for structured data is a fool’s errand. Vector embeddings are great for “vibes” and similarity, but they’re terrible for math and precise filtering. If your data is in a table, query the table. Don’t turn your table into a series of text chunks and hope the embedding model finds the right row.
The Enterprise Context Gap: Why Agents Stall in Production
Most AI agents work beautifully in a demo with a curated dataset. Then they hit the enterprise production environment and immediately stall. This isn’t a model problem; it’s a context problem.
Discovery Chaos vs. Access Gridlock
In a large company, information is either impossible to find (Discovery Chaos) or locked behind a permission wall the agent can’t navigate (Access Gridlock).
An agent is only as smart as the data it can access. If your agent has to navigate a labyrinth of SharePoint folders and outdated Confluence pages, it will encounter the same friction a new hire does. The difference is that an agent will encounter that friction 1,000 times a second.
The ‘sales_final_v2’ Problem
Enterprise data is noisy. You’ll find three different versions of a “Pricing Strategy” document, and the one titled Pricing_Final_v2_UPDATED_MARCH.pdf is actually the wrong one.
When an agent retrieves three conflicting pieces of information, it doesn’t know which one to trust. It’ll either hallucinate a compromise or simply freeze. This is why provenance is non-negotiable. Your context platform must attach metadata to every token:
Who wrote this?
When was it last updated?
What’s the “authority score” of this source?
If the agent knows that the “Official Handbook” outweighs a “Slack message from 2022”, it can resolve conflicts autonomously.
Myths vs. Reality: Context Management Edition
Myth: “Large context windows eliminate the need for RAG”.
Reality: Absolutely not. Large windows are great for processing a single large document, but they’re terrible for navigating a massive knowledge base. Noise dilution is real. The more irrelevant data you cram into a window, the more likely the model is to miss the critical detail. RAG isn’t just about finding data; it’s about filtering it.
Myth: “Stateless agents are easier to scale”.
Reality: This is the “Dumber Agent Paradox”. While stateless agents are easier to deploy (no session management), they’re significantly “dumber” because they’ve to re-reason through the entire history every time. You end up spending more on tokens and latency because the agent is constantly rediscovering things it should already know. It’s a waste of money.
Myth: “More data equals more intelligence”.
Reality: In the world of context, less is more. Precision beats recall every time. I’d rather give an agent three perfectly relevant sentences than a 50-page document that might contain the answer.
The Path Forward: The Context OS
We’re reaching the limits of what we can achieve by simply making models “smarter”. The next leap in AI won’t come from a larger parameter count or a better training set. It will come from the emergence of a Context OS.
Think of an operating system that sits between the LLM and the world. This OS manages the world-state, handles the “biological decay” of memory, manages permissions, and curates the high-signal token stream in real-time.
In this paradigm, the LLM becomes a stateless CPU, a reasoning engine that’s swapped in and out, while the Context OS maintains the persistent state of the agent’s world. We will stop talking about “prompts” and start talking about “state synchronization”.
If you want to build reliable agents today, stop obsessing over the model. Start building the infrastructure that feeds it.