Imagine this

Imagine you have an exam on Harry Potter ⚯ ͛. I’m the professor, and I have just one question for you:Why was Harry able to survive Voldemort’s attack as a baby?

Now imagine three versions of the exam. In the first, you answer from your memory. In the second, you are given the exact chapter containing the relevant information. In the third, you are given all seven Harry Potter books, thousands of pages, and told: the answer is somewhere in there 🙃.

Sooo, which version should be easier? For a human, probably the second. And strangely enough, something similar happens with Large Language Models. Giving a model more information can help, but after a point, more context does not necessarily mean better access to information, better comprehension, or better reasoning.

And that will be our topic today. In this article, we’ll talk about context management, reasoning, agents, and what actually happens when we keep feeding more information to an LLM.

What is context ?

At its core, a Large Language Model can be simplified as a function: f(text_x) → text_y. It receives an input, text_x, and generates text_y autoregressively, one token at a time, until it reaches a stopping condition.

The context window is the maximum amount of information the model can process during one inference. The actual information provided to the model, text_x, is its context. And right now you’re probably thinking: okay… well that doesn’t tell me much 🤔🤔.

But here is the important catch: the context IS NOT just your user prompt. Your user prompt is the message you explicitly type and send. The context is everything the model actually receives when generating its response. When you chit chat with your favorite LLM, your latest message is therefore not sent to the model independently from everything that happened before. Instead, a software layer around the model, often called the harness, assembles text_x that resembles to:

SYSTEM
You are an AI assistant. Follow these rules...

USER
Here is my question...

ASSISTANT
Previous answer...

USER
Follow-up question...

TOOL
Search results / retrieved documents / API output...

ASSISTANT
Generate the next answer.

The exact composition depends on the provider. The harness may keep some previous messages, remove others, summarize older information, retrieve external documents, or inject new information before calling the model. So when you send "Why did Harry survive?", that sentence is your user prompt, while the LLM may actually receive something much closer to: system prompt, conversation history, assistant response, tool outputs and user prompt.

This distinction became increasingly important as models learned to operate over much larger contexts. Today, the impact of frontier models comes not only from their scale or their reasoning capabilities, but also from their ability to understand amounts of context. That translate into understanding long conversations, full documents, codebases and use retrieved knowledge (RAG), etc.

But they also created a new engineering problem: What should we actually put inside that context? And that is where context management starts 🌟.

⚠️
One thing to remember: User prompt ≠ Model context. Keep that in mind, because everything that follows comes from this distinction.

The Context Paradox

So far, everything sounds pretty good. A larger context window means we can give the model more information input. But there is a catch. Being able to fit more information into the context does not necessarily mean that the model will use all of that information equally well.

And this brings us back to our Harry Potter exam. Remember the third version? I give you all seven books and tell you: “the answer is in there, my friend !” Technically, you have everything you need. But is that really better than having the exact chapter? The same thing can happen with LLMs. You may have experienced it yourself after talking with Claude or GPT for a very long time: you keep adding context, instructions, yet at one point the model can become less consistent, miss things you already told it, or produce increasingly wrong answers.

And this is where context engineering starts to become tricky. Whether you are using RAG, full in-context learning, or simply maintaining a long conversation, the model still has to find the right pieces of information, distinguish them from everything else, and decide what to do when several pieces of information compete.

This gives us three of well-known families of context-management problems:

  • 1️⃣ POSITION: Where is the relevant information?
  • 2️⃣ DILUTION: How much noise surrounds it?
  • 3️⃣ CONFLICT: What happens when the knowledge sources contradict each other?
💡
A larger context gives the model access to more information. Context management determines how easy we make that information to actually use.

1️⃣ Lost in the Middle

Let’s start with our first problem: POSITION. Does it actually matter where we put information inside the context?

In 2024, Liu et al. published a paper with a pretty explicit title: Lost in the Middle: How Language Models Use Long Contexts. Their question was simple: if the correct information is already somewhere inside the context, does its position change the probability that the model will actually use it?

To test this, they evaluated several LLMs on two tasks: multi-document question answering and key-value retrieval. In both cases, the relevant information remained available, but its position within the context was systematically changed. This allowed the researchers to measure whether models performed differently when the information appeared at the beginning, in the middle, or at the end of the context.

As you expected, the result was quite surprising ! The performance was often better when the relevant information appeared at the beginning or at the end of the context, and degraded when that same information was placed somewhere in the middle.
This produced the now famous U-shaped curve:

The phenomenon became known as Lost in the Middle, the information never disappeared from the context, only its position changed.

🙋‍♂️ And why does this phenomenon exist? We do not fully know yet. Humans show a similar serial position effect: we tend to remember the beginning and the end of a sequence better than its middle. Our writing often follows the same pattern, placing key ideas in introductions, abstracts, summaries, and conclusions. Since LLMs are trained largely on human-written text, it is tempting to connect the two. But this is only an analogy, not a scientific explanation of the phenomenon. Some research suggests that positional biases in how transformer models process long contexts may contribute to the effect.

The practical takeaway is simple: do not bury critical information in the middle of a huge context. Make it prominent, and reinforce it near the beginning or at the end, or close to where it needs to be applied.

💡
Presence ≠ Accessibility. Information can be inside the context and still be harder for the model to access depending on where it is placed.

2️⃣ Signal dilution

Our second problem is DILUTION. This time, the relevant information does not move. Instead, we progressively add more and more context around it. The information is still there, but the haystack around our needle keeps growing.

And when we talk about dilution, there are actually two different kinds of signal we care about: information and instructions.

Information Dilution 🫠

Can the model still identify the right information among everything else?

A classic way to test this is the Needle in a Haystack benchmark, popularized by Greg Kamradt. The idea is simple: insert a random fact, the needle, inside a long unrelated document, the haystack, then ask the model to retrieve it.

The test varies two parameters: context length and needle position. The resulting heatmap shows where retrieval succeeds or fails. And as we can see above, two models can have very different effective long-context capabilities even on exactly the same task.

Figure Ré)écrn°8 of DeepSeek-V3 Technical Report (DeepSeek Research)


In this example, GPT-4’s retrieval performance degrades as context length increases, whereas DeepSeek-V3 remains stable across the full 128K context, likely due to better long-context training and architectural optimizations.

The thing is that traditional NIAH has one big weakness: the question and the needle usually share the same words. Finding the answer to “What is the special magic number for San Francisco?” is much easier when the context literally contains “The special magic number for San Francisco is 93847.”

This is what NoLiMa, Long-Context Evaluation Beyond Literal Matching, tested at ICML 2025. NoLiMa removes this lexical shortcut: the model must understand the semantic relationship between the question and the relevant information. Thus, the performance drops much faster. Among 13 models advertising context windows of at least 128K tokens, 11 fell below 50% of their short-context baseline at only 32K tokens. GPT-4o dropped from 99.3% to 69.7%.

Table n°3 of NoLiMa: Long-Context Evaluation Beyond Literal Matching (Modarressi et al.)
✔️
So context-window size tells you how much can fit, not how much can be reliably retrieved. The larger the haystack, the more important it becomes to control what you put inside it.

Instruction Decay 🥀

But data is not the only signal inside a context. Instructions can get diluted too.

Imagine starting a conversation with: “Always answer using bullet points” / “Never reveal this information.” Then you keep chatting, adding documents and using tool calls. Twenty or fifty turns later 🔄, the instruction may still be somewhere in the context, but the model may simply stop following it as reliably.

And this is not just something you sometimes feel when using ChatGPT. In 2026, Google researchers studied exactly this phenomenon in We Are What We Repeatedly Do: Improving Long Context Instruction Following (Robinette et al., 2026).

They introduced VerIFY, a benchmark that keeps an instruction active throughout a multi-turn conversation and checks whether the model still follows it as more turns are added. As the conversation gets longer, the instruction compliance tends to decrease.
Here are the result from the paper:

Figure n°1 of We Are What We Repeatedly Do:
Improving Long Context Instruction Following (Google DeepMind)

So signal dilution is not only about finding the right data. It is also about keeping the right instructions influential enough to guide the answer.

In practice, LLMs can gradually lose track of instructions buried in a long context, even when those instructions were clear at the start. If a rule really matters, repeat it when it becomes relevant and keep it explicit and easy to notice, rather than assuming that mentioning it once will be enough.

💡
The objective is not to minimize context. It is to maximize useful signal while minimizing retrieval burden. For data: give the model the information it actually needs. For instructions: keep the important rules explicit when they need to be applied.

3️⃣ Knowledge conflict

Our last problem is different: the model have two sources of information:

  • The first is its parametric knowledge: information learned during training and encoded in the model’s parameters.
  • The second is the context knowledge: what is provided at inference time (your prompt, tool calls, system prompt, etc).

I tried to create a useful mental model below of how these two sources can interact. A and B represent different answers, while ∅ means that a source does not contain the relevant information.

But there is one important catch: we can inspect and largely control the context we provide, but we cannot directly inspect the model’s parametric knowledge. It is a latent prior, not a SQL database we can query before every prompt.

The interesting case is when the two disagree. Imagine a model that learned that an API uses /v2, while the latest documentation retrieved in its context says /v3: this is a context-memory conflict, one of the main categories formalized in Knowledge Conflicts for LLMs: A Survey. And the context does not automatically win: in Context-faithful Prompting for Large Language Models, researchers show that models can favor their parametric knowledge over conflicting contextual evidence, and that how the context is presented can influence this arbitration.

Could we simply build an LLM that knows how to reason but has no factual knowledge of its own? Researchers are trying to separate these two capabilities. In this NAACL 2025 paper, all the knowledge required to solve the task is provided externally so that reasoning can be studied independently from factual recall. More recent work such as SynthWorlds goes further by creating synthetic worlds where memorized real-world knowledge is deliberately useless. But with current LLM training, parametric knowledge is difficult to avoid: learning language from massive corpora also means learning factual and semantic regularities from those corpora.

That’s the part we can’t directly control: the model comes with its own parametric knowledge. We can’t rewrite that at inference time, but we can make the contextual knowledge easier to trust and prioritize by making its authority, freshness, provenance, and internal consistency explicit.

✔️
So instead of giving the model a naked fact like “the API uses /v3”, give it qualified evidence: “According to the official API documentation updated in August 2026, /v2 is deprecated and /v3 is the current endpoint, this what need to be used.”

Reasoning LLM

We just spent an entire chapter on context and how to manage it. But giving the model the right information is only half the job. It still has to reason over that information properly. And even with everything it needs in front of it, it can still go dumb. That’s where structure and enough computation matter: they help turn the context into the right answer.

We’ll focus on three things in this chapter: build on what we just learned to give the model the right context, structure its reasoning and understand its power and limits.

1️⃣ Give it the right context

As we just saw, a model can only reason from the information available in its context, but more context is not automatically better.

A useful way to think about context quality is as a fraction:

[Context Quality∝Useful SignalRetrieval Burden][ \text{Context Quality} \propto \frac{\text{Useful Signal}}{\text{Retrieval Burden}} ]

The numerator is everything that genuinely helps solve the task: relevant facts, constraints or examples. The denominator is everything the model has to search through unnecessarily: irrelevant/redundant informations, noisy tool outputs, etc.

So your objective is neither maximum context nor minimum context. It to give your model the right context! Enough information to solve the problem, with as little unnecessary retrieval burden as possible.

💡
Reasoning starts with evidence. Maximize useful signal and minimize retrieval burden: no amount of reasoning can reliably recover missing information, and reasoning over the wrong evidence only produces a more elaborate wrong answer.

2️⃣ Structure its reasoning and let it talk

You know that LLM is autoregressive: each generated token becomes part of the context used to generate the next one. For complex tasks, this means that intermediate steps can help the model build toward a better final decision instead of committing to an answer too early.

So NEVER just ask for the conclusion at the beginning. Give the model a structure to follow: a checklist, intermediate questions, validation steps, or a sequence of criteria to evaluate. Then ask for the final decision after those steps have been processed, let’s take a real example: 

SYSTEM
You are reviewing an insurance claim.
Do not give the final decision immediately.
Evaluate each criterion in order, record your conclusion for each one,
then provide the final decision only after all checks are complete.

USER
Review this claim using the following checklist:

1. Is the policy active on the date of the incident?
2. Is the incident covered by the contract?
3. Are any exclusions applicable?
4. Are the declared dates consistent?
5. Are the claimed amounts within the contractual limits?

For each check, answer PASS / FAIL / UNCERTAIN with one short justification.

Only after completing all five checks, give the final decision:
APPROVE / REJECT / NEEDS REVIEW.

This is the intuition behind approaches such as Chain-of-Thought Prompting: complex problems can benefit from explicit intermediate structure before producing the final answer.

But don’t confuse reasoning with verbosity. The objective is not to make the model talk more, but to give it useful intermediate computation before it commits to a conclusion.

✔️
For complex decisions: structure first, reason second, conclude last. Give the model a path to follow instead of asking it to jump directly to the answer.

3️⃣ Understand its power and its limits

We just saw how to give an LLM the right context and structure its reasoning. But before asking it to reason further, we need to understand what an LLM is actually good at, and where its limits begin.


🍓 Remember the famous question: “How many r’s are there in strawberry?” Older models get it surprisingly wrong. Why? Because a standalone LLM is basically an arts major, not a STEM major: great at understanding and generating language, much less reliable when asked to perform precise operations. It’s not a calculator, a database, or a deterministic program. Text is processed as tokens, so operations that look trivial to us, mortals, are not necessarily natural for an LLM.

But you might say: my favorite LLM can do this today, so what changed? Better training and better reasoning help, but the biggest shift is tool use: modern LLM systems can offload the exact part of the task instead of forcing the model to do everything itself. And that distinction matters: understanding what needs to be done and executing it reliably are not the same capability.

This map gives us a pretty good sense of where raw LLM reasoning works well, and where it starts to break down:

⚠️
An LLM is extremely powerful at understanding and reasoning over language. But for exact and deterministic execution, relying on a plain LLM alone is often not enough. This is exactly what we will overcome next with agentic systems.

Agentic Systems

Until now, we have improved our little monster LLM 👾 in two directions. First, the context management: give it the right information, avoid burying the signal, keep important instructions visible and manage knowledge conflicts. Then, the reasoning: structure the task, let the model intermediate steps before deciding, and understand where plain LLM is unreliable.

We are already far beyond the simple f(text_x) → text_y we started with 😆. But notice what has not changed: everything still happens inside a single call. One context goes in, one answer comes out. Our model reads better and thinks better, but it still gets exactly one shot at the problem.

Getting past that limit means moving beyond a single LLM call. To make the distinction clear, we’ll introduce three levels: standalone LLMs, augmented LLMs, and workflows or agentic systems.

1️⃣ Standalone LLM: reasoning is not augmentation

A standalone LLM is the simplest setup: give the model a context, let it reason, get an answer. It can use Chain of Thought or hidden reasoning, but it still has one shot. It cannot look something up, run something, check the result, or come back with new information.

⚠️
The limitation is not reasoning itself, it is being closed inside one context. Augmentation gives the model a way out.

2️⃣ Augmented LLM: reaching outside the model

An augmented LLM is a standalone LLM that can reach outside its current context. Instead of being stuck with what it was given at the start, it can bring new information in or trigger actions through the surrounding system.

  • Tool calling: instead of approximating an operation, the system delegates it to a tool (calculator, DB, API, Python scripts, etc.), and the result comes back as new context.
  • Retrieval: whether you use full context or RAG, the idea is the same: external information is brought into the model’s context at inference time.
  • External memory: useful information persists outside the current context window and can be brought back later (think about your SKILLS.md).

Just one thing… the model never executes anything. It cannot do an HTTP call, query your warehouse or run Python. What it produces is still text: a structured intention that says call check_policy_status with policy_id = 4471. The harness, which is the same runtime layer that assembled text_x back in the first chapter: reads that intention, actually executes it, and injects the result back into the context as a new TOOL message.

💡
We covered the harness and theses mechanisms in more detail in The Story of LLMs: From a Stochastic Parrot to an Autonomous Assistant.

3️⃣ Agentic systems

Our augmented LLM can now reason, retrieve, and act. But there is still something missing: who decides what to do after each call?

  • If you are the developer, you can decide everything upfront. The model calls a tool, then your code sends the result somewhere else, then another step runs. The model may be smart inside each step, but it is still following your path. You just created a workflow !
  • Or you can give part of that control back to the model. It calls a tool, sees what came back, and decides for itself: Do I have enough? Should I search again? Try something else? Stop? The harness keeps the loop running, but the model chooses the next move.
✔️
And of course, you rarely have to choose one extreme. You can let the model explore freely while still setting boundaries: maximum five iterations, only these tools, stop when this condition is met. That middle ground is where a lot of modern agentic systems live, and it is very close to the intuition behind ReAct (Reason → Act → Observe → Reason).

Here’s my little infographic summarizing what we’ve been talking about:

Let’s also clears up one common confusion: agentic does not mean multi agent, let’s me explain:

  • A standalone reasoning LLM is not an agent. It can spend a lot of internal computation thinking like through hidden Chain of Thought, but from the outside it is still one call: context in, answer out.
  • One augmented LLM can already be agentic. Give it tools, retrieval or memory, let the harness return the results, and most importantly, let the model decide whether to go around the loop again. You already have an agent.
  • Multi agent comes after that. Instead of letting one agent handle the whole loop, you can split the work across several agents. Maybe the flow is fixed by you, maybe they delegate dynamically, maybe it is somewhere in between. But having several agents can be an interesting architectural choice: it lets you split the problem, isolate contexts, specialize roles, and parallelize parts of the work.

And this is where multi agent becomes especially interesting for us, because it brings us straight back to our Context Paradox. We already saw what happens when one context gets too large: information becomes harder to find, signal gets diluted, and instructions lose influence.

With multiple agents, you do not necessarily need one model to carry everything at once. You can split the problem into smaller and cleaner contexts. Imagine one subagent reading 80,000 tokens of documentation and sending only 300 useful tokens back to the parent. The parent does not need the entire document. It only gets what matters for its own task. But here is another catch (if they were no catch, how can you justify your compensation :-D): The context problem does not disappear, It just moves ! 

Instead of asking only:

What context should I give the LLM?

you now have to ask:

What should each agent receive, what should it keep, and what information should cross into the next context?

The 1 million dollar question 💵

✔️
That is the real engineering problem. Splitting the work is easy. Deciding what should survive between contexts is much harder.

Back to Hogwarts ⚡

Let’s go back to our Harry Potter exam. At the beginning, I gave you all seven books and asked one question. The answer was somewhere in there, but as we saw, having everything available is not the same as using everything effectively.

With an agentic approach, you could do something more natural: create a meta-agent to coordinate the task, then assign one agent = one book. Each one reads its volume, finds the relevant evidence, and sends only the useful bits back to the parent agent.

It’s basically how you would organize the same task with humans. You probably wouldn’t ask one intern to read seven books in one sitting and remember everything. You would split the work, give everyone a clear scope, and collect the useful conclusions. 

✔️
That’s the KEY idea behind agentic systems: decompose the problem, control the context, and orchestrate how useful information flows through the system.

👉 Continue with The Story of LLMs: From a Stochastic Parrot to an Autonomous Assistant to see how Chain of Thought, Tool Calling and ReAct work under the hood.

Categorized in:

Artificial Intelligence, The Fundamentals,

Tagged in:

, , , ,