The Story of LLMs: From a Stochastic Parrot to an Autonomous Assistant

Let’s me tell you a story

My fellow AI engineers 👋 – or whatever we’re calling ourselves these days. We went from Data Miner, to Data Scientist, to AI Engineer so quickly that it’s becoming hard to describe what we actually are.

Let me tell you a story. The first time I saw an LLM – GPT-2, to be precise – I really wasn’t that impressed. I could see it was a meaningful step forward, but I thought it was just another NLP trend. We had already gone from Seq2Seq models for machine translation to increasingly capable autocomplete systems, so this felt like the next incremental step.

Then I changed my mind 🤔. GPT-3 was the real turning point. Not because it was flawless – it wasn’t – but because it fundamentally changed what a language model could be. Instead of training a dedicated model for every task, you could simply describe what you wanted in natural language. GPT-4 pushed this even further with a dramatic leap in reasoning and reliability. Then came GPT-5 with its million-token context window, making it practical to work over entire codebases, books, or complex technical documentation in a single conversation. Since late 2025, I don’t think I’ve written a single line of code entirely by myself – and I don’t think I’m an exception anymore.

But here’s the question nobody really answers: how did we go from a stochastic token predictor to a tool that Terence Tao (maths genius) uses to do mathematics?

---

First of all: What is an LLM, really?

Let’s go back to basics. Strip away all the hype, and an LLM is just this: f(text) → text

That’s it. A function that takes text as input and returns text as output. Under the hood, it’s predicting the next token, one at a time, autoregressively. GPT-2 did this. GPT-4o does this. Claude does this. The fundamental mechanism hasn’t changed. See this article for more details about LLM internals.

"The cat sat on the" → couch: 42%, floor: 28%, car: 8%...
                               ↓
                            "couch"

One token is selected, appended to the context, and the process repeats. Token after token, this simple loop generates conversations, code, advices, and everything else we ask it to produce.

---

Underlying #0 – Scale, baby, scale

A certain orange-haired 🍊 president once said, “Drill, baby, drill.” In AI, the equivalent isn’t oil, it’s compute and data. Every new generation of LLMs has been trained with more GPUs, more parameters, and better optimization than the previous one.

GPT-2 had 1.5 billion parameters. GPT-3 jumped to 175 billion: a 100x leap in a single generation. Frontier proprietary models like GPT-4 or Claude no longer disclose their parameter count. But open models give us a sense of the scale: Qwen3 reached 235 billion parameters, and Kimi K2 1 trillion – and those are the ones we can actually verify.

What surprised researchers wasn’t that bigger models performed better. It was how predictably they improved. In 2020, OpenAI showed that language models obey remarkably simple scaling laws: as model size, data, and compute increase together, the training loss decreases according to power laws. Even more surprisingly, entirely new capabilities emerge as a consequence of this scaling. Models don’t simply become better at predicting the next word. They become surprisingly capable of translation, coding, mathematical reasoning, and scientific writing.

Why does this happen? The honest answer is that we still don’t fully know. 🤯 Scaling laws are one of the strongest empirical results in modern deep learning, but there is still no accepted theoretical explanation for why they hold so consistently. The important takeaway is simple: scaling works. And for nearly a decade, every generation of frontier models has largely been the result of pushing the same recipe further than anyone else.

---

Ability #1 – Remember

The context window

Before we talk about agents, tools, and shiny things, we need to talk about something more fundamental: how much text can the function actually see at once?

Early models had a context window of 4,000 tokens – roughly 3 pages of text. That’s not a lot. It shaped everything about how we built systems around LLMs. You couldn’t feed a full document. You couldn’t keep a long conversation. You had to chunk, truncate, summarize, and stitch – all by hand.

# Before: the chunking nightmare
def process_document(doc, max_tokens=3000):
    chunks = split_into_chunks(doc, size=max_tokens)
    summaries = []
    for chunk in chunks:
        summary = llm.complete("Summarize: " + chunk)
        summaries.append(summary)
    # Then summarize the summaries... and hope you didn't lose anything important haha
    return llm.complete("Combine summaries: " + "\n".join(summaries))

This wasn’t just inconvenient – it was architecturally constraining 🚧. The entire RAG ecosystem (embeddings, vector databases, retrieval pipelines) was built largely as a workaround for small context windows. If you could fit everything in context, you wouldn’t need to retrieve.

Now, context windows have gone from 4k to 128k to 1M tokens. One million tokens is roughly 750,000 words (or full 750 pages of Letter/A4): an entire codebase, a year of Slack messages, or a full legal contract corpus.

Why does this matter? Because an LLM can only reason over what it can see. Increasing the context window doesn’t make the model inherently smarter – it gives it more information to think with. 🧠 Instead of asking “What is the answer based on this paragraph?”, we can now ask “What is the answer based on the entire repository?” The quality of the reasoning often improves simply because the model no longer has to guess what happened outside its field of view.

This also changes how we design AI systems. Rather than spending engineering effort deciding what to retrieve, summarize, or discard, we can increasingly provide the raw source material directly. In many workflows, context engineering is replacing prompt engineering.

Conversation history

Even with a large context window, maintaining a conversation across turns used to be your problem. You serialized the history, you managed the size, you sent it all back on every call – again and again 🔁 – manually.

# Before: manual context management
history = []
def chat(user_message):
    history.append({"role": "user", "content": user_message})
    response = llm.complete(messages=history)  # resend everything, every time
    history.append(response.message)
    return response.text
# If history exceeded the context window: good luck 🤞

Now, providers handle this with a thread or response ID. You pass a reference, not the full history.

# Now: stateful by reference
r1 = llm.complete(input="Hi, my name is Yassine")
r2 = llm.complete(input="Do you remember my name?", previous_id=r1.id)
# That's it.

This is a significant improvement because you no longer need to manage conversation state yourself. Instead of repeatedly serializing, transmitting, and deserializing the entire history, you simply reference the previous response. This reduces implementation complexity, network overhead, and latency.

RAG – long-term memory

The context window, even at 1M tokens, has limits. For large, dynamic, or private knowledge bases – internal docs, legal contracts, product catalogs – you need to retrieve before you generate.

Before, that meant building the entire pipeline yourself: chunk documents, embed them, store vectors, retrieve on query, inject into context. Hundreds of lines of infrastructure for what is conceptually a simple idea.

# Before: the full RAG stack, hand-rolled
chunks = split(document)
embeddings = embed_model.encode(chunks)
vector_db.store(embeddings)

# At query time
relevant = vector_db.search(embed_model.encode(query), top_k=5)
answer = llm.complete(context=relevant, question=query)

Now, providers expose this as a native tool.

# Now: RAG as a native capability
kb = llm.create_knowledge_base(files=["contracts.pdf", "policies.pdf"])
answer = llm.complete(
    input="What does the contract say about termination?",
    tools=[file_search(kb)]
)

---

Ability #2 – Speak Precisely

The base function of an LLM is simple: it generates text. The problem is that this text is inherently unstructured and probabilistic. That’s ideal for conversations, but much less so for software systems.

If you want to extract structured data from a document, insert records into a database, or pass the output to another service, you need a format that can be parsed deterministically. Reliability is the key requirement.

Before – extracting structured data meant praying:

# Before: asking the model to behave
response = llm.complete(
'Extract info as JSON: Alice, 32, admin. Respond ONLY with valid JSON.' )
try:
    data = json.loads(response.text)
except json.JSONDecodeError:
    # Was it wrapped in ```json? Did it add a comment? A missing comma?
    # Start your retry logic here, gooood luck haha, etc..
    ...

Nowadays – you define a schema. The provider enforces it at generation time.

# Now: schema-enforced output
class UserSchema(BaseModel):
    name: str
    age: int
    role: str  # "admin" | "user" | "guest"

response = llm.parse(input="Extract: Alice, 32, admin", schema=UserSchema)
# Guaranteed: correct types, no missing fields, no malformed JSON
user = response.parsed  # ready to use

What’s actually happening under the hood? The provider converts your schema into a grammar, then applies constrained decoding. At each generation step, instead of sampling freely from 100,000 tokens, the model can only pick tokens that are valid given the current state of the schema. This runs at the logits level, before sampling.

If the model just generated {"age":, the only valid next tokens are digits. It is structurally impossible to output {"age": "thirty-two"}. There is no post-hoc validation, this a hard constraint during generation.

---

Ability #3 – Act

So far, our LLM can reason, remember, and respond precisely. But there’s one fundamental limitation left: the model is blind to everything outside its context window – and passive. No live data, no external APIs, no actions. It can’t check the weather, query a database, or trigger a deployment. If it wasn’t in the training data or the prompt, it simply doesn’t exist to it.

Tool calling broke both walls at once 👊.

Declaring a tool

Before, giving the model access to an external function meant a prompt hack and a lot of faith:

# Before: prompt-based function calling
prompt = """
If you need the weather, write: CALL_WEATHER(city="CITY_NAME")
Otherwise respond normally.
"""
# Then parse the text output with regex
match = re.search(r'CALL_WEATHER\(city="(.+?)"\)', response.text)
if match:
    result = get_weather(match.group(1))
    # re-inject and call again...

Now, you declare tools as schemas. The model returns a structured call so no parsing, no regex and no prayers haha 🙏

# Now: structured tool declaration
tools = [{
  "name": "get_weather",
  "description": "Returns current weather for a city", # Prompt engineering
  "parameters": {"city": {"type": "string"}}
}]

response = llm.complete(messages=msgs, tools=tools)

if response.tool_calls:
    call = response.tool_calls[0]
    # call.name == "get_weather"
    # call.arguments == {"city": "Paris"}
    # In the runtime, you are responsible for DOING the tool call.
else:
    print(response.text)  # Model chose to answer directly

Under the hood? Three mechanisms, none of them fully documented by providers:

1. Context injection: The tool definition is serialized and injected into the system prompt as text. The model reads it like any other instruction. This is why description matters as much as the schema – it’s what the model uses to decide whether to call the tool.
2. Fine-tuning: Models are trained on millions of tool calling examples with specific output formats, likely using special tokens (<|tool_call|>, <function_calls>, etc.) per provider.
3. Constrained decoding: Once the model decides to call a tool, arguments are generated under schema constraints, same as Ability #2.

Native provider tools

Some tools are so common – web search, code execution, document retrieval – that providers built them in directly. You activate them, they handle the loop.

# Before: rolling your own web search
results = bing_api.search(query)
context = format_results(results)
answer = llm.complete(f"Context:\n{context}\n\nQuestion: {query}")

# Now: one line
answer = llm.complete(input=query, tools=[web_search()])

MCP: The standard for tool discovery

Before MCP, every tool integration was bespoke🤵‍♂️. Each application had to define its own tool schemas, connection logic, authentication, error handling, and execution layer. Integrating the same tool across multiple projects often meant rewriting the same plumbing over and over again.

MCP (Model Context Protocol) standardizes this interface. Instead of hardcoding tool definitions into every application, an MCP server exposes them through a common protocol. Any MCP-compatible client can connect, discover the available tools at runtime, and invoke them without knowing in advance what the server provides.

Think of it as USB for LLM tools, not because it distributes tools, but because it defines a common interface between tool providers and tool consumers

# Before MCP: hardcoded tool declarations per project
tools = [tool_a, tool_b, tool_c]  # written by hand, every time

# Now with MCP: discovered at runtime
tools = mcp_client.list_tools()  # whatever the server exposes today

What’s actually happening under the hood ?

AT STARTUP 🚀

🔌 MCPClient connects → initialize()
    🤝 handshake + transport negotiation (HTTP, SSE or stdio)
    🔍 list_tools() → discover available tools at runtime
    🧰 passed to the LLM as tools=[...], just like any standard tool declaration

AT INFERENCE ⚡

🧠 LLM decides → tool_call("fetch_data", {"id": "123"})
🏃 Runner → MCPClient.call_tool("fetch_data", {"id": "123"})
         📡 JSON-RPC over the negotiated transport
         🛠️ MCP Server executes the tool and returns the result
🏃 Runner → 📥 injects the result into the context
🧠 LLM continues generation

---

Ability #4 – Plan & Execute

The base LLM function answers once and stops. No iteration, no self-correction and no multi-step planning. But real-world problems rarely fit in a single question-response single shot. “Analyze our Q3 sales, cross-reference with market trends, and flag the three biggest risks”, that’s a workflow of complex multiple tasks to do 🔁.

To turn the LLM from a responder into an actor, we need to give it two things: the ability to reason before acting, and the ability to chain that reasoning across multiple steps.

1️⃣ Level 1 – Making the LLM reason inside a single call

The simplest form of reasoning costs you nothing extra: just ask the model to think out loud 🎤.

Chain of Thought (CoT): “Think step by step and gives your final answer at the end” is the classic example for a CoT. The model writes its reasoning as part of its output, it’s just tokens, but structured tokens that force the model to work through the problem before committing to an answer. The reasoning IS the output. You see every step.

User → LLM("think out loud step by step, solve X")
     → "First, I need... Then I should... Therefore the answer is..."
     → User
# 1 call, reasoning are visible in the response.

Extended thinking (Hidden CoT): Some models (o3, Claude, DeepSeek R1) go further. They dedicate an internal token budget to reasoning before producing the final answer. The reasoning is no longer part of the output because it happens upstream, invisibly. Still 1 API call for 1 LLM call. The thinking is just longer, and sometimes exposed.

User → LLM → [internal reasoning tokens, hidden] → final answer → User

# 1 call, the model "thinks" before answering.

The transparency varies wildly by provider:

• Claude: Provides full thinking block that you can browse in a response variable called response.thinking.
• DeepSeek R1: Same philosophy, the reasoning is exposed inside <think>...</think> before the answer.
• OpenAI o3: The reasoning happens, you pay for it, but you only get a summary in a response variable called response.reasoning_summary.

2️⃣ Level 2 – Orchestrating across multiple calls: ReAct

CoT is powerful, but the model is still isolated; it can’t fetch live data mid-reasoning, can’t act on the world. ReAct (Yao et al., 2022), that stands for Reason 🤔 and Act ⚒️, fixes this by interleaving reasoning and acting in a loop:

Reason → Act → Observe → Reason → Act → Observe → ... → Answer

Reason  : LLM thinks about what to do next
Act     : LLM calls a tool (search, database, API...)
Observe : Runner executes the tool, injects the result back
Repeat  : LLM reasons again with new information

Each iteration is a separate LLM call. The LLM doesn’t remember previous steps on its own – the runner maintains state and re-injects history at every call.

And under the hood? Three mechanisms make the ReAct pattern work, although providers only document part of the implementation:

• Prompting: Before the inference, the runner injects a system prompt that teaches the model how to solve the task in a CoT mode: reason step by step and not first-shot, use the available tools whenever they reduce uncertainty, wait for tool results before planning the action, only return a final answer once enough information has been gathered.
• ReAct loop: Each iteration is a brand new inference. The model reasons over the current context, either emits a tool call or a final answer, then stops. If a tool is requested, the runner executes it, injects the observation back into the conversation, and asks the model to reason again from this updated state.
• Stateless: The model has no execution state between calls. The runner recreates it by replaying the system prompt, user request, previous assistant messages, tool calls and tool outputs before every inference. How providers compress or summarize long conversations to fit the context window remains largely undocumented.

ReAct also has a notable variant – Tree of Thoughts 🌳 – where the runner explores multiple reasoning branches in parallel instead of a single linear path, then evaluates and keeps only the most promising ones before continuing.

---

In practice – before and after

Before agent SDKs, you implemented the ReAct loop yourself. Every step, every tool dispatch, every state update – by hand. And it was fragile.

# Before: artisanal ReAct loop
messages = [{"role": "user", "content": user_input}]
while True:
    response = llm.complete(messages=messages)
    messages.append(response.message)
    if "CALL_WEATHER" in response.text:  # parsing text with regex 😬
        result = get_weather(extract_city(response.text))
        messages.append({"role": "user", "content": f"Observation: {result}"})
    else:
        return response.text  # hope it's actually done...
# Reason → Act → Observe → Repeat. All by hand.

Now, the SDK implements the ReAct loop for you. You declare what the agent can do – the runner handles the rest.

# Now: declare intent, let the runner orchestrate the ReAct loop
agent = Agent(
    instructions="You analyze data and answer questions.",
    tools=[get_weather, query_db, search_web]
)
result = agent.run("Analyze last quarter's sales and flag any risks.")

---

Ability #5 – See & Hear 🖼️

The base function speaks one language: text. Everything that wasn’t text had to be converted before the model could touch it. That meant separate models, separate pipelines, and a lot of information loss along the way.

Before  →  f(text) → text
Now     →  f(text | image | audio | document) → text | image | audio

Before, every modality was its own pipeline, and each step was a degradation:

# Before: three pipelines, three failure modes

# Image: OCR → text → LLM
text = ocr.extract(image)
answer = llm.complete("What's in this chart?\n" + text)
# Lost: layout, colors, spatial relationships 📉

# Audio: transcription → text → LLM
text = whisper.transcribe(audio)
answer = llm.complete("Summarize this meeting:\n" + text)
# Lost: tone, speaker identity, hesitations 📉

# Document: text extraction → LLM
text = pdf_parser.extract("insurance_contract.pdf")
# You get: "Dental coverage • • • •"
# Lost: which bullet means "included", which column maps to which tier,
#       the visual hierarchy, the color coding 📉

Now, you pass the raw modality and the model perceives it natively, not through a text proxy:

# Now: one call, native perception
response = llm.complete(
  input=[
    image("chart.png"),  # sees layout, colors, spatial structure
    audio("meeting.mp3"), # hears tone, speaker identity
    document("insurance.pdf"), # reads the viz table, the hierarchy
    ]
)

What’s actually happening under the hood ? Each modality is converted into tokens that the attention mechanism can process alongside text tokens. An image becomes a grid of patch embeddings. Audio becomes a sequence of spectral features. A PDF is rendered visually and processed as an image, the model sees what a human sees, not what a text extractor outputs.

The key insight: the transformer architecture doesn’t care what generated the tokens. Text tokens, image patch tokens, audio tokens; they all flow through the same attention layers.

---

Wrapping up

We started by explaining that every LLM can be reduced to a function as simple as this: f(text) → text, predicting the next token, one token at a time. That’s true for every LLM, regardless of its size, capabilities, or provider.

But look at what we unpacked across these five abilities. Some of them came from the model itself like a bigger context window, constrained decoding, native multimodality, emergent reasoning. The model got genuinely better, and scaling is largely why. Others came from what we built around it: tool calling, RAG pipelines, agent loops, MCP. That’s the harness (the new word of the moment! ): everything in the AI system except the model itself.

And harness engineering is quickly proving to be as important as model engineering. Two teams can deploy the exact same LLM and achieve completely different outcomes depending on how their harness is designed.

So yes, the stochastic parrot is still there 🦜. We finally figured out how to put it to work. 😃