Imagine this
You’ve been sent into the Valley of the Kings 🐫. A tomb has just been opened. On the outer sarcophagus, three royal names are carved side by side: Yann LeCun, Geoffrey Hinton, and Yoshua Bengio.
The three jointly received the 2018 Turing Award, and from the outside, the story seems straightforward: these are the great pharaohs who helped lead us toward the AI revolution we are living through today.
But the names on the lid do not tell the whole story. To understand who actually built the ideas behind modern Large Language Models, we need to go deeper, paper by paper, breakthrough by breakthrough.
Think of each paper as an ancient Egyptian cartouche: an inscription carrying the names of the researchers behind one specific discovery. As we move deeper into the tomb, we will see which names keep appearing, which disappear, and, more importantly, what each of them actually contributed.
My name is Yassine, and today we’re going through ten papers published between 1986 and 2022 that helped build the foundations of the Large Language Models we use today. Grab a torch 🔦 !
Three Names on the Sarcophagus Lid
Before we start opening the cartouches, let’s first meet the three pharaohs whose names are carved on the lid 🏺.
Geoffrey Hinton
Geoffrey Hinton is a British Canadian computer scientist and one of the central figures in the development of modern neural networks. After studying psychology at Cambridge and completing a PhD in AI at the University of Edinburgh, much of his research focused on a fundamental question: how can neural networks learn useful representations from data?
His work became closely associated with backpropagation, representation learning and, later, the resurgence of deep learning. His group also played a major role in AlexNet, the 2012 breakthrough that dramatically improved image classification on ImageNet (worldwide competition to classify images into 1,000 categories).
One paper matters particularly for our expedition: Learning Representations by Back Propagating Errors, published in 1986 with David Rumelhart and Ronald Williams. It will be the first cartouche we are going to open.
Yoshua Bengio
Yoshua Bengio is a Canadian computer scientist whose research sits especially close to the lineage of modern language models. After completing his PhD at McGill University, he joined Université de Montréal and built the research group that would eventually become Mila (Montreal Institute for Learning Algorithms, one of the world’s leading deep learning research labs).
His work spans representation learning, neural language models and sequence modeling. One of his major contributions was helping move NLP away from treating words only as isolated discrete symbols, toward learned vector representations, where relationships between words can be captured mathematically.
Early neural language models were already built around a familiar objective: predict the next word from the previous ones. Modern LLMs extend the same principle to tokens: given a context, predict what comes next.
Yann LeCun
Yann LeCun is a French computer scientist best known for convolutional neural networks and computer vision.
After studying electrical engineering at ESIEE Paris and completing a PhD in computer science, he developed LeNet, one of the early successful CNNs used for handwritten character recognition (think MNIST dataset dear students).
His main territory has always been computer vision: teaching machines to learn useful representations directly from images. Today, that research has evolved toward World Models, with the broader goal of building systems that can understand and predict how the world works.
Part I – Building the Core Machinery
So now that we’ve met the three great pharaohs of this era, as mentioned in the introduction, it’s important to understand that they are not the same, and that today’s GenAI was not built in one step but paper by paper, across decades. Each paper we open is a beautiful cartouche.
I’m not going to unpack every equation or implementation detail. Some of these papers assume a solid ML background anyway. Instead, for each one, we’ll focus on three things: Why did this paper appear at that moment? What problem did it solve? And what did it make possible later?
Because foundational research rarely knows where it will lead. The people experimenting with electricity were not thinking about transistors, GPUs or GPT. These researchers were solving very specific problems. Decades later, those solutions ended up fitting together.
The first five cartouches build the core machinery behind modern language models:
Learning → Memory → Sequence generation → Attention → Transformer.
1️⃣ Learning Representations by Back-propagating Errors (Rumelhart et al., 1986)
Why now? Multilayer neural networks could represent complex functions, but training their hidden layers was difficult. There was no widely adopted, efficient way to determine how each internal weight contributed to the final error.
What does the paper do? It popularizes backpropagation for multilayer networks: compute the error at the output, propagate its gradient backward through each layer using the chain rule, then adjust the weights to reduce that error.
What does it unlock? Neural networks can learn useful internal representations across multiple layers. Backpropagation becomes the core training machinery behind most modern neural network architectures (CNN, RNN, LSTM, GRU, etc).
2️⃣ Long Short-Term Memory (Hochreiter & Schmidhuber, 1997)
Why now? Recurrent neural networks could process sequences, but they struggled to learn relationships separated by many time steps because gradients tended to vanish or become unstable (think vanishing 🫠 or exploding 💣 gradients of RNN).
What does the paper do? It introduces LSTM, a recurrent architecture with a dedicated memory mechanism designed to preserve useful information and gradients over long sequences.
What does it unlock? Neural networks become much better at learning long-range dependencies, making recurrent models practical for language modeling, speech and translation.
3️⃣ Sequence to Sequence Learning with Neural Networks (Sutskever et al., 2014)
Why now? Neural networks could process sequences, but tasks like translation require mapping an input of arbitrary length into an output of another arbitrary length.
What does the paper do? It introduces an encoder-decoder architecture using two LSTMs. One reads the input sequence and compresses it into a representation, the other uses that representation to generate the output sequence.
What does it unlock? End to end neural machine translation becomes viable. But the whole input sequence has to be compressed into one fixed size vector: as sentences get longer, more and more information must compete for the same limited representation space, creating a major information bottleneck.
4️⃣ Neural Machine Translation by Jointly Learning to Align and Translate (Bahdanau et al., 2014)
Why now? Seq2Seq works, but its fixed-size representation becomes increasingly restrictive as sentences get longer.
What does the paper do? It introduces attention. Instead of relying on one compressed vector, the decoder can look back at all the encoder states and assign more weight to the parts of the input that are most relevant to the word it is currently generating.
What does it unlock? Models no longer need to compress everything into a single vector. They can dynamically retrieve the information they need from different parts of the input.
5️⃣ Attention Is All You Need (Vaswani et al., 2017)
Why now? Attention had already improved recurrent models, but RNNs and LSTMs still processed sequences step by step. That dependency made training hard to parallelize and increasingly expensive at scale.
What does the paper do? It removes recurrence entirely and introduces the Transformer. Instead of passing information sequentially from one token to the next, each token can directly attend to other tokens through self attention.
For every token (roughly, a word for non technical readers), the Transformer creates three vectors: a Query, a Key and a Value. The intuition is simple: Query = what am I looking for? Key = what information do I contain? Value = what information can I pass forward?
Take the sentence: Harry Potter has a brown wand. When processing “wand”, the model compares the Query of wand with the Keys of all the other tokens. These comparisons produce attention scores, which determine which tokens matter most for wand. If “brown” gets a high score, the model gives more weight to its Value, so the information carried by brown contributes more strongly to the new representation of wand.
What does it unlock? Tokens no longer need to wait for the previous token to be processed. Much of the sequence can be computed in parallel, making training dramatically more scalable across data, compute and model size.
STOP: The Architecture Is Here 🚀
By 2017, the turning point was here. The essential machinery was now on the table: neural network, backpropagation, sequence modeling, attention and the Transformer.
At the same time, a new generation of AI labs, including the recently founded OpenAI, was beginning to push a different question: not just how to design better architectures, but how far these architectures could go when trained at much larger scale.
The next shift was less about inventing a completely new architecture and more about changing the recipe: take the Transformer, pretrain it on massive amounts of text, then keep scaling data, compute and model size.
That is where modern LLMs really start to take shape. 🚀
Part II, From Transformers to Modern LLMs
6️⃣ Improving Language Understanding by Generative Pre-Training (Radford et al., 2018)
Why now? Huge quantities of text exist without labels. The question is whether a model can learn general language representations directly from that text before being adapted to a specific task.
What does the paper do? It pre-trains a Transformer autoregressively, predicting the next token, then fine-tunes the model on downstream tasks.
What does it unlock? The first GPT establishes the basic generative recipe behind modern LLMs: learn from large amounts of raw text first, then adapt the model afterward.
7️⃣ BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (Devlin et al., 2018)
Why now? The Transformer provides a powerful architecture, but NLP systems still rely heavily on models trained or adapted separately for individual tasks.
What does the paper do? BERT pre-trains a Transformer encoder on large amounts of text using masked language modeling, allowing each token representation to use context from both directions.
What does it unlock? One pretrained model can be adapted to many NLP tasks. Large-scale pre-training becomes a central paradigm in language modeling.
8️⃣ Language Models are Unsupervised Multitask Learners (Radford et al., 2019)
Why now? GPT showed that generative pre-training transfers across tasks. The next question is whether scaling the same recipe can reduce the need for task-specific fine-tuning.
What does the paper do? GPT-2 scales the model and its training data and demonstrates zero-shot behavior on several tasks expressed directly as text.
What does it unlock? A language model starts looking less like one NLP tool and more like a general-purpose system. Scaling itself becomes part of the recipe.
9️⃣ Language Models are Few-Shot Learners (Brown et al., 2020)
Why now? GPT-2 suggests that broader capabilities appear as language models grow. The next question is how far that trend can go.
What does the paper do? GPT-3 scales the autoregressive Transformer to 175 billion parameters and performs tasks from instructions or examples placed directly in the prompt, without updating its weights.
What does it unlock? In-context learning. The prompt itself becomes a way to adapt model behavior without retraining, establishing the interaction model still used with LLMs today.
🔟 Training Language Models to Follow Instructions with Human Feedback (Ouyang et al., 2022)
Why now? GPT 3 can generate remarkably fluent text, but next token prediction alone does not teach a model to understand or follow a user’s intent. Worse, because it is trained on massive amounts of raw internet text, it can also reproduce undesirable patterns from that data and respond in ways that are irrelevant, unsafe or simply inappropriate for the user’s request.
What does the paper do? InstructGPT adds a human alignment layer on top of pretraining. The model first learns from human written demonstrations, then humans tell the system which answers they prefer by ranking several candidate responses. A reward model learns those preferences, and the model is then fine tuned with reinforcement learning from human feedback, RLHF, to produce answers that better match user intent.
What does it unlock? The modern conversational LLM: not just a model that generates text, but one trained to behave like an assistant.
To wrap up
To wrap up our article 🏺, here is a visual recap of the 10 papers that progressively built the machinery behind today’s LLMs. Each one solved a very specific problem, but together they form a surprisingly coherent chain !
