The History and Evolution of Large Language Models: From ELIZA to AI Agents

Think about the keyboard on your phone and the times when you type something, and it can predict the next word. For example, when you type “see you” and it suggests “tomorrow” just above the keys. This little feature is guessing the next word based on what you have already typed and is actually the same basic job ChatGPT and Claude do today (predicting the next token). 

The difference is everything that happened in between. 

In 1966, a program called ELIZA could take your sentence and turn it back into a question.

In 2026, we can type something like “the checkout tests are failing, please fix them,” and an AI agent can open the code, run the tests, find the bug, and suggest a fix while you go do something else at the same time.

But what surprises me most is how uneven that journey was. 

It took around 50 years to get from ELIZA to decent machine translation, and then only about 5 years to get from GPT-3 to agents that write code by themselves. 

In this article, I will examine the history of large language models, divided into 8 eras. 

Each one is a point where something important changed, whether that was the architecture, the size of the models, how they were trained to behave, or what they were allowed to do:

1. The pre-Transformer world (1950s to 2016)
2. The Transformer revolution (2017)
3. The scaling race (2018 to 2020)
4. Alignment and RLHF (2021 to 2022)
5. Multimodal models and open weights (2023)
6. The reasoning revolution (2024)
7. Open source disrupts the economics (2025)
8. The agentic era (2025 to 2026)

The 8 eras of large language models, from rule-based chatbots to autonomous agents. Image by Author

You do not need a technical background to follow along, but if you do want a quick, detailed refresher on LLMs first, I recommend brushing up with our: What is an LLM? article.

What Is a Large Language Model?

Before we go through the different eras, I want to make sure we are on the same page. 

Let me define what an LLM is:

A large language model (LLM) is a neural network trained on huge amounts of text to predict the next token (subword), which, at a large enough size, lets it answer questions, write code, summarize documents, and follow instructions.

Two words will come up again and again, so let me make sure you have understood them. 

• A token is a small chunk of text (a subword), so for example the word “unbelievable” might be split into three tokens: “un”, “believ” and “able”.
• Parameters are the numbers inside the model that get adjusted during training. I like to think of them as knobs, and so when I say GPT-3 had 175 billion parameters, that means it had 175 billion knobs to tune.

The “large” part is really about scale. 

The model is trained on an enormous amount of text and slowly gets better at guessing what comes next, and it turns out that getting good at this one simple task forces the model to pick up grammar, facts, and even some reasoning along the way.

The most important thing I want you to take away from this section is keeping “predict the next token” in mind, because almost every era below is either a better way of doing it, a bigger version of it, or a way of making it more useful.

Era 1: The Pre-Transformer World (1950s to 2016)

The pre-Transformer era is the long stretch, roughly 60 years, where computers handled language using hand-written rules, statistics, and later recurrent neural networks.

Progress happened, but it was very slow, and almost everything was built for one narrow task. 

If NLP is new to you, then I recommend this article on What is Natural Language Processing? to cover the basics. 

ELIZA and the rule-based years

ELIZA was a chatbot Joseph Weizenbaum wrote at MIT, described in his 1966 paper. 

Its most famous script, DOCTOR, pretended to be a therapist by spotting keywords in a sentence and slotting them into ready-made replies. 

For example, if you typed “My mother does not understand me”, it could answer “Tell me more about your family.”

I would like to state that there was no understanding at all, but rather just pattern matching. 

What I find interesting, however, is how people reacted to it. 

Weizenbaum later wrote that his own secretary asked him to leave the room so she could talk to ELIZA in private, even though she knew exactly how it worked.

A conversation with ELIZA. It only matches keywords, yet it can feel surprisingly human. Image source.

This habit of reading understanding into fluent text became known as the ELIZA effect, and we are still arguing about the same thing with LLMs today.

Through the 1990s and 2000s, rules were swapped with statistics to create more advanced models.

N-gram models looked at the previous few words and counted which word most often came next in their training text. 

That is exactly the phone keyboard idea from the intro of the article, and it powered spell checkers, speech recognition, and the first version of Google Translate in 2006.

Word2Vec: turning words into numbers

Another key result in this era was Word2Vec, which was published by Tomas Mikolov and colleagues at Google in 2013. 

They essentially turned each word into a list of numbers (a vector) based on the words that usually appear around it. 

The key (and now very famous result) was that words used in similar ways end up with similar numbers, such as where king - man + woman lands very close to queen.

In my opinion, this is the most underrated milestone in the whole timeline because every LLM in 2026 still starts by turning tokens into vectors like this. 

This article on Text Embedding goes much deeper into this segment.

RNNs, LSTMs, and the problem with long sentences

A Recurrent Neural Network reads a sentence one word at a time and keeps a running memory, a hidden state, updated after every word. 

I like to think of this as someone reading a long paragraph while trying to hold the whole thing in their head, and by the end, the first few words have mostly faded. 

This is the vanishing gradient problem, and it meant plain RNNs struggled to connect words that were far apart.

The long short-term memory (LSTM) network, from Sepp Hochreiter and Jürgen Schmidhuber in 1997, fixed a lot of this with “gates” that decide what to remember and what to forget.

LSTMs carried NLP for almost 20 years. 

For example, in 2014, Sutskever et al. used 2 LSTMs to translate whole sentences, and Bahdanau et al. added an attention mechanism that let the model look back at specific input words while translating. 

The highlight of this approach was Google Neural Machine Translation in 2016, which stacked 8 LSTM layers on each side and showed just how far it could be pushed.

Why progress was so slow

A very big flaw in the RNN Architecture was the lack of parallelization. 

RNNs have to finish word 9 before it can start word 10, so it can’t take advantage of GPUs, which are built to do thousands of calculations at once. 

On top of that, datasets were small and labeled by hand, and a lot of the work went into designing features for each task individually.

The result was that nothing generalized. 

A sentiment classifier, a translator, and a question-answering system were 3 separate models with 3 separate datasets, and more importantly, what one learned did not help the others.

Era 2: The Transformer Revolution (2017)

This era marks a pivotal shift.

The Transformer, introduced in 2017 by Ashish Vaswani and 7 co-authors at Google in “Attention Is All You Need”, dropped recurrence entirely and relied on self-attention instead. 

Every LLM in this article, from GPT-1 to GPT-6 Astra, is built on top of it.

Most people assume the transformer architecture history starts here, but the attention idea was already around in 2014, bolted onto LSTMs. 

What Google’s team did was ask a bolder question: what if attention is the only thing you need and you throw the recurrent part away?

The original Transformer: an encoder and a decoder built from repeated attention blocks. Image source. 

What self-attention actually does

Self-attention lets every token in a sentence look at every other token at the same time and decide how much each one matters. 

For example, let’s take the sentence “The trophy did not fit in the suitcase because it was too big.” 

To work out that “it” means the trophy and not the suitcase, the model needs to connect words that are far apart, and self-attention gives it a direct line between them.

I like to think of the difference this way: an RNN reads a book one word at a time and hopes it still remembers chapter 1 by chapter 10, whereas a Transformer spreads every page across a table and looks at all of them at once.

And the results were very strong. 

The larger Transformer model set a new record on the WMT 2014 English-to-German benchmark with a BLEU score of 28.4, after training for just 3.5 days on 8 NVIDIA P100 GPUs, a small fraction of what the previous best models cost. 

Transformers vs RNNs

You might be asking:

“Why do Transformers scale when RNNs didn’t?”

The answer is because the whole sentence is processed at once. 

Training a Transformer works well on GPUs because of parallelization, and the direct connections between tokens mean long-range context does not get lost. 

Each layer is also the same simple block repeated, so making a model bigger mostly means stacking more blocks.

The important thing to note from this era is that for the first time, using more compute and more data in a model reliably made it better. 

Therefore, the question shifted from “can we design a better architecture?” to “can we afford a bigger one?” and this question shaped the next 9 years.

Era 3: The Scaling Race (2018 to 2020)

The scaling race era is the 2018-to-2020 stretch when labs discovered that training a Transformer on more text with more parameters made it better at almost every language task. 

The trick that made this possible was pre-training: train one model on a huge pile of unlabeled text first, then adapt it to specific tasks. 

This is also where the history of ChatGPT and GPT models begins.

GPT-1 and BERT: two different bets (2018)

Google and OpenAI each took half of the Transformer (the Encoder and Decoder, respectively). 

OpenAI’s GPT-1 (June 2018) used the decoder half to predict the next word left to right, whereas Google’s BERT (October 2018) used the encoder half and learned by filling in hidden words using context from both sides.

In 2018, BERT looked like the clear winner, pushing the GLUE benchmark score to 80.5, a 7.7-point jump, and Google started using it in Search in 2019 for about 1 in 10 English queries in the US. 

However, it was GPT’s simple next-word approach that grew into chatbots. 

GPT-2: too dangerous to release? (2019)

GPT-2 (February 2019) had 1.5 billion parameters, more than 10 times GPT-1’s size, trained on 40GB of text from 8 million web pages. 

Its famous demo sample was a fake news story about scientists discovering unicorns in the Andes and was coherent for several paragraphs.

What made headlines was that OpenAI initially withheld the full model, worried about mass-produced fake news, releasing it in stages instead. 

Looking back, I think this mattered less for GPT-2 itself and more because it kicked off the debate over how and when to release models, a debate still going on in 2026.

Scaling laws and GPT-3 (2020)

Scaling laws are formulas showing that a language model’s error drops smoothly as you add more parameters, data, and compute. 

Kaplan et al. (2020) at OpenAI found these trends held across more than 7 orders of magnitude, and that fed the scaling hypothesis: keep making models bigger, and they keep getting better.

Loss drops smoothly as compute, data, and model size grow. Image source.  

For the next few years, that bet mostly paid off.

GPT-3 (May 2020) had 175 billion parameters, over 100 times GPT-2, trained on about 300 billion tokens. 

Its big surprise was few-shot learning, where you show the model a few examples inside the prompt, and it figured out the task with no extra training. 

Here’s the actual translation example from the paper, where the model was never trained specifically to translate:

Translate English to French:sea otter => loutre de merpeppermint => menthe poivréecheese =>

Few-shot prompting puts the examples in the prompt instead of retraining the model. Image source.

I would like to mention that this is where prompting actually became a real skill. 

Instead of collecting a labeled dataset and training a new model per task, you could describe the task in plain English and show a couple of examples.

One more result worth knowing, even though it technically landed in 2022: Hoffmann et al. at DeepMind showed that most LLMs at the time were undertrained, meaning they had far more parameters than the data supported. 

Their Chinchilla model, at 70 billion parameters trained on 1.4 trillion tokens, beat both the 280B Gopher and the 175B GPT-3.

The rule of thumb that came out of it, roughly 20 training tokens per parameter, changed how every lab spent its compute budget from then on.

Era 4: Alignment and the RLHF Breakthrough (2021 to 2022)

Let me first define what Alignment means:

Alignment is the work of getting a model to do what the user actually wants, rather than just continuing whatever text it was given.

GPT-3 was capable, but it had been trained to predict internet text, and internet text is not a helpful assistant. 

If you asked it to “explain the moon landing to a 6-year-old in a few sentences”, it would have replied with a list of similar questions, because on the internet, a question like that is often followed by more questions. 

Therefore, we needed a method to solve this problem.

RLHF in plain terms

This is where RLHF comes in. Reinforcement learning from human feedback (RLHF) uses human preferences to teach a model which of its answers are good. 

OpenAI applied it to GPT-3 to build InstructGPT in 3 steps:

• Human labelers write example answers, and the model is fine-tuned on them
• The model writes several answers to the same prompt, and labelers rank them (later, a reward model learns to predict those rankings)
• The language model keeps generating answers while an algorithm called PPO nudges it toward whatever the reward model scores highest.

The three steps of RLHF: supervised fine-tuning, reward modeling and PPO. Image source.

I like to think of this process as a chef. 

First, a chef copies recipes from an experienced chef, then customers taste a few dishes and say which they prefer, and then finally, the chef keeps cooking and adjusting based on what customers seem to like.

The result that stood out to me was that human labelers preferred answers from a 1.3-billion-parameter InstructGPT model over answers from the 175-billion-parameter GPT-3, even though the RLHF-trained model was over 100 times smaller.

If you want to understand this in more depth, I would highly recommend this article on RLHF and its cousin RLAIF, which Anthropic used in its Constitutional AI work in December 2022.

ChatGPT (November 2022)

This is also the era in which ChatGPT was released. 

It was released on November 30, 2022, and was built on a GPT-3.5 model fine-tuned with RLHF for conversation and was launched as a free research preview model.

It crossed 1 million users in about 5 days and reached an estimated 100 million monthly users by January 2023, making it the fastest-growing consumer app ever at the time.

In my honest opinion, ChatGPT is the best example in this whole timeline of a product breakthrough that was really a training breakthrough. 

Most of the raw ability had been sitting there since GPT-3, but RLHF turned a capable text predictor into something anyone could just talk to. 

For practitioners, the gap between “scores well on benchmarks” and “is actually useful” finally closed.

Era 5: Multimodal and the Mainstream (2023)

The next era is all about multi-modality:

• A multimodal LLM takes in more than one type of input, such as text and images (and later audio and video), and reasons across all of them in one model.
• 2023 was also the year LLMs split into 2 avenues: closed models behind an API, and open-weight models you download and run yourself.

GPT-4 (March 2023)

This is also when GPT-4 was released and accepted both text and images, meaning you could show it a whiteboard sketch or a chart and ask questions about it. 

OpenAI reported it scored around the top 10% of test takers on a simulated bar exam, versus the bottom 10% for GPT-3.5.

GPT-4 vs GPT-3 results. Image source.

One thing I would like to mention is that the GPT-4 Report disclosed no parameter count, no architecture details, and no training data description. 

From here on, frontier labs mostly stopped publishing how their models were built due to competition.

Llama and the open-weights movement

On the other side, though, open-weight models were also increasing in popularity. 

Meta released LLaMA in February 2023 across 4 sizes from 7B to 65B parameters, and its 13B version reportedly beat the 175B GPT-3 on most benchmarks while training only on public data.

Now I would like to state that LLaMA 1 wasn’t a GPT-4 competitor, and it was shipped under a research-only license. 

However, the real shift came with Llama 2 in July 2023, which did allow commercial use, followed by Mistral 7B in September 2023 under Apache 2.0.

Fine-tuning also got much cheaper around this time due to new algorithms such as QLoRA, where the authors showed a 65B model could be fine-tuned on a single 48GB GPU.

The rest of 2023 and early 2024 was a race across new input types and longer context windows. 

Anthropic’s Claude 2 (July 2023) could read 100,000 tokens in one go (roughly a full novel). 

Google’s Gemini 1.0 (December 2023) was trained on text, images, audio, and video from the start rather than having vision bolted on. 

Later, OpenAI’s GPT-4o Model could also answer spoken questions at an average of 320 milliseconds, which is close to normal conversation speed.

For practitioners and developers, they had two main choices: either call a closed API for the best quality or self-host an open model for more control, more privacy, and lower cost.

Era 6: The Reasoning Revolution (2024)

This is the era where models really started advancing, mainly due to reasoning models. 

A reasoning model is an LLM trained with reinforcement learning to write out a long chain of thought before answering. 

Essentially, the model would spend extra compute the moment you ask it something, allowing it to think for longer

You can think of it as two students answering a tricky math question. 

The first would just speak out whatever number comes to mind, whereas the second would grab paper, work through it step by step, and check the answer before writing it down. 

Earlier LLMs were the first student and reasoning models are the second.

Chain-of-thought started as a prompting trick. 

Wei et al. (2022) showed that adding worked examples to a prompt improved math and logic scores, and Kojima et al. (2022) found that simply adding “Let’s think step by step” helped too. 

OpenAI’s o1 (September 2024) turned this into an actual training objective, using reinforcement learning to teach the model to produce reasoning steps that lead to correct answers. From this, performance kept increasing with both more training and more thinking time.

o1 vs GPT-4o capability. Image source. 

One thing that really fascinates me in this era was the jump in the American Invitational Mathematics Examination (AIME), a very hard high school math competition. 

GPT-4o solved an average of 12% of AIME 2024 problems compared to o1 being able to solve 74% on a single attempt, and 83% when OpenAI took the most common answer across 64 attempts.

The rest of the industry followed quickly, though most of it landed in early 2025 rather than 2024:

• Google’s Gemini 2.0 Flash Thinking (December 2024)
• Anthropic’s extended thinking in Claude 3.7 Sonnet (February 2025)
• Google making thinking the default in Gemini 2.5 Pro (March 2025)

This created a real trade-off. 

More thinking gave better answers on hard math, coding, and planning problems, but it would cost more and take longer, so using a reasoning model to pull a date out of an invoice is overkill.

Era 7: Open-Source Disrupts the Economics (2025)

This era marks the time when open-source models really start to make a comeback, especially Chinese open-source models.

This era was led by DeepSeek-R1, which matched top-tier reasoning performance for a fraction of the cost, under permissive licenses. 

For me and many others, it broke the assumption that only certain well-funded US labs could build frontier models. 

DeepSeek-R1 (January 2025)

DeepSeek-R1, released by the Chinese lab DeepSeek on January 20, 2025, under the MIT license, performed on par with OpenAI’s o1 on math and coding, scoring 79.8% on AIME 2024 in a single attempt!

DeepSeek-R1’s RL training pipeline: rollout, inference, and training modules with VRAM management across phases. Image source. 

There was also a very famous “$5 million” figure that was being rumored at this time. 

The DeepSeek-V3 technical report (R1’s base model) estimated about $5.6 million for the final training run alone, based on 2.788 million H800 GPU hours at an assumed $2 per hour, but specifically excluded earlier research, experiments, and hardware. 

So the real total was definitely higher, but still far below what people assumed a frontier model cost.

This is why on January 27, 2025, NVIDIA lost close to $600 billion in market value in a single day, the largest one-day drop for any US company at the time, as part of a wider sell-off of roughly $1 trillion.

The result I find most interesting scientifically is DeepSeek-R1-Zero, where DeepSeek applied reinforcement learning directly to the base model with no supervised fine-tuning at all. 

Its AIME 2024 score climbed from 15.6% to 71.0%, with the model teaching itself to go back and re-check its own work. 

However, I would like to mention that R1-Zero’s actual reasoning text was hard to read and often mixed languages, so the R1 model DeepSeek actually shipped used a small amount of supervised “cold-start” data first.

After DeepSeek, open models kept arriving, many using a mixture-of-experts (MoE) design, where only a small group of specialist “experts” inside the model activates for each token. 

This is how DeepSeek-V3 managed 671 billion parameters in total while only using 37 billion per token. 

Alibaba’s Qwen3 (April 2025), Meta’s Llama 4 (April 2025), and Mistral’s Mistral 3 (December 2025) all followed the same broad pattern, mostly under Apache 2.0.

For developers, getting access to a good model stopped being the hard part, and the real challenges moved to evaluation, serving infrastructure, and data.

Era 8: The Agentic Era (2025 to 2026)

Here is where we start moving from simple chatbots to agentic architectures.

An AI agent is a system that uses an LLM as its brain and is wrapped in a loop that lets it use tools, remember things, plan several steps ahead, and check its own work until a goal is achieved. 

The LLM stops being the product and becomes one part of a bigger system.

If you ask a chatbot how to book a train ticket, it tells you the steps, whereas an AI agent actually opens the website, picks the train, fills in the form, and comes back to confirm the payment with you.

From chatbot to agent loop

This idea is older than it looks, though, as ReAct (October 2022) had models switch between reasoning and taking actions. 

What changed in 2025 was that reasoning models got much better at planning; models were trained specifically to use tools. 

Plus, Anthropic’s Model Context Protocol (MCP), open-sourced in November 2024, gave agents a standard way to plug into tools and data. Coding agents such as Claude Code and OpenAI’s Codex also emerged in this era.

It’s worth noting that under the hood, the core loop is surprisingly small.

Here’s a simplified sketch I have made in Python (llm and the tool format are placeholders, not a real library, so treat this as pseudocode):

def run_agent(goal, llm, tools, max_steps=10):
    """Minimal agent loop: think, act, observe, repeat."""
    history = [{"role": "user", "content": goal}]

    for step in range(max_steps):
        # Ask the LLM what to do next, given everything so far
        decision = llm(history, tools=tools)
        history.append({"role": "assistant", "content": str(decision)})

        # The model has decided the goal is met
        if decision["type"] == "final_answer":
            return decision["content"]

        # Otherwise, run the tool it asked for and feed the result back
        tool = tools[decision["tool_name"]]
        result = tool(**decision["arguments"])
        history.append({"role": "tool", "content": str(result)})

    return "Stopped: step limit reached"

Everything else in a real agent, such as memory, permissions, retries, and self-correction, is built around this loop. 

For the bigger picture, take a look at LLM Agents Explained, and if you want to build one, Developing LLM Applications with LangChain is a good hands-on course.

The 2026 landscape

By 2026, no single lab is clearly ahead, and each major model family plays to its own strengths. 

OpenAI’s GPT-6 Astra (September 2026) was the first model to hit the highest cybersecurity risk tier in OpenAI’s own safety framework, which means some of its more advanced capabilities are gated behind public access. 

Anthropic, Google DeepMind, and others have shipped their own flagships around the same period, each with its own mix of reasoning ability, context length, and safety gating.

I believe what matters more than the exact leaderboard is the shift in what’s being competed over: long, multi-step agentic tasks rather than single-answer benchmarks.

What comes next: physical AI

The next frontier is the physical world. Vision-language-action (VLA) models can turn camera images and instructions into robot movements, while world models can learn to predict how an environment will change.

Reasoning is also moving to the edge, with small models running directly on phones and robots rather than in data centers, and newer architectures such as Mamba are challenging the Transformer on very long sequences, though Transformers still dominate.

Summary: The History of LLMs at a Glance

In the table below, you can see the major milestones in the development of LLMs: 

Looking at all 8 eras together, I see 4 patterns:

• New architectures (the LSTM, the Transformer, mixture-of-experts) each removed a technical bottleneck.
• Scale kept increasing, first as parameters and data, and since o1, as thinking time too.
• Alignment, from RLHF to today’s cyber safeguards, decided whether raw capability turned into something people could actually use safely.
• Each era fixed the biggest limitation of the one before it, such as RNNs couldn’t run in parallel, so the Transformer arrived.

Final Thoughts

Weizenbaum spent a lot of his later career warning people not to trust a machine just because it sounds like it understands them. 

Sixty years after ELIZA, the machines can do a lot more, but his warning can be seen as checking what’s actually happening underneath the fluent text.

This is also why I believe this history is worth knowing. 

Every tool that we use today was built to fix a specific limitation, so when you choose between a fast model and a reasoning model or between an API and an open-weights model, you’re really choosing which era’s trade-off fits your problem best.

If you want to start building with modern LLMs yourself, I recommend Large Language Models (LLMs) Concepts and Developing Large Language Models.