How we got here
From the first rule systems through two winters to language models: the line of development that explains why today's systems look the way they do.
The line in five steps
| Period | What happened | Why it stalled |
|---|---|---|
| 1950s to 1960s | First programs, the perceptron, large expectations | Compute and theory were missing |
| 1970s to 1980s | Expert systems with rules from specialists | Maintaining rules cost more than the benefit |
| 1990s to 2000s | Statistical methods, databases grow large | Image and speech tasks stayed hard |
| 2012 to 2017 | Deep networks win image competitions | Not a break but the start of scaling |
| From 2017 | Transformers, then language models | Ongoing |
What made the difference
Not a better idea but three circumstances at once: the internet supplied training data at a scale that had not existed. Graphics cards made matrix arithmetic a hundred times cheaper. And a handful of technical tricks made deep networks trainable at all.
The two winters
The first, from around 1974. The 1960s expectations of machine translation and general problem solving did not materialise. Reports to funders were damning and funding collapsed.
The second, from around 1987. Expert systems were commercially widespread and failed on maintenance: every rule had to be entered by specialists, and rule bases became contradictory faster than they delivered value.
Both times the technology was not wrong; the expectation of its maturity was. That experience is why the field remains reserved about announcements to this day.
What 2012 and 2017 meant
| Year | Event | Consequence |
|---|---|---|
| 2012 | A deep convolutional network wins an image competition decisively | Image processing changes method within two years |
| 2017 | The transformer architecture is published | Sequence processing becomes parallelisable and therefore scalable |
| 2020 | Scaling laws are formulated | Size becomes a plannable investment rather than an experiment |
The point about 2017 is not intelligence but parallelisability. Earlier methods had to compute word by word. A transformer processes the whole sequence at once and can therefore be spread across thousands of cards.
Why scaling worked
The empirical observation carrying recent development is plain: loss, parameter count and data volume are related by a power law over wide ranges. Double compute and data together and the loss falls predictably.
That turned a research question into an investment decision. To know what a model with ten times the budget will do, you no longer have to build it to estimate it.
The open points
- Data limit. High-quality text on the web is finite. Whether synthetic data can substitute is open, see synthetic data.
- Energy limit. Compute demand grows faster than hardware efficiency.
- Reliability. Hallucination follows from the design; it cannot be trained away.
- Evaluation. Benchmarks are increasingly contaminated and defensible yardsticks are missing.
What follows for practice
Two lessons from the history still hold:
- A method becomes useful when it becomes cheap enough, not when it becomes clever enough. Expert systems failed on maintenance cost, not on logic.
- Expectations about maturity are the most common source of error. Both winters followed promises the technology could not keep. Anyone planning today is well advised to tie value to measured tasks rather than to the announcement of the next model.
The timeline to take away
FREE ACCOUNT
Timeline and pattern card
The turning points with year and effect, and the pattern of expectation and disappointment that has repeated four times.
Worksheet2 items
Related courses and sources
AI Index Report
An annual report with sourced figures on models, cost, adoption and regulation. Useful when a statement needs a source rather than an impression.
When a statement needs a source rather than an impression.