- New learning overwrites old because both share the same weights.
- Catastrophic forgetting is the price of distributed representation.
- EWC did not solve it; replay currently works best.
Catastrophic forgetting is the abrupt loss of a neural network's ability to perform a task it had already learned, caused by training it on a new one. The old skill does not merely fade. Rather, it collapses, often within a handful of training steps, because both skills are stored in the same set of weights and nothing in the new training objective is protecting the old.
Cognitive psychology calls the same phenomenon catastrophic interference, the term Michael McCloskey and Neal Cohen used when they named it in 1989. Machine learning has settled on "forgetting." Readers arrive on both, and they mean the same thing.
Why it matters
The phenomenon is the price of the property that makes neural networks useful in the first place.
Key figure
100% to 0%
Accuracy on 17 addition facts after learning 17 more
Robert French, whose 1999 review in Trends in Cognitive Sciences remains the canonical account, located the cause in one phrase: "a single set of shared weights." That architecture is also what "gave the networks their remarkable abilities to generalize and degrade gracefully." The same design decision buys generalization and sells the past.
The problem has not gone away with scale. A 2025 preprint by Yun Luo and colleagues (arXiv:2308.08747v5, not peer reviewed) found "catastrophic forgetting is generally observed in LLMs ranging from 1b to 7b parameters" during continual instruction tuning, and that "as the model scale increases, the severity of forgetting intensifies." That 1b to 7b span is the range they tested, not evidence about frontier-scale systems.
The practical stakes sit in fine-tuning. Every time a pretrained transformer is adapted to a specialized task, its general competence is on the table.
A chatbot losing the thread of a long conversation is not catastrophic forgetting. That is a context-window limit, the weights untouched.
How it works
A network has no filing cabinet. There is no address inside it where "task A" is kept. What it knows about A is spread across the very parameters that gradient-based learning will now adjust for B.
James Kirkpatrick and colleagues stated the mechanism tightly in 2017: forgetting happens "when the network is trained sequentially on multiple tasks because the weights in the network that are important for task A are changed to meet the objectives of task B." Nothing stops it, because the loss function for task B contains no term referring to task A.
Interference was expected. The abruptness was not. French attributes it to "weight cliffs," which he defines as "areas where moving even small distances over the weight-landscape would radically disrupt prior learning."
Fixes sort into three families, a working convention of the field rather than a scheme from any one paper.
Regularization writes the missing term back in: a penalty that grows as the weights drift from where task A left them. Treating every weight as equally precious would freeze the network and block task B, so the penalty has to be selective.
Elastic weight consolidation (EWC), the best-known member, "remembers old tasks by selectively slowing down learning on the weights important for those tasks." It ranks importance using the diagonal of the Fisher information matrix, which estimates how sharply task A's performance responds to nudging each weight.
Replay restores the missing task from the other side, and assumes the old data is still on hand, which after a licensing expiry or a privacy deletion it often is not.
Anthony Robins' 1995 pseudorehearsal escapes that. The network feeds itself random inputs, records its own current outputs, and trains on those self-generated pairs alongside the new task, rehearsing what it believes rather than what it was shown. French's reservation still stands: rehearsal "is a very contrived kind of learning" compared with how people acquire anything.
Parameter isolation abandons sharing altogether. Each task gets its own weights, by freezing a subnetwork once trained or growing new capacity for each arrival, so there is nothing shared to overwrite. The cost is that capacity keeps growing and the network stops transferring what it knows between tasks, which was the point of using one. A fly-inspired network published in 2026 takes the approach close to its logical limit, and pays a price for it worth reading about.
Key context
The original demonstration, 1989. McCloskey and Cohen trained a backpropagation network on the 17 one's addition facts, then trained it on the two's facts.
As French recounts, accuracy on the one's facts "had dropped from 100% to 20%" within the first few trials on the new material. By five more trials it was 1%. By 15, "no correct answers from the previous one's addition problems could be produced by the network."
The denominator is 17 problems. The 1989 chapter is paywalled; every figure here is quoted from French's peer-reviewed review of it.
The two fixes worth knowing
Both families answer one question: when the old task's data is gone, what keeps its knowledge from being overwritten? Elastic weight consolidation (EWC) protects the weights. It scores how much the old task depended on each parameter, then penalizes the new training for moving the high scorers, so the network holds its old answers by refusing to move where it matters. Replay protects the data. It mixes old examples back into the new training, or, when the originals are gone, examples the network generates for itself. Guarding weights is the cheaper option, and the numbers below are what it costs.
Key figure
0.11 points
What EWC gains over doing nothing on class-incremental split MNIST
What a fix is worth depends on the test, 2019. EWC was published under the title "Overcoming catastrophic forgetting in neural networks." How far that holds turns on what the network is told at test time.
Gido van de Ven and Andreas Tolias split MNIST's 10 handwritten digits into five two-way tasks and ran each method 20 times. Told which task it is on, EWC scores 98.64%. Made to infer both the digit and the task, it scores 20.01%, against 19.90% for training with no protection at all. That is 0.11 points, on a problem where chance is about 10%. Generative replay, same benchmark, scores 90.79%.
The gap is not a flaw in the method so much as a property of the benchmark it was celebrated on. Sebastian Farquhar and Yarin Gal argue that permuted MNIST "represents an unrealistic best case scenario for continual learning," and their objection is concrete: "the sensor input in a Mars rover will never be permuted no matter what terrain one moves onto."
Thirty-seven years after the phenomenon was named, replay leads, regularization is a partial measure, and the bill for distributed representation has not been paid.
FAQ
Why don't humans forget catastrophically?
Human forgetting is gradual, and the leading explanation is architectural separation. The complementary learning systems account of James McClelland, Bruce McNaughton and Randall O'Reilly in 1995 has the hippocampus acquiring fast and the neocortex extracting structure slowly. Brains appear to avoid the problem rather than solve it.
Has catastrophic forgetting been solved?
No. EWC in 2017 was the most publicized attempt, and it works well when the system is told which task it is being tested on. When it has to infer that, van de Ven and Tolias measured it at 20.01% on split MNIST against 19.90% for no protection at all. Replay methods do better, and none of them close the gap.
What is the difference between catastrophic forgetting and a model running out of context?
Weights versus context window. Catastrophic forgetting is a permanent change to a model's trained parameters; a chatbot losing track of an earlier turn has parameters that are entirely unchanged. The first is damage, the second is a buffer limit.
What is the stability-plasticity dilemma?
Stephen Grossberg's 1982 framing of the general problem, and the conceptual parent of catastrophic forgetting by seven years. He asked how internal representations "can maintain themselves in a stable fashion against the erosive effects of behaviorally irrelevant environmental fluctuations yet can nonetheless adapt rapidly" to the fluctuations that matter. French restated it as designing a system "simultaneously sensitive to, but not radically disrupted by, new input." Catastrophic forgetting is what the failure looks like in a neural network.
Related Reading
Sources
- French, R.M. (1999) "Catastrophic forgetting in connectionist networks," Trends in Cognitive Sciences 3(4):128-135. DOI 10.1016/S1364-6613(99)01294-2.
- McCloskey, M. & Cohen, N.J. (1989) "Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem," The Psychology of Learning and Motivation 24:109-164. Paywalled; quoted here via French (1999). DOI
- Ratcliff, R. (1990) "Connectionist Models of Recognition Memory," Psychological Review 97(2):285-308. DOI 10.1037/0033-295X.97.2.285
- Grossberg, S. (1982) "Processing of Expected and Unexpected Events During Conditioning and Attention," Psychological Review 89(5):529-572.
- Kirkpatrick, J. et al. (2017) "Overcoming catastrophic forgetting in neural networks," PNAS 114(13):3521-3526. DOI 10.1073/pnas.1611835114
- Kirkpatrick, J. et al. (2016) arXiv:1612.00796v2, preprint version of the above.
- Farquhar, S. & Gal, Y. (2019) "Towards Robust Evaluations of Continual Learning," arXiv:1805.09733v3. Preprint.
- van de Ven, G.M. & Tolias, A.S. (2019) "Three scenarios for continual learning," arXiv:1904.07734. Preprint.
- Luo, Y. et al. (2025) "An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning," arXiv:2308.08747v5. Preprint, under review.
- McClelland, J.L., McNaughton, B.L. & O'Reilly, R.C. (1995) "Why there are complementary learning systems in the hippocampus and neocortex," Psychological Review 102:419-457. Cited via French (1999).
- Robins, A. (1995) "Catastrophic forgetting, rehearsal and pseudorehearsal," Connection Science 7:123-146. Cited via French (1999).
Fact Check: Claim-by-Claim Verification Verified
All 17 direct quotations in this entry were verified against full primary texts held on disk, and every factual claim is supported. The central and most contestable argument, that elastic weight consolidation did not overcome catastrophic forgetting, rests on correctly quoted benchmark numbers from an uninvolved third party and on those authors' own verbatim conclusion. Two errors were caught and corrected before publication: a researcher's given name and an unsupported statistical characterization.
Sources used for verification
- French, R.M. (1999) Catastrophic forgetting in connectionist networks, Trends in Cognitive Sciences 3(4):128-135 - citeseerx.ist.psu.edu
- McCloskey, M. & Cohen, N.J. (1989) Catastrophic Interference in Connectionist Networks, Psychology of Learning and Motivation 24:109-164 - doi.org (paywalled; quoted via French 1999)
- Grossberg, S. (1982) Processing of Expected and Unexpected Events During Conditioning and Attention, Psychological Review 89(5):529-572 - sites.bu.edu
- Kirkpatrick, J. et al. (2017) Overcoming catastrophic forgetting in neural networks, PNAS 114(13):3521-3526 - pnas.org
- Kirkpatrick, J. et al. (2016) arXiv:1612.00796v2, preprint of the above - arxiv.org
- van de Ven, G.M. & Tolias, A.S. (2019) Three scenarios for continual learning, arXiv:1904.07734v1 - arxiv.org
- Farquhar, S. & Gal, Y. (2019) Towards Robust Evaluations of Continual Learning, arXiv:1805.09733v3 - arxiv.org
- Luo, Y. et al. (2025) An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning, arXiv:2308.08747v5 - arxiv.org
