- A transformer is an architecture, not a model like GPT.
- The 2017 paper enabled parallel attention across whole sequences.
- Encoder-only, decoder-only, and encoder-decoder families produced today's AI.
A transformer is a neural-network architecture that processes sequential data, such as text, images, audio, or protein chains, by computing learned relationships between every element in the sequence at once. The name has no connection to electrical transformers; in machine learning it refers to the way the network transforms an input sequence of tokens into useful internal representations.
Why It Matters
Key figure
2017
year the transformer architecture was introduced
The transformer is the most-leveraged single idea in recent AI. Almost every large model a general reader has heard of is a transformer underneath, from chat assistants like GPT, Claude, and Gemini to scientific systems like AlphaFold for proteins and Whisper for speech. Understanding the term is closer to understanding the steam engine in 1830 than picking a brand of car today.
It is the architecture under nearly all generative AI systems shipping today. Every large language model in commercial use is a transformer trained at scale, which is why advances in transformer training tend to ripple across the entire field within months.
It also reaches well beyond chat. DeepMind's AlphaFold 2 uses transformer-based attention to predict protein structures. The same lab's AlphaGenome applies the architecture to long stretches of DNA, as covered in our piece on how an AI model reads DNA's hidden regulatory switches. The same core mechanism powers Vision Transformers for image recognition and OpenAI's Whisper for audio.
The architecture also shapes the failure modes that most concern researchers. Large language models routinely invent confident, unsupported statements, a behavior we cover in detail under AI hallucination. The pattern is in part a property of how transformers predict tokens from learned statistics rather than from any external source of truth.
How It Works
Key figure
173,000+
citations of "Attention Is All You Need" by 2025
A transformer takes an input sequence, splits it into tokens (subword units for text, patches for images, frames for audio), and converts each token into a vector. These token vectors are passed through a stack of transformer blocks, each containing two sub-layers: a self-attention layer and a small feed-forward network. The MIT 6.390 course notes describe this two-sub-layer block as the basic repeating unit.
Self-attention is the central move. For each token, the network learns three projections, called query, key, and value, all introduced by Vaswani et al. in 2017. The model compares one token's query against every other token's key to decide how much attention to pay to each, then mixes their value vectors using those weights. The result is a representation in which each token has been informed by the rest of the sequence.
The original encoder-decoder transformer from the paper"Attention Is All You Need" (2017). Each repeating block combines self-attention, a feed-forward network, and residual connections around layer normalization; the decoder adds a masked self-attention layer plus a cross-attention layer that pulls keys and values from the encoder output. Modern systems use one half: encoder-only (BERT), decoder-only (GPT, Claude, Gemini), or both (T5). Image by dvgodoy (CC BY 4.0).
Multi-head attention runs several self-attention operations in parallel, each with its own query, key, and value projections. Different heads learn to track different relationships, such as syntax in one head and topical reference in another, and their outputs are concatenated before passing to the feed-forward layer.
Self-attention has no inherent sense of order, since computing weights between tokens treats the sequence as a set. To restore position, transformers add positional embeddings, either learned or sinusoidal, to the token vectors before the first block. The architecture's other big advantage is full parallelization across the sequence, which lets GPUs and TPUs train on far larger datasets than the recurrent networks it displaced.
Key Context
The architecture was introduced in "Attention Is All You Need," posted to arXiv on June 12, 2017 by Ashish Vaswani and seven coauthors, and presented at NeurIPS in December that year. All eight authors were at Google at the time, working across Google Brain and Google Research, with one University of Toronto intern, Aidan Gomez, on the team. The Vaswani transformer achieved 28.4 BLEU on the WMT 2014 English-to-German translation benchmark, more than two BLEU above the prior state of the art.
The paper had collected more than 173,000 citations by 2025, placing it among the most cited scientific works of the 21st century. The field then split into three families during 2018 and 2019. Google's BERT, released in October 2018, used the encoder half; OpenAI's GPT-1, from June 2018, used the decoder half; Google's T5, in 2019, used both. Current systems include Mixture of Experts transformers, long-context variants built on FlashAttention, and reasoning-trained models such as OpenAI's o-series.
FAQ
Is a transformer the same thing as a large language model?
No. A transformer is an architecture, a blueprint for a neural network; a large language model is a specific trained instance of that architecture, such as GPT-4 or Claude. Every modern LLM is a transformer, but transformers also power non-language systems like AlphaFold for proteins and Whisper for speech.
What did the transformer replace, and why did self-attention win?
It replaced recurrent neural networks and their LSTM variants, which processed sequences one token at a time. Self-attention computes relationships across the whole sequence in parallel, which fits GPU and TPU hardware far better and lets the model see long-range dependencies in a single step. Those two advantages drove a step change in trainable scale.
What is the difference between encoder-only, decoder-only, and encoder-decoder transformers?
Encoder-only models like BERT read a whole input at once and produce representations, which suits classification and search. Decoder-only models like GPT generate one token at a time, which suits chat and writing, and underpin systems explored in our entry on agentic AI. Encoder-decoder models like T5 and the original Vaswani transformer use both halves, mapping an input sequence to an output sequence for tasks like translation.
Why is it called a transformer when it has nothing to do with electrical devices?
The name refers to how the network transforms one sequence of vectors into another through repeated attention and feed-forward operations. The Vaswani et al. 2017 paper uses the name in that mathematical sense; coauthor Jakob Uszkoreit reportedly chose it. The shared word with electrical transformers is a coincidence with no shared theory.
Is the transformer still the dominant architecture in 2026?
Yes. Every frontier system in commercial use is a transformer, including Mixture of Experts variants like Mixtral and Gemini 1.5, long-context models built on FlashAttention, and reasoning-trained models like OpenAI's o-series. Alternatives such as Mamba and other state-space models show promise on specific tasks but have not displaced transformers at scale.
Related Reading
Sources
- Primary Sources:
- Attention Is All You Need (Vaswani et al., arXiv:1706.03762, June 2017)
- Attention Is All You Need, NeurIPS 2017 proceedings PDF
- Transformers, MIT 6.390 Introduction to Machine Learning course notes
- Additional Context:
- Transformer: A Novel Neural Network Architecture for Language Understanding (Google Research blog, August 2017)
- What is a Transformer? (Stanford HAI AI Definitions)
Fact Check: Claim-by-Claim Verification Verified
Independently verified by Claude and Perplexity (sonar-pro-search) across two rounds. All core claims supported against the original arXiv paper (Vaswani et al. 2017), the NeurIPS 2017 proceedings, MIT 6.390 course notes, Stanford HAI, and the Google Research blog. Two minor phrasings were tightened before publication.
Sources used for verification
- Attention Is All You Need - Vaswani et al., arXiv:1706.03762
- Attention Is All You Need (NeurIPS 2017 PDF)
- Transformers - MIT 6.390 Introduction to Machine Learning
- What is a Transformer? - Stanford HAI
- Transformer: A Novel Neural Network Architecture for Language Understanding - Google Research blog, August 2017
