Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.
They were not proposing another small improvement to recurrent neural networks. They were removing recurrence itself.
That decision eventually became the architectural foundation for BERT, GPT-style models, T5, and most modern large language models.
An encoder-only Transformer, a decoder-only Transformer, and an encoder-decoder Transformer use closely related building blocks, but they impose very different information-flow constraints.
Understanding those constraints makes a lot of current LLM behavior much easier to reason about.
This article builds from intuition to the actual attention equations, then connects the mathematics to the three major Transformer architectures. What attention actually does
ran out of tells us this is probably a resource. server tells us which kind of memory we mean. crashed tells us the event associated with it.
For each token, dynamically decide which other tokens are useful, and combine information from them.
This is different from a traditional feed-forward network, where each position can be processed more independently.
It is also fundamentally different from an RNN. An RNN processes a sequence step by step:
The Transformer instead lets every token interact with other tokens in parallel:
At least, that is the basic encoder-style version. Later we will see that decoder-only models intentionally restrict those connections.
The original Transformer paper demonstrated that a purely attention-based architecture could outperform recurrent models on machine translation while being substantially more parallelizable during training. That paper came from Ashish Vaswani and colleagues at Google and the University of Toronto.
The core operation responsible for this is scaled dot-product attention. The mathematics of attention
If the query and key point in similar directions in representation space, the score is high.
As vector dimensionality increases, the magnitude of dot products tends to increase as well. Without scaling, the softmax can become excessively sharp, pushing probabilities toward 0 and 1 and making optimization harder.
For this token, pay 70% of the attention to position 3, 15% to position 4, and so on.
The output for a token is therefore a weighted combination of information from other tokens.