Cursus
Développer des LLM
16 h
In late 2017, a research paper titled Attention Is All You Need introduced a revolutionary mechanism that would fundamentally reshape artificial intelligence. At the heart of this breakthrough was self-attention, a simple yet powerful idea that enables models to understand relationships within data by determining which elements deserve focus.
Today, self-attention powers everything from ChatGPT's conversational abilities to advanced image recognition systems. It is definitely one of the most transformative innovations in machine learning history.
In this guide, I'll walk you through self-attention from its conceptual foundations to its applications, showing you how the mechanism helps models capture long-range dependencies and process information in parallel, capabilities that were previously unattainable with traditional neural networks.
If you are new to transformers, consider taking one of our introductory courses on Large Language Models (LLMs) Concepts and Transformer Models with PyTorch.
What is Self-Attention?
In my experience teaching this concept, the best starting point is a fundamental challenge in sequence processing: how can a model determine which parts of its input are most relevant at any given moment?
Let's begin with the intuition before diving into any math.
Conceptual understanding of self-attention
Self-attention is a mechanism that weighs the importance of different elements in a sequence relative to each other. When processing the sentence "The animal didn't cross the road because it was too tired," self-attention helps the model understand that "it" refers back to "animal" rather than "road", a relationship that spans multiple words.
This is actually the example I always use when introducing the concept to students. It clicks immediately because it mirrors how we naturally read.
What makes self-attention particularly powerful is its ability to compute these relationships for all positions in a sequence simultaneously. Unlike recurrent neural networks that process sequences step-by-step, self-attention examines every element in relation to every other element in a single forward pass.
This parallel computation dramatically reduces training time while capturing dependencies regardless of their distance in the sequence.
It's important to distinguish self-attention from related mechanisms. In self-attention, a sequence attends to itself. Each element queries information from all positions within the same sequence.
Cross-attention, by contrast, allows one sequence to attend to another, as when a decoder attends to encoder outputs during machine translation.
Masked self-attention adds an additional constraint: it prevents positions from attending to subsequent positions, ensuring that predictions depend only on known outputs, essential for autoregressive text generation.
But where did this idea actually come from, and what problems was it designed to solve?
Origins and motivation for self-attention
Before self-attention emerged, the field grappled with significant limitations in sequence modeling.
Recurrent neural networks, including LSTMs and GRUs, process sequences sequentially, creating a fundamental bottleneck: They struggled with long-range dependencies because information had to pass through many intermediate states, leading to vanishing gradients and forgotten context.
The evolution toward self-attention began with attention mechanisms in RNN-based encoder-decoder architectures. In 2014, Bahdanau and colleagues introduced an attention mechanism that allowed decoders to access all encoder hidden states rather than relying on a single fixed-length context vector.
This breakthrough addressed the information compression problem, where variable-length input sequences were squeezed into fixed-length representations regardless of their complexity.
Bahdanau's graphical illustration of the proposed model
A year later, Luong proposed simplified attention mechanisms, including the scaled dot-product approach that would later influence transformer design. While these early attention mechanisms improved performance, they still depended on sequential RNN processing.
Global attentional model (Luong)
The true revolution came when researchers asked: What if we eliminate recurrence and rely solely on attention? This question led to the transformer architecture, where self-attention became the primary mechanism for understanding sequential relationships.
The Mathematical Framework of Self-Attention
Now that we've established why self-attention matters, let's examine how it actually works at a mathematical level.
Query, key, and value representations
Self-attention transforms input sequences through three learned projections: queries, keys, and values. Think of these as different "views" of the same information, each serving a distinct purpose in the attention computation.
For an input sequence with embedding dimension d_model, we create three weight matrices: W_Q, W_K, and W_V. When we multiply our input matrix X by these weights, we obtain:
-
Query matrix
Q = X · W_Q: represents "what I'm looking for" -
Key matrix
K = X · W_K: represents "what I offer" -
Value matrix
V = X · W_V: represents "what I actually contain"
I find this library analogy helpful: you walk in with a query in mind, the index cards are your keys, and the actual books are your values.
Typically, these matrices project to a smaller dimension d_k (often d_model divided by the number of attention heads), which manages computational complexity while maintaining representational capacity. This dimensionality reduction especially becomes important when scaling to longer sequences.
With Q, K, and V in hand, we can now compute attention itself.
Scaled dot-product attention
The attention computation itself follows an elegant formula. First, we calculate attention scores by taking the dot product between queries and keys. This operation measures the compatibility between what each position is looking for and what every other position offers.
However, as dimensionality increases, these dot products can grow very large in magnitude, pushing the softmax function into regions with extremely small gradients. To prevent this numerical instability, we scale by dividing by the square root of the key dimension.
Scaled dot-product attention
This scaling factor maintains appropriate variance throughout the network, ensuring stable gradients during training. Without it, models with high-dimensional embeddings would struggle to learn effectively, as the softmax would output near-one-hot distributions that fail to distribute attention meaningfully.
Before moving on, there's one important gap we need to address: Positional encoding.
Positional encoding
Here's the part that surprised me most when I first studied this: self-attention is completely permutation-invariant. Shuffle the input tokens, and you'll get the same output. Just shuffled accordingly.
It took me a moment to wrap my head around the implication: the mechanism is powerful, but essentially blind to order. For tasks where order matters (like understanding language), we need to explicitly inject positional information.
The original transformer used sinusoidal position encodings, applying sine and cosine functions of different frequencies to each dimension. This choice wasn't arbitrary. The wavelike patterns allow models to learn to attend to relative positions, and the encoding extends naturally to unseen sequence lengths.
Positional encoding visualization
More recent innovations include RoPE (Rotary Position Embedding), which rotates query and key vectors in a multi-dimensional space by angles proportional to their positions. After rotation, the dot product between queries and keys naturally encodes relative distance.
RoPE has gained widespread adoption in models like LLaMA due to its strong extrapolation capabilities.
Another approach, ALiBi (Attention with Linear Biases), takes a different path by adding position-dependent penalties directly to attention scores. Rather than modifying embeddings, ALiBi biases attention weights to favor nearby tokens, with the bias intensity varying across different attention heads.
This method demonstrates impressive extrapolation to sequence lengths far beyond those seen during training.
Softmax and attention weights
After computing scaled attention scores, we apply softmax across the key dimension. This normalization converts raw scores into a probability distribution, ensuring that attention weights sum to one for each query position.
The softmax function emphasizes the highest-scoring relationships while suppressing irrelevant connections. However, this can sometimes lead to overly peaked distributions, particularly during early training when the model hasn't yet learned nuanced attention patterns.
To mitigate this, practitioners often apply dropout to attention weights, randomly zeroing out some connections to encourage robustness.
Computing context vectors
With attention weights in hand, we compute the final output as a weighted sum of the value vectors.
Each position receives a context-aware representation that aggregates information from across the entire sequence, with contributions proportional to the attention weights. This weighted combination allows information to flow dynamically based on learned relationships rather than fixed patterns.
Single-head attention gets us surprisingly far, but it has a ceiling. Here's where things get more interesting.
Multi-Head Attention and Multiple Representation Subspaces
While single-head attention is powerful, it has a fundamental limitation: it can only capture one type of relationship at a time.
Motivation and architecture
Multi-head attention addresses this by running several attention computations in parallel, each focusing on different aspects of the input relationships. For instance, there might be one head each specializing in:
- Local syntactic dependencies
- Long-range semantic relationships
- Positional patterns
From what I've observed working with these models, this specialization often emerges organically during training. You don't design it, which I think is one of the most fascinating aspects of the whole mechanism.
The implementation involves splitting the Q, K, and V matrices along the dimension axis, processing each split through separate attention heads, and concatenating the results.
If we have 8 heads and 512-dimensional embeddings, each head operates on 64-dimensional subspaces. This parallel processing doesn't increase the complexity of the sequence length but does multiply the model's capacity to learn diverse patterns.
What different heads learn
Research into what different attention heads actually learn has revealed fascinating patterns.
In translation models, some heads consistently attend to positional relationships, focusing on neighboring words regardless of content. Other heads specialize in syntactic roles, identifying subject-verb or verb-object relationships. A third category attends to rare or high-information tokens like proper nouns and technical terms.
Interestingly, interpretability studies show that many heads appear redundant, and models can retain performance even when a significant fraction of heads are pruned. This redundancy likely contributes to robustness, ensuring that critical patterns have multiple pathways through the network.
In encoder-decoder attention, heads in later decoder layers prove most essential for translation quality, while encoder self-attention heads can often be reduced substantially.
Self-Attention Within the Transformer Architecture
With the mathematical foundations established, let's see how self-attention integrates into the complete transformer architecture.
The encoder: all-to-all attention
In the transformer encoder, self-attention layers allow each position to attend to every other position in the input sequence. This "all-to-all" pattern enables bidirectional context flow, where each token builds a representation informed by the entire sequence.
Encoder architecture
Each encoder layer combines multi-head self-attention with a position-wise feed-forward network. The feed-forward network processes each position independently but identically, applying the same learned transformation across all tokens.
Residual connections and layer normalization surround both sub-layers, stabilizing gradients and enabling deeper architectures. Stacking multiple encoder layers creates increasingly abstract representations, with early layers capturing surface patterns and deeper layers encoding semantic relationships.
The decoder: masked attention and causality
The decoder modifies self-attention to respect autoregressive constraints. When generating text, the model can only condition on tokens it has already produced. Peeking at future tokens would constitute cheating during training.
Decoder architecture
Masked self-attention accomplishes this by setting attention weights to negative infinity (which becomes zero after softmax) for all positions following the current one.
This ensures that predictions depend only on previous context, and the necessary causal ordering for generation tasks is maintained. During inference, this causality becomes natural: we literally don't have future tokens to attend to yet.
Cross-attention: connecting encoder and decoder
Between the decoder's self-attention layers lies cross-attention, which enables the decoder to condition on encoder outputs. Here, queries come from the decoder (representing "what I need to generate"), while keys and values come from the encoder (representing "what the input provides").
Transformer architecture
This cross-attention mechanism is essential for sequence-to-sequence tasks like translation, where the decoder must align output words with relevant input words. Unlike self-attention, cross-attention creates dependencies between two different sequences, allowing information to flow from source to target.
Theory aside, what has self-attention actually enabled in practice? The answer is broader than most people expect.
Applications of Self-Attention Across Domains
Having explored how self-attention works within the transformer architecture, let's examine where this mechanism has made a real-world impact.
What makes self-attention particularly valuable is its domain-agnostic nature. The same mathematical framework that helps models understand language relationships can also capture patterns in images, audio, and even combinations of different data types.
In my view, this domain-agnostic quality is what makes self-attention genuinely exciting. I'd argue it's less a NLP tool and more a general-purpose relationship-learning mechanism.
Natural language processing applications
Transformer models excel at capturing long-range dependencies that earlier architectures missed. In NLP, self-attention powers breakthrough models across virtually every task.
- Machine translation: Alignment of source and target phrases regardless of their distance
- Sentiment analysis: identification of contextual cues that modify meaning
- Question answering: Connection of questions to relevant passage segments, even when separated by many tokens.
As you can see in the image below, the architecture of the models differs depending on whether the focus is on the decoder or the encoder.
Architecture comparison: Transformer vs. GPT vs. BERT
Models like BERT use bidirectional self-attention in their encoder-only architecture, enabling them to build rich contextual representations for classification and understanding tasks.
GPT models like the recently released GPT-5.3 Codex use masked self-attention in their decoder-only design, achieving remarkable text generation capabilities.
T5 employs the full encoder-decoder transformer architecture, treating all NLP tasks as text-to-text problems.
Computer vision applications
The Vision Transformer (ViT) adaptation brought self-attention to computer vision by treating images as sequences of patches. Instead of processing pixels through convolutional layers, ViT splits images into fixed-size patches (typically 16×16), flattens each patch into a vector, and processes the sequence through transformer layers.
This approach captures global relationships that convolutional networks' local receptive fields might miss. ViT and its variants now achieve state-of-the-art results in image classification, object detection, semantic segmentation, and other fields:
- Medical imaging: Correlation of subtle anomalies across different regions of a scan.
- Satellite imagery analysis: Connection of distant features like deforestation sites to their causes.
Speech, recommender systems, and multimodal learning
Self-attention's utility extends to speech processing, where it captures temporal dependencies in audio sequences more effectively than recurrent models. In recommender systems, self-attention helps model user preference patterns by attending to relevant historical interactions.
Perhaps most exciting are multimodal applications that combine vision and language. Models like CLIP use cross-attention between image and text representations, enabling zero-shot image classification and image generation from text descriptions.
What all these systems have in common: They demonstrate self-attention's fundamental capability to learn relationships between any sequential or structured data, regardless of modality.
Advanced Self-Attention Variants and Recent Developments
While self-attention has been remarkably successful across domains, new challenges arise when deploying transformer models at scale. As models grow to billions of parameters and process contexts stretching into hundreds of thousands of tokens, memory bandwidth and computational costs become critical bottlenecks.
The focus in developing new models lies increasingly on making self-attention faster and more efficient without sacrificing its core capabilities. I find this tension between capability and efficiency to be one of the most interesting open problems in the field right now. Let’s take a look at some of the current developments.
Grouped-query and multi-query attention
Standard multi-head attention maintains separate key and value projections for each head, creating substantial memory overhead during inference. Multi-query attention (MQA) reduces this by sharing a single key-value (KV) head across all query heads, drastically reducing KV cache size and accelerating decoding. However, MQA can degrade quality.
Grouped-query attention (GQA) is a more balanced approach, dividing query heads into groups that share key-value pairs. With 32 query heads and 8 groups, each group of 4 queries shares one key-value head. For instance, the Mistral 3 model family uses this approach and is able to achieve near-full-attention quality with significant speed gains.
Hybrid architectures combining attention with state space models
While GQA addresses memory efficiency, another line of research tackles attention's fundamental computational bottleneck from a different angle.
Recent work explores combining self-attention with state-space models to improve efficiency. Systems like S4 and Hyena use structured state spaces to model long-range dependencies with linear rather than quadratic complexity.
These approaches address self-attention's fundamental limitation: computational cost scales quadratically with sequence length, making extremely long contexts prohibitively expensive.
Inference-time scaling and test-time optimization
Beyond reimagining the architecture itself, researchers have also found clever ways to optimize existing attention mechanisms.
Beyond architectural changes, recent advances optimize attention during inference. Quantization reduces KV cache precision to 8 or 4 bits, dramatically cutting memory with minimal quality loss.
Flash Attention reorganizes GPU memory access to minimize data movement, achieving 2-3× speedups. Test-time compute optimization, where models perform multiple passes during generation, suggests that strategic attention application might yield better results than purely scaling model size.
Conclusion
Self-attention is one of the most transformative mechanisms in machine learning history. By enabling models to dynamically determine which elements of their input deserve focus, it solved long-standing challenges in capturing long-range dependencies and processing sequences efficiently.
The mathematical elegance of scaled dot-product attention, enhanced through multi-head architectures, provides both theoretical soundness and practical effectiveness.
From powering state-of-the-art language models to revolutionizing computer vision and enabling multimodal AI systems, self-attention continues expanding its reach. Recent innovations in grouped-query attention, hybrid architectures, and inference optimization show that we're still discovering ways to make this mechanism more efficient and capable.
If you want to get into all the details and get some hands-on practice, I recommend enrolling in our Developing Large Language Models skill track.
