Attention Mechanism

Simple Definition

The attention mechanism is the core innovation that makes modern AI language models work. When an AI generates a response, it doesn’t process words one at a time in order. It uses attention to weigh which parts of the input are most relevant to each part of the output.

In plain terms: attention lets the model decide what to focus on. When generating the word “it” in a sentence, the model uses attention to figure out that “it” refers to something mentioned earlier, and gives that earlier word more weight in its calculations.

Why It Was a Breakthrough

Before the attention mechanism, AI models processed text sequentially, one word at a time, and struggled to connect information across long distances in text. If a sentence had a pronoun referring back to something mentioned 200 words earlier, the model would often lose track.

The attention mechanism solves this by looking at all words simultaneously and calculating how much each word should influence every other word in the output.

A Simple Analogy

Imagine reading a mystery novel and a character mentions a detail in chapter 2 that becomes important in chapter 10. If you read normally, you’d have to remember that detail for 8 chapters. The attention mechanism is like being able to instantly highlight every passage relevant to the chapter you’re reading right now, no matter how far back it appeared.

Self-Attention

The most important form is “self-attention”, where every word in the input attends to every other word to build up a rich understanding of relationships and context before any output is generated.

This is what allows a model to understand:

“Paris is the capital of France. It has the Eiffel Tower.” → “it” refers to Paris, not France
The subject of a very long sentence at the start of a paragraph

Attention and Context Windows

The attention mechanism is also why context window size matters. To compute attention, the model must compare every token against every other token in the context. This scales with the square of context length, making longer contexts exponentially more expensive to compute.

Transformer, the architecture built around the attention mechanism
LLM, all modern LLMs are transformer models that use attention
Context Window, the limit on how much text the attention mechanism can consider at once
Neural Network, the broader structure inside which attention mechanisms operate
Embedding, attention operates on the embedded representations of tokens