What Is a Transformer? Attention Explained Simply
A plain-English guide to the transformer, the design behind modern language models: what attention means, how it works step by step, and a tiny Python example you can run.
Short answer: a transformer is the neural network design behind modern language models such as ChatGPT. Its key idea is attention: for every word (token), the model looks at all the other words in the text and decides which ones matter most for understanding it. This lets it handle context far better than older models that read text strictly one word at a time.
Why Was the Transformer Invented?
Earlier language models, called recurrent networks, read a sentence one word after another and carried a memory forward. That was slow, because each step waited for the previous one, and the memory of early words faded in long texts.
The transformer was introduced in a 2017 research paper titled "Attention Is All You Need". It processes all the tokens of a text together, which suits modern hardware, and it connects distant words directly. That is why it scaled up into today's large language models.
What Does "Attention" Mean in Simple Words?
Take the sentence: "The bank near the river was flooded." The word "bank" could mean a place for money or the side of a river. A human looks at "river" and "flooded" to decide. Attention does the same: when the model works on "bank", it gives a higher weight to "river" and a lower weight to "the".
The result is a new representation of "bank" that is a weighted mix of the words around it, so it now carries the right meaning.
How Does Attention Work, Step by Step?
Tokens become vectors. Text is split into tokens, and each token becomes a list of numbers (an embedding). Position information is added so the model knows word order.
Each token asks and offers. From each vector the model makes three versions: a query (what am I looking for?), a key (what do I contain?) and a value (what information do I pass on?).
Scores are computed. A token's query is compared with every key. A higher match means a higher score.
Scores become weights. A function called softmax turns the scores into weights that add up to 1.
Values are mixed. The token's new vector is the weighted sum of all the values.
A Tiny Attention Example in Python
This toy version skips the learned query, key and value layers and uses made-up two-number vectors, but it shows the scoring, softmax and mixing steps.
import math
def softmax(scores):
exps = [math.exp(s) for s in scores]
total = sum(exps)
return [e / total for e in exps]
def dot(a, b):
return sum(x * y for x, y in zip(a, b))
# toy vectors for the words "the", "bank", "river"
vectors = {"the": [0.1, 0.0], "bank": [1.0, 0.5], "river": [0.9, 1.0]}
query = vectors["bank"]
words = list(vectors)
scores = [dot(query, vectors[w]) for w in words]
weights = softmax(scores)
for w, wt in zip(words, weights):
print(w, round(wt, 2))
# new representation of "bank": a weighted mix of all the word vectors
mixed = [sum(wt * vectors[w][i] for w, wt in zip(words, weights)) for i in range(2)]
print([round(x, 2) for x in mixed])
It prints weights of about 0.13 for "the", 0.4 for "bank" and 0.47 for "river". "River" gets the most attention, so the new "bank" vector leans towards it.
What Is Inside a Full Transformer?
Part | What it does |
|---|---|
Embeddings and positions | Turn tokens into vectors that also know their place in the text |
Multi-head attention | Runs several attention calculations at once, each free to notice a different relationship |
Feed-forward layers | Process each token's vector further after attention |
Stacked layers | Repeats the block many times so understanding gets deeper |
Output layer | Produces probabilities for the next token |
Models such as GPT use this design to predict the next token again and again, which is how they write text. Other models, such as BERT, use the same ideas mainly to understand text. Transformers are also used for images, audio and code.
What Are the Limits?
Attention compares tokens with each other, so the work grows quickly as texts get longer. That is one reason every model has a context window. Transformers also need large amounts of data and computing power to train, and they can still produce confident mistakes.
Common Mistakes
Thinking attention means the model "understands" like a human: it is learned maths over numbers
Mixing up transformers with the Transformers movies or electrical transformers when searching
Believing every AI model is a transformer: many other designs exist
Key Takeaway
A transformer reads all tokens together and uses attention to weigh which other tokens matter for each one. Stack that idea many times, train on huge text, and you get the language models behind modern chatbots.
Ready to start? Understand tokens, models and how LLMs generate answers, with plain explanations and no heavy maths. Begin with the free course How AI Works: Foundations for Everyone.
Related articles
More in LLM
What Are Open-Source LLMs? Run One Locally With Ollama
What open-source LLMs are, why people run them locally, and how Ollama lets you download and use one with a few commands.
3 min readNov 9, 2026What Are Top-k and Top-p in LLMs? How Do They Control AI Output?
Top-k and Top-p are sampling techniques that control which candidate tokens an LLM can choose from during generation. Understanding them is important for controlling creativity, diversity, and consistency in AI applications.
2 min readAug 30, 2026What Are Tokens and Tokenization in LLMs? Why Are They Important?
LLMs don't process text exactly like humans do. They process text as tokens. Understanding tokenization is essential for understanding context windows, API costs, model limits, and LLM performance.
2 min readAug 29, 2026What Is a Context Window in LLMs? Why It Matters
The context window is how many tokens an LLM can consider in one request: instructions, chat history, documents and its own answer. Learn why it matters and how to manage it.
3 min readAug 29, 2026What Is AI Automation? Real Examples
AI automation puts a language model inside a workflow to handle messy, text-heavy steps. See real examples, a small code pattern and the risks to watch.
4 min readNov 14, 20267 AI Project Ideas for Your Portfolio
Seven small, finishable AI project ideas for a beginner portfolio, with the skill each one teaches and how to present it so others trust your work.
4 min readNov 14, 2026
Comments
Sign in to join the discussion.


