Model Architecture
Attention Mechanism
The part of a transformer that lets the model focus on the most relevant words regardless of where they appear in a sentence.
Definition
The attention mechanism is what allows language models to understand meaning in context. When reading 'The bank refused the loan because it had too many debts,' a human knows 'it' refers to the applicant, not the bank. The attention mechanism lets the model make this kind of connection, weighing which words are most relevant to each other when generating or interpreting text. Without attention, language models struggled with long-distance dependencies in text.
Related Terms
Transformer
The core architecture that powers most modern LLMs, introduced by Google in 2017.
Self-Attention
When a model weighs how much each word in a sentence relates to every other word.
Large Language Model (LLM)
A type of AI trained on vast amounts of text that can read, write, summarise, and reason with language.
Heard enough terminology — ready to talk outcomes?
We translate AI concepts into measurable business results. No upfront fees — you pay only when independently verified results are delivered.
Disclaimer
This definition is provided for educational and informational purposes only. It represents a general explanation of a technical concept and does not constitute professional, technical, or investment advice. Artificial intelligence is a rapidly evolving field; terminology, techniques, and capabilities change frequently. Coaley Peak Ltd makes no warranty as to the accuracy, completeness, or currency of the information provided. Nothing on this page should be relied upon as the sole basis for commercial, technical, legal, or investment decisions without independent professional advice.
Document reference: ISO_webpage_knowledge-base_glossary_v1
Last modified: 29 March 2026
Knowledge Base·Model Architecture·Attention Mechanism