Model Architecture
Self-Attention
When a model weighs how much each word in a sentence relates to every other word.
Definition
Self-attention is the specific mechanism within a transformer where the model examines its own input and calculates how relevant each token is to every other token in the sequence. This allows it to build a rich, context-aware representation of the text, understanding that 'Paris' in 'the capital of France is Paris' is closely related to both 'capital' and 'France,' even if they are far apart in a longer document.
Related Terms
Attention Mechanism
The part of a transformer that lets the model focus on the most relevant words regardless of where they appear in a sentence.
Transformer
The core architecture that powers most modern LLMs, introduced by Google in 2017.
Embedding
A numerical representation of a word or concept that captures its meaning and relationships.
Heard enough terminology, ready to talk outcomes?
We translate AI concepts into measurable business results. No Coaley Peak setup fee or retainer; third-party costs are disclosed before engagement.
Disclaimer
This definition is provided for educational and informational purposes only. It represents a general explanation of a technical concept and does not constitute professional, technical, or investment advice. Artificial intelligence is a rapidly evolving field; terminology, techniques, and capabilities change frequently. Coaley Peak Ltd makes no warranty as to the accuracy, completeness, or currency of the information provided. Nothing on this page should be relied upon as the sole basis for commercial, technical, legal, or investment decisions without independent professional advice.
Document reference: ISO_webpage_knowledge-base_glossary_v1
Last modified: 29 March 2026
Knowledge Base·Model Architecture·Self-Attention