Model Architecture

Tokenisation

The process of breaking text into tokens before the model processes it.

Definition

Before a language model can process text, the text must be converted into tokens — the chunks the model actually works with. Tokenisation is this conversion process. Common words become single tokens; rarer words may be split into multiple tokens. 'running' is one token; 'antidisestablishmentarianism' might be six. This process is invisible to the end user but affects both the cost of using the model (tokens are the unit of pricing) and how well the model handles specialist terminology.

Related Terms

Token

The basic unit an LLM reads and writes — roughly equivalent to a word or part of a word.

Vocabulary

The complete set of tokens a model knows and can use.

Byte Pair Encoding (BPE)

A common tokenisation method that splits text into frequently occurring character sequences.

Heard enough terminology — ready to talk outcomes?

We translate AI concepts into measurable business results. No upfront fees — you pay only when independently verified results are delivered.

← Back to glossary

Disclaimer

This definition is provided for educational and informational purposes only. It represents a general explanation of a technical concept and does not constitute professional, technical, or investment advice. Artificial intelligence is a rapidly evolving field; terminology, techniques, and capabilities change frequently. Coaley Peak Ltd makes no warranty as to the accuracy, completeness, or currency of the information provided. Nothing on this page should be relied upon as the sole basis for commercial, technical, legal, or investment decisions without independent professional advice.

Document reference: ISO_webpage_knowledge-base_glossary_v1

Last modified: 29 March 2026

Knowledge Base·Model Architecture·Tokenisation