Tokens

What are Tokens?

Tokens are the units of information that large language models process. A token may be a whole word, part of a word, punctuation or another small piece of text. Before an LLM can work with language, text is converted into tokens that the model represents mathematically and uses to predict what is most likely to come next.

Tokens matter because they are effectively the working units of an LLM. Context-window limits, processing costs and the amount of material a model can consider at one time are commonly measured in tokens rather than words. For leaders, the concept provides a useful bridge between human language and the statistical machinery of generative AI.

Back to the glossary