What is Tokenization?

Tokenization is how AI models chop text into pieces before reading it. The pieces, called tokens, are usually chunks of words rather than whole words. Models see, count, and charge by these tokens, not by words or letters.

A tokenizer carries a fixed vocabulary, typically 50,000 to 200,000 entries, built by finding the character combinations that appear most often in training text. Common words such as “the” survive as single tokens, while rarer ones get split, so unhappiness might become un, happi, and ness. In English this averages out near three-quarters of a word per token, but the same sentence in Japanese or Hindi can cost two or three times more tokens, since the vocabulary was built mostly from English text. That gap matters because API pricing is per million tokens, so the identical question is literally more expensive in some languages. Tokenization also explains a famous failure: a model that can't count the letters in “strawberry” isn't seeing letters at all, it's seeing two or three chunks. Numbers get chopped at odd places too, which is one reason arithmetic trips models up.

Building with LEGO. A castle isn't molded in one piece; it's assembled from a fixed catalog of standard bricks, and the builder only ever thinks in bricks. Ask about the plastic grain inside one brick and you get a blank stare, because the brick is the smallest thing that exists to them.