Tokenizer fertility. The tax a tokenizer levies for the life of a model.

A tokenizer trained on English learns the merges that make English cheap. Feed it anything else and the merges stop applying, so the same amount of content costs several times more tokens. Below are four samples of similar length, measured with one byte level BPE trained on English. Switch the metric, and hover a bar to see why it costs what it does.

One tokenizer is chosen once, before a single parameter is trained, and then it taxes every token the model ever reads or writes. Vocabulary design is quietly an act of policy, deciding in advance which languages the model will find cheap and which it will find expensive.