
A tokenizer is not a static dictionary but a replayable list of deterministic merge rules that must be applied identically during both training and real-time inference. When early users interacted with large language models, they often noticed a baffling inability to perform simple character-level tasks, such as counting the letter “r” in the word “strawberry” or reversing a basic string










