machine learning1 min read
I animated what GPT-4o's tokenizer does to "strawberry": real token ids, a tiny BPE run, and why the space belongs to the word (6 min video) [D]
I made this explainer for my small channel and would like feedback from people learning this, especially if anything is wrong.
By /u/RajucgReddit - r/ML
Read the full story
Originally published by Reddit - r/ML on Nov 1, 2026.
Comments
Sign in to join the discussion.