16 min read • October 14, 2026 • Episode 1 of 11

AI BasicsLLMsTokensEpisode 1

What Is a Token in AI? How ChatGPT Turns the Internet Into Numbers

Q: How many tokens is a word?

In English, one token is about three quarters of a word, so a word is on average a little more than one token. Common short words are one token each. Rarer or longer words are cut into pieces: "unbelievably" is three tokens in GPT-4's tokenizer, "un", "belie" and "vably".

Q: How many tokens does GPT-4 have?

GPT-4's tokenizer, cl100k_base, has a vocabulary of exactly 100,277 tokens. GPT-2 had 50,257 and GPT-4o's tokenizer, o200k_base, has 200,019.

Q: What is Byte Pair Encoding?

Byte Pair Encoding (BPE) builds a tokenizer by counting. It finds the pair of neighbouring symbols that appears most often in the training text, gives it a new symbol, and repeats until the list reaches the size you want, around 100,000 for GPT-4.

Q: Why can ChatGPT not count the letters in a word like strawberry?

The model reads tokens, not letters. In GPT-4's tokenizer, " strawberry" is a single token, number 73700, so the individual letters are never in front of the model.

Q: Why do some languages use more tokens than English?

A tokenizer's merges follow whatever was common in the text it was built from, and English was common. The same sentence takes 8 tokens in English, 13 in Dutch and 19 in Hungarian in GPT-4's tokenizer. Providers bill per token, so the same task can cost more in some languages. Newer tokenizers narrow the gap.

Q: Is a token the same as a word?

No. A token can be a whole word, part of a word, a single character, a number chunk or a space attached to a letter. On average, English text has about 0.75 words per token.

Q: How can I count tokens in my own text?

Paste the text into Tiktokenizer and choose the model, or use OpenAI's open source tiktoken library. Each model family has its own tokenizer, so counts differ between models.

Q: Who made this video series?

Dominik Gabor, an AI automation consultant at TwinScale AI in Leeuwarden, the Netherlands. The script, research and drawings are his, and the narration is an AI voice (Google Gemini). Every claim is checked against a primary source before an episode goes out.

To GPT-4, "hello world" is not two words. It is two numbers: 15339 and 1917. This is the full written walkthrough of how the internet becomes those numbers, step by step, with the real token counts, by Dominik Gabor.

Dominik Gábor

AI Automation Consultant | TwinScale AI, Leeuwarden

Quick Answer

What is a token in AI?

A token is a chunk of text, often a whole word or part of one, that a large language model reads as a single number. One token is about three quarters of an English word, so 100 tokens is roughly 75 words. To GPT-4, the words "hello world" are two tokens, 15339 and 1917. The model never sees letters, only these numbers.

This is the written version of Episode 1 of Understand LLMs from Zero, the video series I make at TwinScale AI. The video is 8 minutes. This post goes slower and deeper: every step, every number and every drawing from the video, plus the parts that did not fit. If you would rather watch first, here is the 8-minute video on YouTube, and it is also embedded at the end of this post.

I am Dominik Gabor, an AI automation consultant in Leeuwarden, the Netherlands. I started this series because I could not find one clean explanation of how ChatGPT works when I needed it, so I made the one I wanted. Episode 1 starts at the very beginning: what the model reads.

Key Takeaways

A token is a chunk of text read as one number. Often a whole word, often a piece of one.
GPT-4 has exactly 100,277 tokens in its list. Every text it reads or writes is built from those.
The list is built by counting. Byte Pair Encoding merges the most common neighbouring symbols, again and again, until about 100,000 remain.
Tokens explain odd behaviour. Letter counting in "strawberry" and higher cost for Dutch or Hungarian both come from tokenization.
You pay in tokens. The language, the length and the format of what you send change the bill.

The picture at the top is the whole post on one page. Here it is as a table you can search and quote:

Step	What happens	The real number
1. Crawl	Labs copy the public web	2.8 billion pages in one Common Crawl snapshot (Sept 2024)
2. Filter	Spam, menus, duplicates and personal data are removed	About 44 TB of text left (FineWeb)
3. Bytes	Every character becomes bytes	256 possible byte values, "hello" is 104 101 108 108 111
4. Merge	The most common neighbouring pair becomes a new symbol, repeated	23 symbols become 12 after three merges in the example
5. Tokens	The finished list of symbols is the vocabulary	100,277 tokens in GPT-4, 15 trillion tokens to train on

What Is a Token in AI?

A token is the unit a large language model (LLM) reads and writes. It is a chunk of text, often a whole word, sometimes a piece of a word, sometimes a single character or a space with a letter attached. The model turns every chunk into a number, and from that point on it only works with numbers.

Token

A chunk of text, often a whole word or part of one, that a language model reads as a single number. One token is about four characters of English, or three quarters of a word.

Here is GPT-4's tokenizer (OpenAI's cl100k_base, computed with the open source tiktoken library) on the same two words, written three ways:

Text	Tokens	Token numbers	What changed
hello world	2	15339, 1917	Nothing: the plain version
Hello world	2	9906, 1917	A capital H is a different token
hello world (two spaces)	3	15339, 220, 1917	An extra space gets its own token, 220

Look at the second token, 1917. It is not "world". It is " world", with the space glued to the front. Inside the tokenizer, the space belongs to the word that follows it. A capital letter, an extra space or a typo gives the model a different number to work with. Every tiny detail changes the numbers.

Three rows showing hello world, Hello world and hello world with two spaces, each split into token chips with numbers, the changed chip outlined in red — Real tokens in GPT-4's tokenizer. Top: hello world is 15339 and 1917. Middle: a capital H changes the first token to 9906. Bottom: two spaces add a token, 220, in the middle.

Where do these numbers come from, and why these ones? To answer that, we follow the internet all the way down, in order: download it, filter it, turn it into bytes, then merge the bytes into tokens.

Step 1: How AI Labs Download the Internet

Nobody at an AI lab clicks through websites by hand. They start from crawls: automated copies of the public web. The best known source is Common Crawl, a nonprofit that has been collecting web pages since 2007 and shares them for free. One monthly crawl from September 2024 alone holds about 2.8 billion pages.

A grid of web page icons with an arrow to a box reading Common Crawl, 2.8 billion pages — Common Crawl, a nonprofit collecting web pages since 2007. One crawl from September 2024: 2.8 billion pages.

A crawl is a copy of what the web looked like that month. Training a model starts here, because the model has to learn from somewhere, and the public internet is the largest pile of human writing that exists.

Step 2: Why Raw Web Pages Get Filtered

A raw web page is a mess. Think about what sits on a typical page: menus, cookie banners, ads, spam and links to forty other pages. The article you came for is a small part of it.

A raw web page titled How to bake bread, labelled with its menu, an advertisement, a fake prize banner and a cookie banner, and a green box around the article — One recipe page, as the crawler sees it: the useful text is a small part of the page. Train on all of it and the model learns to write cookie banners.

So the labs filter hard. A good public example is FineWeb, a dataset that Hugging Face built and documented in detail. Every page runs through a chain:

URL blocklist. Whole websites, such as adult sites, are thrown out.
Text extraction. The code, menus and navigation are stripped, and the readable text stays.
Language filter. A page stays only when a classifier is at least 65 percent sure it is English.

Quality rules. Pages full of repeated lines, or with barely a real sentence on them, are dropped.
Deduplication. Pages that are copies of each other are removed, because the web repeats itself constantly.
Anonymising. Email addresses and IP addresses are replaced, so those personal details do not end up in the training data.

Six cards in order: URL blocklist, text extraction, language filter with a 65 percent bar, quality rules, deduplication and anonymising — The whole FineWeb chain in order. The language filter keeps a page when the classifier is at least 65 percent sure it is English.

How Much Text Is Left? 44 Terabytes and 15 Trillion Tokens

What survives is about 44 terabytes of text, cleaned out of 96 snapshots of the web. That sounds enormous, but it would fit on a few hard drives you can buy today. Hugging Face also counted it another way: about 15 trillion tokens, counted with GPT-2's tokenizer. A different tokenizer would give a different count, which is a good first hint that the token is a choice, not a law of nature.

Three cards: 96 snapshots of the web, 44 TB of clean text and 15 trillion tokens, the last one outlined in red — FineWeb in numbers: 96 snapshots, about 44 TB of text, about 15 trillion tokens. The last number is the one the rest of the post is about.

Hold on to the word token. Everything that follows is about how 44 terabytes of English becomes 15 trillion of them.

Why a Neural Network Cannot Read Letters

The engine inside every large language model is a neural network, and a neural network cannot read letters. It only does maths. It takes in a row of symbols, one after another, and every symbol must come from a fixed list. So the question is: how do you turn 44 terabytes of text into a row of symbols from a fixed list, and how long should that list be?

A grid of keyboard keys labelled the keys, a question WHICH KEYS, the trade-off between a short list with a long row and a long list with a short row, and a robot labelled neural network — The problem: the network takes in one symbol after another, and every symbol must be one of a fixed set of keys. Which keys?

There is a trade-off hiding in that question. A short list of symbols makes the row very long. A long list makes the row short, but gives the model more symbols to learn. The model can only look at a limited number of symbols at once, so long rows are expensive. The art is to find the best balance. Tokenization is that balance, in two steps: bytes first, then merges.

Bits and Bytes: The First Shortcut

Computers already have one answer. Every character is stored as bits, ones and zeros, using a standard called UTF-8. The word "hello" becomes 40 ones and zeros. That works, but look at the trade: only two symbols, and a very long row.

The word HELLO as five letters, each with its eight bit code, 40 bits in a row — The word hello as bits: 40 ones and zeros. Two symbols, one very long row.

So the first fix is to group the bits, eight at a time. Each group of eight is a byte, and a byte can take 256 different values, from 0 to 255. Now "hello" is just five symbols: 104, 101, 108, 108 and 111. The same text in a row eight times shorter, from a list of 256 symbols instead of 2.

The five letters of hello grouped into bytes with the values 104, 101, 108, 108 and 111 — Eight bits make one byte. Hello is five bytes: 104, 101, 108, 108, 111. A row eight times shorter, and 256 possible symbols instead of 2.

Can we push that trade further? Yes, and this is the clever part.

Byte Pair Encoding Explained With One Sentence

Byte Pair Encoding (BPE) started life as a data compression idea. In 2016, Sennrich, Haddow and Birch showed that it also works for cutting text into subword pieces, and language models have used versions of it ever since. The idea fits in one rule: find the pair of neighbouring symbols that appears most often, give that pair a brand new symbol, and swap it in everywhere.

Try it on a small sentence: "the theme of the thesis". In bytes, that is 23 symbols. Count every pair of neighbours. The pair t followed by h shows up four times, and no pair shows up more often. So it gets a new symbol, number 256, and the row drops from 23 symbols to 19.

The sentence the theme of the thesis as 23 byte symbols with the pair t h ringed in red four times, then the row after replacing it with symbol 256, 19 symbols — Merge one. The most common pair, t then h, appears four times. It becomes symbol 256 and the row drops from 23 symbols to 19.

Then do it again. Now the most common pair is the new symbol followed by an e, which spells "the". It becomes symbol 257, and the row drops to 15. Next, a space followed by "the" becomes symbol 258, and the row is down to 12.

Four rows of the sentence: the start with 23 symbols, then 19, 15 and 12 symbols after each merge — Three merges in a row: 23 symbols down to 19, 15 and then 12. Every merge makes the row shorter and adds one symbol to the list.

Every merge makes the row shorter and adds one symbol to the list. On a real dataset, this runs over billions of pages, and common pieces of words, and whole frequent words, become single symbols. That is why " the" is one token in GPT-4 while a rare word is cut into pieces.

How Many Tokens Does GPT-4 Have?

When do you stop merging? More symbols make the rows shorter but give the model a bigger list to learn. Around 100,000 symbols works well in practice. GPT-4 uses exactly 100,277 of them, and each one is a token.

A bar chart of three tokenizer vocabulary sizes: GPT-2 50,257, GPT-4 100,277 and GPT-4o 200,019 — The size of the list is a design choice. GPT-4's list has 100,277 symbols, and each one is what we call a token.

The number is a design choice, and it has moved over time. Here are three of OpenAI's tokenizers, with the same three sentences run through each (all computed with tiktoken):

Tokenizer	Used by	Vocabulary	English sentence	Dutch sentence	Hungarian sentence
gpt2	GPT-2	50,257	8 tokens	14 tokens	21 tokens
cl100k_base	GPT-4	100,277	8 tokens	13 tokens	19 tokens
o200k_base	GPT-4o	200,019	7 tokens	9 tokens	15 tokens

The sentences are "Artificial intelligence is changing the world.", "Kunstmatige intelligentie verandert de wereld." and "A mesterséges intelligencia megváltoztatja a világot." A bigger list brings the Dutch and Hungarian counts down. More on that below.

What Real Tokens Look Like

Here is one whole sentence through GPT-4's tokenizer: "Tokens are how AI reads the internet." That is eight tokens: Tokens, " are", " how", " AI", " reads", " the", " internet" and the full stop. Most are whole words with the space glued to the front. On average, one token is about three quarters of an English word, which is why AI companies count tokens, not words.

The sentence Tokens are how AI reads the internet split into eight coloured token chips with the numbers 30400, 527, 1268, 15592, 16181, 279, 7757 and 13 — Eight tokens, eight numbers: 30400, 527, 1268, 15592, 16181, 279, 7757 and 13. Rule of thumb: 1 token is about 3/4 of a word.

Does that mean 15339 is bigger or more important than 1917? No. These numbers are names, not amounts. Think of each token as an emoji, one little picture out of a hundred thousand, and the number is just its place in the list. The model has to learn what every token means from how it is used, the same way you learned words by hearing them in sentences.

15 Trillion Tokens: What a Model Actually Trains On

Put it all together. Every page that survived the filters becomes one enormous row of token numbers. For FineWeb, that row is about 15 trillion tokens long. If you read it nonstop, day and night, at 250 words a minute, you would need about 85,000 years.

A kraft coloured page with rows of coloured token numbers and the label 15 trillion tokens long — What the model sees during training: not web pages, not words, just numbers, one after another.

That row is what a model sees during training. Not web pages, not words, just numbers, one after another. A row of numbers is not intelligence yet. Episode 2 shows what the model does with it.

Why ChatGPT Struggles With Letters: The Strawberry Problem

This also explains some strange AI behaviour. Ask a model how many r's are in the word "strawberry", and many models got it wrong. Why? Because the model never saw the letters. In GPT-4's tokenizer, " strawberry" with a space in front is one single token, number 73700. Counting the letters inside it is like counting the strokes inside an emoji you have only ever seen whole.

The word strawberry as ten letters with three r's marked, then as one token chip 73700, then as three chips str, aw and berry — One token, 73700. The letters are not in front of the model, only the chip. Without the leading space the same word is cut into three chunks.

It gets stranger. Without the leading space, "strawberry" is three tokens: "str", "aw" and "berry". The same word is cut differently depending on what sits next to it. Numbers get the same treatment: in GPT-4's tokenizer, "12345678" becomes three tokens, "123", "456" and "78", and "2026" becomes two, "202" and "6". The model reads chunks, and the chunks do not always line up with how a human reads.

Why Some Languages Cost More Tokens

Merges reward whatever was common in the text the tokenizer was built from, and English was common. So English packs into few tokens. "Artificial intelligence is changing the world." is 8 tokens in GPT-4's tokenizer. The Dutch sentence is 13. The Hungarian sentence takes 19. AI companies bill per token, so the same question can cost more depending on the language you ask it in.

Rows of coins for the same sentence in English 8 tokens, Dutch 13 tokens and Hungarian 19 tokens, with the GPT-4o counts 7, 9 and 15 below — The same sentence in three languages: 8 tokens in English, 13 in Dutch, 19 in Hungarian. Each token is a coin on the bill.

There is a byte-level reason too. In UTF-8, plain English letters take one byte each, but an accented letter such as é takes two. Hungarian is full of them. And the merges that would glue those pieces together were learned from a pile of text where Hungarian was rarer. Newer tokenizers narrow the gap, as the table above shows: the same Hungarian sentence drops from 19 tokens in GPT-4 to 15 in GPT-4o, and Dutch from 13 to 9. English went from 8 to 7. One sentence is a tiny sample, so for a real budget, count your own text.

How Your Name Looks to an AI

Names are a good test. My own name, "Dominik Gabor", is 4 tokens in GPT-4's tokenizer: "Domin", "ik", " G" and "abor". The company name "TwinScale AI" is 4 tokens too: "T", "win", "Scale" and " AI". The city I work from, "Leeuwarden", is 4 tokens: "Lee", "u", "ward" and "en". Compare that with "hello world", which is 2.

Token chips for Dominik Gabor, TwinScale AI, Leeuwarden and hello world, each with its token numbers and a count of 4, 4, 4 and 2 tokens — A less common name is cut into pieces. A common phrase like hello world is two tokens.

This is the same rule as before. Whole common words get their own token. A less common name is cut into smaller pieces. The model can still learn a name from its pieces, because it learns from how the pieces appear together, but the name takes more tokens to write down. Even "ChatGPT" is cut: "Chat", "G" and "PT" in GPT-4's tokenizer, two tokens in GPT-4o.

You can try this yourself in two minutes. Open Tiktokenizer, pick a model, and paste your name, your company or a paragraph of your own writing.

What Tokens Mean for Your Business

You do not need to build a model to care about this. Three practical points follow directly from what you just read:

Tokens are the unit you pay for. AI providers price by the token, input and output separately. A long prompt, a long answer and a long document all cost tokens.
Language changes the bill. If your team works in Dutch, Hungarian or German, the same task can use more tokens than in English. Measure with your own text before you budget.
Models work with chunks, not letters. Tasks that depend on exact spelling, letter counts or long strings of digits are risky for a bare model. Give it a tool, or check the result.

Three cards: tokens are the bill, language changes it, chunks and not letters — The three business points of this post on one page.

When I look at where AI would save a team hours, these are the first questions I ask: what is being read, in which language, how often, and what does a mistake cost. If you want a plain-English version of the business side, see why 80% of AI projects fail and the complete guide to AI automation for SMEs.

How to Count Tokens Yourself

You can check any number in this post in under five minutes. Three ways, from easiest to most useful:

Paste it into Tiktokenizer. Pick a model and see the coloured chips and their numbers, exactly like the drawings above.
Use the rule of thumb. One token is about three quarters of an English word. A 3,000-word document is therefore about 4,000 tokens. For Dutch, German or Hungarian, expect noticeably more.
Count in code. Install the open source tiktoken library with pip install tiktoken, then run import tiktoken; print(len(tiktoken.get_encoding("cl100k_base").encode("your text here"))). Use o200k_base for GPT-4o.

Try it on your own prompt, your own product descriptions and your own language before you budget for an AI project. It takes two minutes and removes the guesswork.

Three cards: Tiktokenizer, the rule of thumb of 3,000 words equals 4,000 tokens, and a few lines of Python with tiktoken — Three ways to count your own tokens.

Watch the video

This post follows the video, so here it is: Episode 1, "ChatGPT Can't Read". The animation shows each step moving, which is easier to follow than a still frame. It ends by setting up Episode 2, where the row of numbers goes into a neural network and the model learns by playing one simple game: guess the next token.

Watch Episode 1 on YouTube (8 minutes, no code, no maths). New episodes of Understand LLMs from Zero go up every Wednesday on the TwinScale AI channel.

▸ Free AI Profit Assessment

Want this applied to your business?

If you run a company and want to know where AI would save your team hours, book a Free AI Profit Assessment. 30 minutes, English, no pitch.

The manual work costing you the most hours
The one process worth automating first
An honest answer on what AI can and cannot do there

Book Your Free AI Profit Assessment →

30 minutes. No obligation. No pitch unless you ask for one.

1Name the costly process

2Check what AI can do there

3Pick the first automation

FREE AI PROFIT ASSESSMENT

Sources

Every number in this post comes from one of these, or was computed with tiktoken (the checks live in the series repo).

Penedo et al., The FineWeb Datasets (2024), and the FineWeb dataset card: the filters, the 96 snapshots and the 15 trillion tokens
Common Crawl, September 2024 crawl archive: 2.8 billion pages
Sennrich, Haddow and Birch, Neural Machine Translation of Rare Words with Subword Units (2016): Byte Pair Encoding for subword units
OpenAI, tiktoken: the tokenizers behind every token number in this post (gpt2, cl100k_base, o200k_base)
OpenAI Help Center, What are tokens and how to count them: about four characters, or three quarters of a word, per token
Tiktokenizer: paste your own text and see the tokens
Andrej Karpathy, Deep Dive into LLMs like ChatGPT (2025): the size of the cleaned web text, about 44 TB

FAQ

What is a token in AI?

How many tokens is a word?

How many tokens does GPT-4 have?

What is Byte Pair Encoding?

Why can ChatGPT not count the letters in a word like strawberry?

Why do some languages use more tokens than English?

Is a token the same as a word?

How can I count tokens in my own text?

Who made this video series?

Share this: X (Twitter) LinkedIn Facebook

Keep Reading

Video Series How Do LLMs Work? The Whole Series, From Zero All 11 episodes AI Strategy Why AI Projects Fail: The 6 Causes and the Fix 14 min read AI Automation The Complete Guide to AI Automation for SMEs in 2026 25 min read

About Dominik Gabor

Dominik Gabor is an AI Automation Consultant based in Leeuwarden, the Netherlands, and the founder of TwinScale AI. He has spent 2+ years testing 27+ AI tools and builds AI automations for small and mid-sized companies. He writes and draws the video series Understand LLMs from Zero, with a new episode every Wednesday.

↑ Back to top