2 Tokenization, Normalization, and Vocabulary

How should raw clinical language be converted into usable tokens?

Chapter 1 took six clean sentences and turned them into a matrix, and the step that did it — splitting text into words — went past almost without comment. This chapter stops there. Everything downstream depends on that step, and on real clinical text it is neither obvious nor harmless.

The goal is not to survey tokenization in general. It is to look closely at the handful of decisions that shape a document-term matrix built from pathology reports.

These code cells are live

Cells run in your browser and share one session, so run them in order from the top. The first one takes fifteen to thirty seconds while Python loads.

2.1 What is a token?

A token is a unit of text that an NLP system treats as an individual piece of input. A token may correspond to a whole word, such as tumor or melanoma, but it may also be a word part, a number, or punctuation, depending on the tokenizer being used. Tokenization is therefore the process of taking a continuous string of text and breaking it into these smaller units. This step matters because NLP models do not operate directly on raw sentences; they operate on sequences of tokens. Different tokenization choices can produce different representations of the same report, which means that decisions about punctuation, capitalization, hyphenation, and medical expressions can affect what information the model ultimately receives.

Start with a real report and the question becomes immediate.

Read the opening line and decide, before writing any code, what the tokens should be:

A mass is located in right feet skin, with 5x4x2cm, firm, brown, black-gray surface. Microscopic Description: Epidermis is necrotic, ulcerated.

Should skin, become skin and ,, or just skin?
Is 5x4x2cm one token, or a number and a unit, or three numbers and a unit?
Is black-gray one token or two?
Should Description: keep the colon?
Are Epidermis and epidermis the same token?
Should punctuation be a token at all?

There is deliberately no universally correct answer. Different answers suit different tasks, which is the point:

Tokenization is a modeling decision

It is not a mechanical preliminary that happens before the interesting work. It is some of the interesting work. Each answer above changes which columns exist in the document-term matrix, and therefore what the model can possibly learn.

2.2 Tokenizing a report

2.2.1 Splitting on whitespace

The simplest rule: break wherever there is a space.

How the code works:

.split() is built into Python. It is a method that every string already has, so there is nothing to install and nothing to import — you can call it on any piece of text at any time.
With no argument it breaks on any run of whitespace: spaces, tabs, newlines, and runs of several together. Given an argument, such as .split(","), it breaks on that instead.
It returns a plain Python list of strings, which is why len() counts the tokens.
It is a real tokenization algorithm. It is just not a very discerning one.

Look at what it produced. Punctuation stays glued on, so skin, and skin are different tokens, as are surface. and surface, and Description: and Description. Capitalisation is preserved, so Epidermis and epidermis would also be different.

2.2.2 What CountVectorizer does instead

The vectorizer from Chapter 1 has its own built-in rule. You can call it directly, without building a matrix.

How the code works:

build_analyzer() hands back the function the vectorizer would use internally. This is the cleanest way to see what a vectorizer is actually doing to your text.
The counts are close — 154 against 153 — but that near-match is a coincidence, and looking only at totals would hide everything that changed.

2.2.3 Comparing them directly

Three behaviours are worth naming.

It lowercases. Epidermis becomes epidermis, so the two forms collapse into one column instead of two.

It strips punctuation. skin, becomes skin. This is almost always what you want.

It splits on hyphens. black-gray becomes black and gray; nucleo-cytoplasmic becomes nucleo and cytoplasmic. Whether that is what you want is much less obvious.

2.2.4 The one that should worry you

2/9 does not become two tokens. It produces nothing at all.

The default pattern is (?u)\b\w\w+\b — a word boundary, then two or more word characters. Single characters never match. So 2 and 9 are both discarded, and the slash was never a candidate.

In the source report that string means two of nine lymph nodes contained tumor. It is among the most prognostically important facts in the document. After the default tokenizer, no trace of it survives into the matrix.

A preprocessing decision can destroy clinical information

Nothing errored. Nothing warned. A number that changes a patient’s stage simply was not there any more, and the only way to discover that was to look.

This is why tokenization gets a chapter rather than a paragraph.

2.3 Why clinical text is hard

Ordinary English tokenizers are tuned for ordinary English. Clinical writing breaks several of their assumptions at once.

Read the results carefully:

HER2-positive splits, separating a biomarker from its status. her2 and positive in the same report no longer state that one belongs to the other.
ER/PR survives as two tokens, but the pairing is gone.
pT2N0 stays whole — good, but it means pT2N0 and pT2N1 are entirely unrelated columns, sharing nothing despite differing in one character.
0/34 vanishes for the same reason 2/9 did.
G2 survives, 2 alone would not.

Add to this: section headers in capitals, inconsistent punctuation, abbreviations that differ between institutions, and OCR damage from scanning. The report above already contains quitely for quietly, granulaes for granules, and esinophilic for eosinophilic — each becomes its own vocabulary entry.

2.4 Regular Expressions: Describing Patterns in Text

2.4.1 Why another tool is needed

We have just watched the default tokenizer discard 2/9, and we know why: its rule says two or more word characters, and 2 and 9 are each one character.

So we would like to say something the rule cannot express:

keep a digit, a slash, and a digit together as one token

Python’s ordinary string methods cannot say that. .split() breaks on fixed characters. .replace() swaps fixed text. Neither can describe a shape — “some digits, then a slash, then some more digits”, where the digits could be anything.

A regular expression is a small language for describing shapes of text. Learning all of it is a project. Learning six symbols is an afternoon, and six is all this chapter needs.

2.4.2 Six symbols

Build one up from nothing. Start with a pattern that is just literal text.

How the code works:

re is Python’s built-in regular-expression module. Like .split(), it ships with Python — the import just makes it available under a name.
re.findall(pattern, text) returns a list of every place the pattern matched.

Now one symbol at a time.

How the code works:

\d means any single digit. On its own it matches digits one at a time, which is why 20 came back as 2 and 0.
+ means one or more of the thing before it, so \d+ grabs whole runs of digits.
\w means a word character: a letter, a digit, or an underscore.
The r before the quote makes it a raw string, so Python leaves the backslashes alone and passes them to re. Always write patterns as r"...".

Two more, and we are done.

How the code works:

Patterns are read left to right, so \d+mm means digits, then the literal letters mm.
\d+/\d+ is the one we wanted: digits, a slash, digits. That is 2/9, recovered.
(?:...) groups symbols together, and ? after it means optional. So (?:\.\d+)? means possibly a dot followed by digits — which is how 1.5 and 20 both match one pattern.

That is the whole vocabulary used in this chapter:

Symbol	Means
`\d`	any digit
`\w`	any letter, digit, or underscore
`+`	one or more of the previous thing
`?`	the previous thing is optional
`(?:...)`	group these together
`\b`	a word boundary — the edge of a word

\b is the least obvious. It matches the place between a word character and a non-word character, without consuming anything, and it is what stops a pattern matching inside the middle of something longer.

Without the boundaries, 20mm contributes a stray 20. With them, only free-standing numbers match.

2.4.3 Three patterns worth having

Now the practical ones are readable rather than intimidating.

How the code works:

\s is any whitespace, so \s? allows an optional space — this matches both 20mm and 1.5 cm.
| means or, so (?:cm|mm|g) matches any of the three units.
[A-Z] is a character class: any one character from the range. {2,30} is like + but with explicit limits.

2.4.4 Cleaning with re.sub

re.findall finds. re.sub replaces.

How the code works:

\s+ matches any run of whitespace — spaces, tabs, newlines together — and replaces the whole run with one space.
This one line is the most common text-cleaning operation there is. OCR output is full of stray line breaks and doubled spaces, and collapsing them costs nothing.

2.4.5 The payoff

We can now write the tokenizer the default rule would not give us.

How the code works:

The alternatives are tried left to right, so the specific patterns get their chance before the general word rule can break them apart. Order matters.
Adjacent strings in Python are joined automatically, which is why the pattern can be written across several lines with a comment on each.

Compare the two outputs. 2/9 and 20mm survive ours and vanish from the default. The tool was worth the six symbols.

How much regex is enough

Regular expressions get a reputation for being unreadable, and long ones deserve it. But the ones in this chapter are all short, and each is built from the six symbols above.

When you meet a long pattern, read it left to right in pieces rather than trying to take it in at once — that is how it was written.

2.4.6 So what? Getting the pattern into the model

Finding 2/9 in a string is not the goal. The goal is a document-term matrix that a model can learn from, and so far we have only proved that we can find things the default tokenizer misses.

The honest question is: and then what? There are three answers, and they are not equally good. Work through all three on a small set of reports where lymph-node status varies.

Nothing. The single most important fact in these six reports — whether the cancer has spread to the lymph nodes — is not in the matrix at all.

2.4.6.1 Answer 1: keep the pattern as a token

Hand CountVectorizer your own tokenizer. This is what the tokenizer= argument is for, and it is why Appendix A spent a section on passing a function as a value.

How the code works:

tokenizer=keep_raw replaces the built-in rule with ours.
token_pattern=None switches off the default pattern. Without it scikit-learn warns that you have set two conflicting things.

The ratios are in the matrix now. But look at what we bought: six ratio columns for six reports, each appearing exactly once. 2/9 and 3/12 both mean cancer has spread, and the matrix treats them as unrelated. A model cannot generalise from a column it sees once.

This is better than losing the information. It is still nearly useless.

2.4.6.2 Answer 2: normalize the pattern into a shared token

Here is where regular expressions earn their place. Instead of preserving the string, recognise what it means and replace it with a token that recurs.

How the code works:

re.sub rewrites the text before tokenizing. Order matters: the 0/n rule runs first, so 0/34 is claimed as negative before the general rule can call it positive.
The two new columns now appear in three reports each — a signal a model can actually use.

Compare the three vocabularies:

Approach	Columns	Node information
Default	26	absent entirely
Keep raw ratios	28	present, but one column per report
Normalized	19	two columns, each in half the corpus

Normalizing made the vocabulary smaller and the signal stronger at the same time. That is not a coincidence: collapsing many rare forms into one meaningful token is exactly what normalization is for, and it is why Section 2.5 matters.

2.4.6.3 Answer 3: pull it out as a number instead

Sometimes the quantity itself matters, not merely its presence. A token cannot express how many nodes were positive. A number can.

How the code works:

re.search finds the first match, unlike findall which finds all of them.
The parentheses in (\d+)/(\d+) are capture groups: m.group(1) is the text matched by the first pair, m.group(2) by the second. This is how you get the pieces out rather than just the whole match.
The result is not a token at all. It is a numeric column sitting beside the text columns.

And that is precisely the idea from Section 1.3.4: a feature is any function from a document to a number. nodes_positive is such a function. So is tumor_mm. Neither is a word count, and both can go into the same matrix as the word counts.

2.4.6.4 Which to use

Situation	Approach
The pattern’s exact form matters and repeats	keep it as a token
Many surface forms mean the same thing	normalize to a shared token
The magnitude matters, or you want to compare or threshold it	extract a number

Most real pipelines use all three. The point of this section is that regular expressions are not for admiring patterns in text — they are for deciding, deliberately, what reaches the matrix.

2.5 Normalization

Normalization reduces superficial variation so that forms meaning the same thing land in the same column.

Five forms become one. That is five columns collapsed into one, each with more evidence behind it.

But the same operation that merges Carcinoma and carcinoma also merges things you may not want merged:

ER is a hormone receptor; er is noise. US may be ultrasound; us is a pronoun. Lowercasing throws that distinction away everywhere, to gain consistency everywhere.

Normalization reduces variation, but it can also remove information

There is no setting that is right in general. There is only a trade-off you should make deliberately, having looked at what it costs on your own text.

2.6 Stopwords and clinical negation

Stopwords are very frequent words — the, of, is, with, no, not — that carry little meaning on their own. Removing them is standard practice, and it shrinks the vocabulary considerably.

In clinical text it is dangerous.

Look at the pairs. Each pair states opposite clinical findings, and after stopword removal each pair is identical. A model cannot distinguish what its input no longer distinguishes.

Frequent words are not automatically useless words

no, not and without are among the most common words in English and among the most consequential words in a pathology report. The general-purpose stopword list was not built with that in mind.

We met this already in Section 1.5.2, where no was dropped for being out-of-vocabulary. Here it is dropped on purpose. The result is the same.

2.7 From tokens to a vocabulary

Some terms, used precisely from here on.

Term	Meaning
Corpus	The whole collection of documents
Document	One report
Token instance	One occurrence of a token
Token type	One distinct token, however often it occurs
Vocabulary	The set of all token types in the corpus
Vocabulary size	How many types — the number of columns
Token frequency	How many times a type occurs in total
Document frequency	In how many documents a type occurs at all

The instance-versus-type distinction is the one that trips people up.

How the code works:

Counter tallies a list into a mapping of item to count.
len(tokens) counts occurrences; len(counts) counts distinct types. The gap between them is repetition.

And this is the direct link back to Chapter 1:

each token type becomes one column of the document-term matrix,
each document becomes one row,
so preprocessing decides which columns exist at all.

Change the tokenizer and you change the matrix. Not the numbers in it — its shape.

2.8 Inspecting a real corpus

Statistics about text mean little until you have looked at the text. Work outward: one report, then a few, then the whole thing.

2.8.1 Stage 1: one report

Notice how many of the frequent tokens are structural — tumor, not, specified — rather than diagnostic. A checklist-style report repeats its own scaffolding.

2.8.2 Stage 2: a small corpus

Ten real de-identified TCGA reports, chosen short enough to read.

How the code works:

.str.len() applies a string operation down a whole column.
.apply(lambda ...) runs any function on every value — the pattern from

Reports differ in length by a factor of two here, and the two token counts never agree. The ratio between them is not even constant: a report thick with numbers and ratios loses proportionally more.

Now compare the vocabularies of two reports:

Two reports on the same organ share their clinical core — adrenocortical, carcinoma, weiss — while differing in scores, measurements and phrasing. That shared core is what a classifier can learn from. The differences are partly real and partly institutional habit, and telling those apart is a large part of the work.

How the code works:

The nested comprehension flattens a list of lists into one long list of tokens.
Even in eight short reports, most of the vocabulary occurs exactly once.

2.8.3 Stage 3: the whole corpus

Nobody reads 9,523 reports. You measure them. These are the real figures for the TCGA corpus this book uses:

Measure	Value
Reports	9,523
Characters per report, median	2,854
Characters per report, longest	26,162
Whitespace tokens, mean	560
Whitespace tokens, median	432
Whitespace tokens, 75th percentile	808
Whitespace tokens, longest report	4,046
Whitespace tokens, shortest report	1

Two of those deserve a second look.

The median is well below the mean — 432 against 560 — which is the signature of a right-skewed distribution: most reports are ordinary, a few are enormous, and the long ones drag the average up. Quoting only a mean would misdescribe the corpus.

And the shortest report in the corpus is one token long. Its entire text is 1. — a document that survived scanning and OCR and made it into the dataset carrying no information whatsoever. Every real corpus has some of these. Finding them is part of inspecting your data rather than trusting it.

2.9 Rare words and out-of-vocabulary terms

The singleton count from the small corpus was not a small-sample artifact. It is what natural language does: a few words occur constantly, and a very long tail occurs once.

Rare terms come from several sources, and they need different responses:

specialised vocabulary — adrenocortical, lymphovascular: rare but meaningful;
abbreviations — LN, EC, NOS: rare and meaningful, if you know them;
measurements — 6.5cm, 172g: each numerically unique, rarely useful as a token;
OCR damage — quitely, granulaes: rare and worthless;
spelling variation — the same idea written three ways across three institutions.

A fixed word vocabulary has no answer for any of them. It is built once from the training corpus, and anything not in it becomes out-of-vocabulary — silently discarded, exactly as nephrectomy was in Chapter 1.

The more specialised the word, the more likely it is to be rare, and the more likely a whole-word vocabulary is to drop it. That is a poor trade in a technical domain.

2.10 Subword tokenization

The way out is to stop insisting that a token be a whole word.

Medical vocabulary is built from reusable parts, and those parts recur across words that a whole-word vocabulary treats as unrelated:

adenocarcinoma   ->  adeno + carcin + oma
adenoma          ->  adeno + oma
carcinoma        ->  carcin + oma
nephrectomy      ->  nephr + ectomy
nephritis        ->  nephr + itis

A vocabulary of pieces rather than words has two advantages. It is smaller, because pieces are shared. And it has no true out-of-vocabulary case: an unseen word is still representable, because its pieces have been seen.

How the code works:

At each position it takes the longest piece it recognises, then continues.
Unrecognised characters fall through singly — which is why nephroblastoma comes out partly as loose letters. A real subword vocabulary is far larger and leaves fewer gaps.

This is a caricature of the real algorithms, but it shows the essential move: a word the vocabulary has never seen is still represented, because its parts have been.

2.11 Byte-pair encoding

Where do the pieces come from? They are learned from the corpus, not supplied by a linguist.

Byte-pair encoding (BPE) is the usual method, and the idea is short enough to state in full:

start with every individual character as a piece;
count every adjacent pair of pieces across the corpus;
merge the most frequent pair into a single new piece;
repeat until the vocabulary reaches the size you asked for.

How the code works:

Frequent letter pairs are merged first, so common fragments assemble themselves out of the data.
Run this for thousands of merges over a real corpus and the pieces come to look like morphemes — not because anyone taught it morphology, but because frequent sequences are frequent for a reason.

That is enough for now. What matters later is that BERT-style and BigBird-style models tokenize this way, so their idea of a token is not your idea of a word.

2.12 Tokenization and model input length

A report does not have one length. It has as many lengths as there are ways to count it.

This matters because models have hard input limits, and the limit is counted in the model’s own tokens — subword pieces, not words.

Model	Maximum input
BERT / ClinicalBERT	512 tokens
Clinical-BigBird	4,096 tokens

Now put that against the corpus. Measured on the 9,523 reports, 42.7% exceed 512 whitespace-separated words. But subword tokenization produces more tokens than whitespace splitting, because long medical words get broken into pieces — so the share exceeding 512 model tokens is higher still. The study this corpus comes from reports that more than 66% of documents pass ClinicalBERT’s limit, which is what motivated moving to Clinical-BigBird and its far longer input.

The consequence is blunt. What does not fit is truncated — silently cut off at the limit. And a pathology report puts its diagnosis wherever the institution’s template puts it, which is often near the end.

Truncation is not a neutral operation

A model that never sees the final section of a report cannot use what the final section says. Before choosing a model, measure how long your documents are in that model’s tokens — not in words, and not in characters.

This is the bridge to the chapters on contextual embeddings and transformers, where the tokenizer stops being a preprocessing choice you make and becomes part of the model you adopt.

2.13 Exercises

2.13.1 Predict before you run

# Two rules explain almost everything you will see:
#   1. text is lowercased and split on anything that is not a word character
#   2. only runs of TWO OR MORE word characters survive
# So single digits and single letters disappear entirely.

2.13.2 Write a tokenizer that keeps what matters

# Build one pattern with | alternatives, and put the SPECIFIC patterns first,
# because re.findall tries them left to right:
#     ratios       \b\d+/\d+\b
#     measurements \b\d+(?:\.\d+)?(?:cm|mm|g)\b
#     hyphenated   \b\w+(?:-\w+)+\b
#     plain words  \b\w{2,}\b

2.13.3 Count a corpus

2.14 Summary

Tokenization defines the units everything downstream is built from. It is a modeling decision, not a mechanical preliminary.
Clinical text breaks ordinary tokenizers. Hyphenated biomarkers, staging strings, measurements and node ratios each fail differently — and 2/9 disappears completely under the default rule.
Normalization reduces variation but can remove information. Lowercasing merges Carcinoma with carcinoma, and also ER with er.
Stopword removal is dangerous here. no evidence of metastasis and evidence of metastasis become identical, and they mean opposite things.
A corpus induces a vocabulary, and that vocabulary is the set of columns in the document-term matrix. Change the tokenizer, change the matrix.
Most of the vocabulary occurs once. Rare and specialised terms are precisely the informative ones, and a fixed whole-word vocabulary discards them.
Subword tokenization removes the out-of-vocabulary problem by representing unseen words as sequences of seen pieces, learned from data by algorithms such as BPE.
Tokenization determines input length, and input length runs into hard model limits. What does not fit is truncated, and truncation is not neutral.

The next chapters take these representations and start building models on them.