3  Tokenization, Normalization, and Vocabulary

How should raw clinical language be converted into usable tokens?

Chapter 1 took six clean sentences and turned them into a matrix, and Chapter 2 did the same to the real reports — where a look inside the nearest-neighbor matches showed many of the matching words to be filler such as the, of and is. The step that produced those words — splitting text into words — went past almost without comment in both chapters. This chapter stops there. Everything downstream depends on that step, and on real clinical text it is neither obvious nor harmless.

The goal is not to survey tokenization in general. It is to look closely at the handful of decisions that shape a document-term matrix built from pathology reports.

ImportantThese code cells are live

Cells run in your browser and share one session, so run them in order from the top. The first one takes fifteen to thirty seconds while Python loads.

3.1 Splitting text into tokens

Every chapter so far has let CountVectorizer decide where one word ends and the next begins. This part looks at that decision: first on one real report, then on the kinds of strings clinical text is full of.

3.1.1 What is a token?

A token is a unit of text that an NLP system treats as an individual piece of input. A token may correspond to a whole word, such as tumor or melanoma, but it may also be a word part, a number, or punctuation, depending on the tokenizer being used. Tokenization is therefore the process of taking a continuous string of text and breaking it into these smaller units. This step matters because NLP models do not operate directly on raw sentences; they operate on sequences of tokens. Different tokenization choices can produce different representations of the same report, which means that decisions about punctuation, capitalization, hyphenation, and medical expressions can affect what information the model ultimately receives.

Start with a real report and the question becomes immediate.

Read the opening line and decide, before writing any code, what the tokens should be:

A mass is located in right feet skin, with 5x4x2cm, firm, brown, black-gray surface. Microscopic Description: Epidermis is necrotic, ulcerated.

• Should skin, become skin and ,, or just skin?
• Is 5x4x2cm one token, or a number and a unit, or three numbers and a unit?
• Is black-gray one token or two?
• Should Description: keep the colon?
• Are Epidermis and epidermis the same token?
• Should punctuation be a token at all?

There is deliberately no universally correct answer. Different answers suit different tasks, which is the point:

NoteTokenization is a modeling decision

It is not a mechanical preliminary that happens before the interesting work. It is some of the interesting work. Each answer above changes which columns exist in the document-term matrix, and therefore what the model can possibly learn.

3.1.2 Tokenizing a report

3.1.2.1 Splitting on whitespace

The simplest rule: break wherever there is a space.

How the code works:

• .split() is built into Python. It is a method that every string already has, so there is nothing to install and nothing to import — you can call it on any piece of text at any time.
• With no argument it breaks on any run of whitespace: spaces, tabs, newlines, and runs of several together. Given an argument, such as .split(","), it breaks on that instead.
• It returns a plain Python list of strings, which is why len() counts the tokens.
• It is a real tokenization algorithm. It is just not a very discerning one.

Look at what it produced. Punctuation stays glued on, so skin, and skin are different tokens, as are surface. and surface, and Description: and Description. Capitalisation is preserved, so Epidermis and epidermis would also be different.

3.1.2.2 What CountVectorizer does instead

The vectorizer from Chapter 1 has its own built-in rule. You can call it directly, without building a matrix.

How the code works:

• build_analyzer() hands back the function the vectorizer would use internally. This is the cleanest way to see what a vectorizer is actually doing to your text.
• The counts are close — 154 against 153 — but that near-match is a coincidence, and looking only at totals would hide everything that changed.

3.1.2.3 Comparing them directly

Three behaviours are worth naming.

It lowercases. Epidermis becomes epidermis, so the two forms collapse into one column instead of two.

It strips punctuation. skin, becomes skin. This is almost always what you want.

It splits on hyphens. black-gray becomes black and gray; nucleo-cytoplasmic becomes nucleo and cytoplasmic. Whether that is what you want is much less obvious.

3.1.2.4 The one that should worry you

2/9 does not become two tokens. It produces nothing at all.

The default pattern is (?u)\b\w\w+\b — a word boundary, then two or more word characters. Single characters never match. So 2 and 9 are both discarded, and the slash was never a candidate.

In the source report that string means two of nine lymph nodes contained tumor. It is among the most prognostically important facts in the document. After the default tokenizer, no trace of it survives into the matrix.

WarningA preprocessing decision can destroy clinical information

Nothing errored. Nothing warned. A number that changes a patient’s stage simply was not there any more, and the only way to discover that was to look.

This is why tokenization gets a chapter rather than a paragraph.

3.1.3 Why clinical text is hard

Ordinary English tokenizers are tuned for ordinary English. Clinical writing breaks several of their assumptions at once.

Read the results carefully:

• HER2-positive splits, separating a biomarker from its status. her2 and positive in the same report no longer state that one belongs to the other.
• ER/PR survives as two tokens, but the pairing is gone.
• pT2N0 stays whole — good, but it means pT2N0 and pT2N1 are entirely unrelated columns, sharing nothing despite differing in one character.
• 0/34 vanishes for the same reason 2/9 did.
• G2 survives, 2 alone would not.

Add to this: section headers in capitals, inconsistent punctuation, abbreviations that differ between institutions, and OCR damage from scanning. The report above already contains quitely for quietly, granulaes for granules, and esinophilic for eosinophilic — each becomes its own vocabulary entry.

3.2 Regular Expressions: Describing Patterns in Text

3.2.1 Why another tool is needed

We have just watched the default tokenizer discard 2/9, and we know why: its rule says two or more word characters, and 2 and 9 are each one character.

So we would like to say something the rule cannot express:

keep a digit, a slash, and a digit together as one token

Python’s ordinary string methods cannot say that. .split() breaks on fixed characters. .replace() swaps fixed text. Neither can describe a shape — “some digits, then a slash, then some more digits”, where the digits could be anything.

A regular expression is a small language for describing shapes of text. Learning all of it is a project. Learning six symbols is an afternoon, and six is all this chapter needs.

Keep regex101.com open in another tab while you read. You type a pattern in one box and some sample text in another, and it does three things Python does not:

• It highlights every match as you type, so you see at once what a change to the pattern catches and what it misses, without rerunning a cell.
• It explains the pattern in plain English, symbol by symbol, in a panel beside it. When a pattern from this chapter, or one you find online, is hard to read, paste it in.
• It lists each match and the parts inside it, which helps when a pattern matches more or less than you meant.

Choose Python from the list of flavors on the left, because the details of regular expressions differ slightly between languages. Paste in a few lines of a report, not a whole one, and never paste in text you would not post publicly.

3.2.2 Six symbols

Build one up from nothing. Start with a pattern that is just literal text.

How the code works:

• re is Python’s built-in regular-expression module. Like .split(), it ships with Python — the import just makes it available under a name.
• re.findall(pattern, text) returns a list of every place the pattern matched.

Now one symbol at a time.

How the code works:

• \d means any single digit. On its own it matches digits one at a time, which is why 20 came back as 2 and 0.
• + means one or more of the thing before it, so \d+ grabs whole runs of digits.
• \w means a word character: a letter, a digit, or an underscore.
• The r before the quote makes it a raw string, so Python leaves the backslashes alone and passes them to re. Always write patterns as r"...".

Two more, and we are done.

How the code works:

• Patterns are read left to right, so \d+mm means digits, then the literal letters mm.
• \d+/\d+ is the one we wanted: digits, a slash, digits. That is 2/9, recovered.
• (?:...) groups symbols together, and ? after it means optional. So (?:\.\d+)? means possibly a dot followed by digits — which is how 1.5 and 20 both match one pattern.

That is the whole vocabulary used in this chapter:

Symbol	Means
\d	any digit
\w	any letter, digit, or underscore
+	one or more of the previous thing
?	the previous thing is optional
(?:...)	group these together
\b	a word boundary — the edge of a word

\b is the least obvious. It matches the place between a word character and a non-word character, without consuming anything, and it is what stops a pattern matching inside the middle of something longer.

Without the boundaries, 20mm contributes a stray 20. With them, only free-standing numbers match.

3.2.3 Three patterns worth having

Now the practical ones are readable rather than intimidating. Try them on a few lines in the style of the melanoma report.

How the code works:

• \s is any whitespace, so \s? allows an optional space — this matches both 20mm and 1.5 cm.
• | means or, so (?:cm|mm|g) matches any of the three units.
• [A-Z] is a character class: any one character from the range. {2,30} is like + but with explicit limits.
• 5x4x2cm comes back as just 2cm. The pattern describes one number and a unit, so that is all it finds. A pattern matches what you wrote, not what you meant.

3.2.4 Cleaning with re.sub

re.findall finds. re.sub replaces.

How the code works:

• \s+ matches any run of whitespace — spaces, tabs, newlines together — and replaces the whole run with one space.
• This one line is the most common text-cleaning operation there is. OCR output is full of stray line breaks and doubled spaces, and collapsing them costs nothing.

3.2.5 The payoff

We can now write the tokenizer the default rule would not give us.

How the code works:

• The alternatives are tried left to right, so the specific patterns get their chance before the general word rule can break them apart. Order matters.
• Adjacent strings in Python are joined automatically, which is why the pattern can be written across several lines with a comment on each.

Compare the two outputs. 2/9 and 20mm survive ours and vanish from the default. The tool was worth the six symbols.

NoteHow much regex is enough

Regular expressions get a reputation for being unreadable, and long ones deserve it. But the ones in this chapter are all short, and each is built from the six symbols above.

When you meet a long pattern, read it left to right in pieces rather than trying to take it in at once — that is how it was written.

3.2.6 So what? Getting the pattern into the model

Finding 2/9 in a string is not the goal. The goal is a document-term matrix that a model can learn from, and so far we have only proved that we can find things the default tokenizer misses.

The honest question is: and then what? There are three answers, and they are not equally good. Work through all three on a small set of reports where lymph-node status varies. Run the next cell first; the three answers below all use its node_reports.

Nothing. The single most important fact in these six reports — whether the cancer has spread to the lymph nodes — is not in the matrix at all.

3.2.6.1 Answer 1: keep the pattern as a token

Hand CountVectorizer your own tokenizer. This is what the tokenizer= argument is for, and it is why Appendix A spent a section on passing a function as a value.

How the code works:

• tokenizer=keep_raw replaces the built-in rule with ours.
• token_pattern=None switches off the default pattern. Without it scikit-learn warns that you have set two conflicting things.

The ratios are in the matrix now. But look at what we bought: six ratio columns for six reports, each appearing exactly once. 2/9 and 3/12 both mean cancer has spread, and the matrix treats them as unrelated. A model cannot generalise from a column it sees once.

This is better than losing the information. It is still nearly useless.

3.2.6.2 Answer 2: normalize the pattern into a shared token

Here is where regular expressions earn their place. Instead of preserving the string, recognise what it means and replace it with a token that recurs.

How the code works:

• re.sub rewrites the text before tokenizing. Order matters: the 0/n rule runs first, so 0/34 is claimed as negative before the general rule can call it positive.
• The two new columns now appear in three reports each — a signal a model can actually use.

Compare the three vocabularies:

Approach	Columns	Node information
Default	26	absent entirely
Keep raw ratios	28	present, but one column per report
Normalized	19	two columns, each in half the corpus

Normalizing made the vocabulary smaller and the signal stronger at the same time. That is not a coincidence: collapsing many rare forms into one meaningful token is exactly what normalization is for, and it is why Section 3.3.1 matters.

3.2.6.3 Answer 3: pull it out as a number instead

Sometimes the quantity itself matters, not merely its presence. A token cannot express how many nodes were positive. A number can.

How the code works:

• re.search finds the first match, unlike findall which finds all of them.
• The parentheses in (\d+)/(\d+) are capture groups: m.group(1) is the text matched by the first pair, m.group(2) by the second. This is how you get the pieces out rather than just the whole match.
• The result is not a token at all. It is a numeric column sitting beside the text columns.

And that is precisely the idea from Section 1.3.4: a feature is any function from a document to a number. nodes_positive is such a function. So is tumor_mm. Neither is a word count, and both can go into the same matrix as the word counts.

3.2.6.4 Which to use

Situation	Approach
The pattern’s exact form matters and repeats	keep it as a token
Many surface forms mean the same thing	normalize to a shared token
The magnitude matters, or you want to compare or threshold it	extract a number

Most real pipelines use all three. The point of this section is that regular expressions are not for admiring patterns in text — they are for deciding, deliberately, what reaches the matrix.

3.3 Cleaning: what to merge, what to drop

Once the text is split, two more decisions shape the columns: which tokens count as the same (normalization), and which do not count at all (stopwords). Both make the vocabulary smaller, and both can remove meaning.

3.3.1 Normalization

Normalization reduces superficial variation so that forms meaning the same thing land in the same column.

Five forms become one. That is five columns collapsed into one, each with more evidence behind it.

3.3.1.1 What CountVectorizer() already does

Every result in Chapter 2 was already normalized, because CountVectorizer() normalizes by default. Its settings say so:

Two kinds of normalization are on:

• It lowercases (lowercase=True). Carcinoma, CARCINOMA and carcinoma all land in one column.
• Its token pattern keeps only runs of two or more letters or digits. Punctuation goes, so positive; becomes positive. Hyphens and slashes split words, so Ki-67 becomes ki and 67. Single characters vanish, which is why 5% and 2/9 leave nothing behind (the problem met in “The one that should worry you” above).

Several things are off:

• Accents are kept. strip_accents=None, so café and cafe stay apart.
• No stopwords are removed. stop_words=None.
• No stemming. Nothing turns nodes into node.
• OCR misspellings stay separate. A scanning error such as carcinma keeps its own column.

One more point, because the word is used in two senses. In machine learning, “normalizing” can also mean rescaling every row of the matrix to the same length, so that a long report and a short one are on an equal footing. CountVectorizer() does not do that: its rows are raw counts, and a long report has bigger numbers. In Chapter 2, the rescaling happened in the comparison instead. Cosine similarity looks only at the direction of each row, never its length. That is part of why Chapter 2 used it. On the real corpus, nearest neighbors with plain (Euclidean) distance on the same counts scores 0.815 instead of 0.853.

3.3.1.2 What it bought on the real corpus

How much did the default normalization actually contribute to Chapter 2’s results? Here is the same nearest-neighbor method (one neighbor, cosine) run three ways, on both of Chapter 2’s splits. Each split builds the vocabulary from its own training reports.

Tokens built by	Vocabulary	Hospitals it has seen	Hospitals it has never seen
splitting on spaces only; nothing normalized	127,909	0.785	0.154
CountVectorizer(lowercase=False)	50,228	0.826	0.257
CountVectorizer() (the default)	38,839	0.853	0.411

The two normalizations together cut the vocabulary by more than two thirds. For hospitals it has seen, they add 7 points. For hospitals it has never seen, they nearly triple the score, and lowercasing alone is worth 15 points of that.

The reason is the hospital effect from Section 2.7. Hospitals have house styles, and case is part of a house style: 31 of the 686 sites write their reports mostly in capitals. If case is kept, CARCINOMA and carcinoma are different columns, and a report written in capitals can only resemble other reports written in capitals. In practice that means reports from the same hospital. Lowercasing erases that fingerprint, so the comparison has to rely more on what the words say and less on how a department types them. The method doesn’t really need it for hospitals it has seen, because it can match their templates anyway. It does need it for a hospital it has never seen.

3.3.1.3 What it costs

The same operation that merges Carcinoma and carcinoma also merges things you may not want merged:

ER is a hormone receptor; er is noise. US may be ultrasound; us is a pronoun. Lowercasing throws that distinction away everywhere, to gain consistency everywhere. Add the losses from the token pattern above: the dropped 2/9, 5% and other single characters, and the split-up Ki-67.

How much did the lost capitals cost on the cancer-type task? Almost nothing. Keeping ER, PR, HER2, US and AS apart from their lowercase twins (the remedy below) gives 0.852 and 0.411, against 0.853 and 0.411 without it. That isn’t because the distinction is unimportant. Cancer type simply doesn’t hinge on it: ER turns up in breast reports whether or not it is merged with er. On a task that does hinge on it, such as reading off a breast tumor’s hormone-receptor status, the same merge could be the whole problem. A cost that doesn’t show up in one task’s accuracy can still be real.

3.3.1.4 Remedies

None of these pitfalls forces a choice between “normalize everything” and “normalize nothing”. Each has a targeted remedy.

Protect the few capitals that matter, then lowercase the rest. CountVectorizer accepts a preprocessor=, a function applied to each report before tokenizing. One that renames a short, hand-picked list of abbreviations before lowercasing keeps most of the benefit and avoids the specific merges you care about:

The default gives one column, us, for both the scan and the pronoun. The protected version keeps us_abbr apart from us.

The other pitfalls have remedies of their own:

• Keep the characters the token pattern throws away. Changing the token pattern, or rewriting 2/9 and 5% into tokens before counting, is exactly what “Regular Expressions” above did with its three answers.
• For scanning errors, change what counts as a piece. Lowercasing can’t merge carcinma with carcinoma. Character n-grams (Section 3.5.3) can, because the two words share most of their short runs of letters.
• Decide by measuring, on a validation set. Whether to lowercase, which abbreviations to protect and which pattern to use are choices like k in Section 2.4. Try them on the validation reports and keep what helps. Never choose by looking at the test set, and never assume the default is right because it is the default. Here the measurement says the default is a very good choice for cancer type, and that its cost will show up on other tasks.

WarningNormalization reduces variation, but it can also remove information

There is no setting that is right in general. There is only a trade-off you should make deliberately, having looked at what it costs on your own text.

3.3.2 Stopwords and clinical negation

Stopwords are very frequent words — the, of, is, with, no, not — that carry little meaning on their own. Removing them is standard practice, and it shrinks the vocabulary considerably.

In clinical text it is dangerous.

Look at the pairs. Each pair states opposite clinical findings, and after stopword removal each pair is identical. A model cannot distinguish what its input no longer distinguishes.

WarningFrequent words are not automatically useless words

no, not and without are among the most common words in English and among the most consequential words in a pathology report. The general-purpose stopword list was not built with that in mind.

We met this already in Section 1.5.2, where no was dropped for being out-of-vocabulary. Here it is dropped on purpose. The result is the same.

3.4 Vocabulary: what becomes a column

The tokens that survive become the columns of the document-term matrix. This part names the pieces, looks at a real corpus in stages, and meets its long tail of rare words, including words the training reports never used.

3.4.1 From tokens to a vocabulary

Some terms, used precisely from here on.

Term	Meaning
Corpus	The whole collection of documents
Document	One report
Token instance	One occurrence of a token
Token type	One distinct token, however often it occurs
Vocabulary	The set of all token types in the corpus
Vocabulary size	How many types — the number of columns
Token frequency	How many times a type occurs in total
Document frequency	In how many documents a type occurs at all

The instance-versus-type distinction is the one that trips people up.

How the code works:

• Counter tallies a list into a mapping of item to count.
• len(tokens) counts occurrences; len(counts) counts distinct types. The gap between them is repetition.

And this is the direct link back to Chapter 1:

• each token type becomes one column of the document-term matrix,
• each document becomes one row,
• so preprocessing decides which columns exist at all.

Change the tokenizer and you change the matrix. Not the numbers in it — its shape.

3.4.2 Inspecting a real corpus

Statistics about text mean little until you have looked at the text. Work outward: one report, then a few, then the whole thing.

3.4.2.1 Stage 1: one report

Notice how many of the frequent tokens are structural — tumor, not, specified — rather than diagnostic. A checklist-style report repeats its own scaffolding.

3.4.2.2 Stage 2: a small corpus

Eight real de-identified TCGA reports, chosen short enough to read.

How the code works:

• .str.len() applies a string operation down a whole column.
• .apply(lambda ...) runs any function on every value — the pattern from

Reports differ in length by a factor of two here, and the two token counts never agree. The ratio between them is not even constant: a report thick with numbers and ratios loses proportionally more.

Now compare the vocabularies of two reports:

Two reports on the same organ share their clinical core — adrenocortical, carcinoma, weiss — while differing in scores, measurements and phrasing. That shared core is what a classifier can learn from. The differences are partly real and partly institutional habit, and telling those apart is a large part of the work.

How the code works:

• The nested comprehension flattens a list of lists into one long list of tokens.
• Even in eight short reports, most of the vocabulary occurs exactly once.

3.4.2.3 Stage 3: the whole corpus

Nobody reads 9,523 reports. You measure them. These are the real figures for the TCGA corpus this book uses:

Measure	Value
Reports	9,523
Characters per report, median	2,854
Characters per report, longest	26,162
Whitespace tokens, mean	560
Whitespace tokens, median	432
Whitespace tokens, 75th percentile	808
Whitespace tokens, longest report	4,046
Whitespace tokens, shortest report	1

Two of those deserve a second look.

The median is well below the mean — 432 against 560 — which is the signature of a right-skewed distribution: most reports are ordinary, a few are enormous, and the long ones drag the average up. Quoting only a mean would misdescribe the corpus.

And the shortest report in the corpus is one token long. Its entire text is 1. — a document that survived scanning and OCR and made it into the dataset carrying no information whatsoever. Every real corpus has some of these. Finding them is part of inspecting your data rather than trusting it.

3.4.3 Rare words and out-of-vocabulary terms

The singleton count from the small corpus was not a small-sample artifact. It is what natural language does: a few words occur constantly, and a very long tail occurs once.

Rare terms come from several sources, and they need different responses:

• specialised vocabulary — adrenocortical, lymphovascular: rare but meaningful;
• abbreviations — LN, EC, NOS: rare and meaningful, if you know them;
• measurements — 6.5cm, 172g: each numerically unique, rarely useful as a token;
• OCR damage — quitely, granulaes: rare and worthless;
• spelling variation — the same idea written three ways across three institutions.

A fixed word vocabulary has no answer for any of them. It is built once from the training corpus, and anything not in it becomes out-of-vocabulary — silently discarded, exactly as nephrectomy was in Chapter 1.

The more specialised the word, the more likely it is to be rare, and the more likely a whole-word vocabulary is to drop it. That is a poor trade in a technical domain.

3.5 Tokens smaller or larger than a word

A whole word is not the only possible unit. Pieces of words cope with rare and misspelled words; pairs of words keep phrases such as no evidence together. The unit also decides how long a report is, which matters once a model has a length limit.

3.5.1 Subword tokenization

The way out is to stop insisting that a token be a whole word.

Medical vocabulary is built from reusable parts, and those parts recur across words that a whole-word vocabulary treats as unrelated:

adenocarcinoma   ->  adeno + carcin + oma
adenoma          ->  adeno + oma
carcinoma        ->  carcin + oma
nephrectomy      ->  nephr + ectomy
nephritis        ->  nephr + itis

A vocabulary of pieces rather than words has two advantages. It is smaller, because pieces are shared. And it has no true out-of-vocabulary case: an unseen word is still representable, because its pieces have been seen.

How the code works:

• At each position it takes the longest piece it recognises, then continues.
• Unrecognised characters fall through singly — which is why nephroblastoma comes out partly as loose letters. A real subword vocabulary is far larger and leaves fewer gaps.

This is a caricature of the real algorithms, but it shows the essential move: a word the vocabulary has never seen is still represented, because its parts have been.

3.5.2 Byte-pair encoding

Where do the pieces come from? They are learned from the corpus, not supplied by a linguist.

Byte-pair encoding (BPE) is the usual method. It is a tokenization technique, widely used in natural language processing, that breaks text into smaller units called tokens. Rather than treating every word as a separate unit, BPE starts with individual characters (or bytes) and repeatedly merges pairs that often occur side by side into larger tokens. For example, the words playing, played and player might be represented using the shared token play, plus smaller tokens for their endings. This keeps the vocabulary to a manageable size, while still letting the model represent an unfamiliar word as a combination of tokens it already knows.

The idea is short enough to state in full:

1. start with every individual character as a piece;
2. count every adjacent pair of pieces across the corpus;
3. merge the most frequent pair into a single new piece;
4. repeat until the vocabulary reaches the size you asked for.

How the code works:

• Frequent letter pairs are merged first, so common fragments assemble themselves out of the data.
• Run this for thousands of merges over a real corpus and the pieces come to look like morphemes — not because anyone taught it morphology, but because frequent sequences are frequent for a reason.

BPE is a real tool, not just an idea. It runs in two stages:

• Training, done once on a large corpus: the merge loop above, run for tens of thousands of merges. The result is a fixed list of pieces.
• Tokenizing, done on every new text: each word is cut into pieces from that list. Each word is cut one way only, end to end, so every character falls in exactly one piece, and the pieces have different lengths.

You will rarely train BPE yourself. A pretrained model such as BERT or BigBird comes with a tokenizer that has already been trained this way (or by a close relative of BPE), and every text must go through that tokenizer before the model can read it. So the model’s idea of a token is not your idea of a word. scikit-learn has no BPE, which is why it was written out by hand above; it returns with those models in later chapters.

3.5.3 Character n-grams: pieces you can use today

Character n-grams also make tokens smaller than a word, but they get there by a different route. BPE learns which pieces to use and cuts each word one way. Character n-grams learn nothing: they keep every run of a fixed length, at every position, overlapping. They are a setting of CountVectorizer, so you can use them today with the classifiers of this book. Instead of whole words, count every run of three characters. A run of n characters is a character n-gram. The characters are not only letters: digits, punctuation and the spaces around words count too. This book calls them runs of letters for short.

How the code works:

• analyzer="char_wb" tells CountVectorizer to count runs of characters instead of words. The wb stands for word boundaries: runs never cross from one word into the next, and each word gets a space at either end, so ca means “a word that starts with ca”.
• ngram_range=(3, 3) asks for runs of exactly three characters.

As a whole word, carcinma is simply out of vocabulary and dropped (Section 2.2). As runs of letters, six of its eight pieces are ones carcinoma also has. A report with the typo still lands close to the reports that spell the word correctly. Shared word parts help in the same way: adenocarcinoma and carcinoma, or nephrectomy and nephritis, now share columns too.

Set this beside BPE. After three merges, BPE cut carcinoma into ['c', 'a', 'r', 'c', 'i', 'noma']: six pieces of different lengths, laid end to end. The chunker gives nine pieces of three characters each, overlapping, so most letters appear in three of them.

	Byte-pair encoding	Character n-grams
Where the pieces come from	Learned from a corpus: frequent pairs are merged	A fixed rule: every run of n characters
Length of a piece	Varies, from one character to a whole word	Fixed by ngram_range
Pieces of one word	One way of cutting it, no overlap	Every run, overlapping
Where you meet it	The tokenizer that comes with a pretrained model	A CountVectorizer setting
Used for	Turning text into the input of that model	Columns of a document-term matrix

3.5.3.1 Varying the settings

Both settings in the chunker line can change. analyzer picks the unit: "char_wb", "char", or "word", which is what you get if you leave analyzer out. ngram_range=(m, n) picks the sizes: every piece made of m up to n units in a row. Here are five combinations on the same short phrase:

How the code works:

• "char" slides across the whole text, spaces included, so it makes pieces such as o c that join the end of no to the start of carcinoma. "char_wb" never crosses from one word into the next. Pieces that span two words record which words sit side by side, the same information word pairs carry (Section 3.5.4).
• "word" turns the same ngram_range=(3, 3) into runs of three words. no carcinoma has only two, so the list is empty. Dropping analyzer from the chunker line does not give you the default size of letter run; it switches the unit back to words.
• (2, 3) gives runs of two and of three characters, and (1, 3) adds single characters, the space among them. Almost every report contains every single character, so those columns make the matrix wider without telling reports apart.

The characters really are characters, not just letters. Compare the default word tokenizer with chunker on a node ratio:

The default tokenizer throws away 2/9, as it did earlier in this chapter. Runs of characters keep 2/9 as one of their pieces, along with /9 and 2/.

Does it help on the real corpus? Here is Chapter 2’s 1-nearest-neighbor classifier (cosine), with the same training and validation split as Section 2.4, and only the tokens changed:

tokens                          columns   validation accuracy
whole words (counts)             32,722         0.827
runs of 3 letters                33,241         0.847
runs of 4 letters               105,251         0.859
runs of 3, 4 and 5 letters      304,255         0.858

About three points of gain, from one changed setting. Three cautions:

• The gain cannot be credited to misspellings alone. Runs of letters also link related words such as adenocarcinoma and carcinoma. This comparison cannot separate the two effects.
• The matrix gets wider. Runs of 3, 4 and 5 letters give almost ten times as many columns as whole words, and take several times longer to compare, for no gain over runs of 4.
• A run of letters means nothing on its own. When you look inside a match, as Section 2.6 did, the “top shared words” become fragments like oma and ca, which are much harder to read than words.

3.5.4 Word n-grams: pairs of words

Runs of letters make a token smaller than a word. The opposite move makes it larger: count pairs of neighboring words as well as single words. Section 1.5.3 did this on the six toy reports, to keep no attached to the word it negates. A sequence of n neighboring words is a word n-gram; a pair is a bigram.

The setting is the same ngram_range as before. ngram_range=(m, n) means “every piece made of m up to n units in a row”, and the unit is a word unless analyzer says otherwise:

How the code works:

• (1, 1) is the default: single words. (2, 2) gives pairs only, and (1, 2) gives both.
• With pairs, no evidence is a column of its own, so a negated finding no longer looks like a positive one.
• The last line is a warning. Stopwords are removed before the pairs are formed, so no is gone before it can pair with anything, and the result is evidence metastasis.

Does it help on the real corpus? The same classifier and split as the table in Section 3.5.3:

tokens                          columns   validation accuracy
whole words (counts)             32,722         0.827
words and word pairs            392,078         0.823
word pairs only                 359,356         0.807

No. Pairs give twelve times as many columns and a slightly lower score. Two reasons stand out:

• Most pairs are rare. Of the 359,356 pairs in the training reports, 228,732 (64%) occur in just one report. Each is a column that almost never helps find a neighbor.
• Many common pairs belong to one hospital. About 600 pairs that occur in at least 20 training reports come almost entirely (90% or more) from a single site. node examination, for example, appears in 62 training reports, all from site BH. Pairs capture a hospital’s template even better than single words do, which is exactly the confounding of Section 2.7.3.

On hospitals the method has never seen, the second point bites. With training and validation reports split by hospital (Lab 3), adding pairs to word counts lowers validation accuracy from 0.421 to 0.377.

That does not make word n-grams useless. Nearest neighbors compares whole reports, so every extra column counts equally, rare or not. A model that learns a weight for each column (Chapter 4) can give no evidence a large weight and the rare pairs almost none. Chapter 5 returns to n-grams with the tools that tame the extra columns: dropping pieces that occur in too few reports, and down-weighting pieces that occur everywhere.

3.5.5 Tokenization and model input length

A report does not have one length. It has as many lengths as there are ways to count it.

This matters because models have hard input limits, and the limit is counted in the model’s own tokens — subword pieces, not words.

Model	Maximum input
BERT / ClinicalBERT	512 tokens
Clinical-BigBird	4,096 tokens

Now put that against the corpus. Measured on the 9,523 reports, 42.7% exceed 512 whitespace-separated words. But subword tokenization produces more tokens than whitespace splitting, because long medical words get broken into pieces — so the share exceeding 512 model tokens is higher still. The study this corpus comes from reports that more than 66% of documents pass ClinicalBERT’s limit, which is what motivated moving to Clinical-BigBird and its far longer input.

The consequence is blunt. What does not fit is truncated — silently cut off at the limit. And a pathology report puts its diagnosis wherever the institution’s template puts it, which is often near the end.

WarningTruncation is not a neutral operation

A model that never sees the final section of a report cannot use what the final section says. Before choosing a model, measure how long your documents are in that model’s tokens — not in words, and not in characters.

This is the bridge to the chapters on contextual embeddings and transformers, where the tokenizer stops being a preprocessing choice you make and becomes part of the model you adopt.

3.6 What it bought: the scoreboard

Every part of this chapter changed how reports become tokens, and several measured the effect. Here they are side by side, with one method throughout (the 1-nearest-neighbor classifier of Chapter 2, cosine) and every number from a validation set, never the test set. There are two validation sets, because there are two questions:

• hospitals it has seen: Chapter 2’s split, made report by report (Section 2.4), so most hospitals have reports in every pile;
• hospitals it has never seen: the same three piles, but split by hospital, so that no hospital appears in more than one pile (Section 2.7.3; this is the split Lab 3 uses).

Tokens built by	Discussed in	Hospitals it has seen	Hospitals it has never seen
splitting on spaces only; nothing normalized	Section 3.3.1	0.778	0.164
CountVectorizer(lowercase=False)	Section 3.3.1	0.807	0.303
CountVectorizer(), the default: word counts		0.827	0.421
word presence (binary=True)	Section 2.4	0.856	0.304
word counts, stopwords removed	Section 3.3.2	0.850	0.517
word presence, stopwords removed	Section 3.3.2	0.867	0.340
runs of 4 letters	Section 3.5.3	0.859	0.489
words and word pairs	Section 3.5.4	0.823	0.377

These are validation scores, so they differ slightly from the test scores quoted earlier for the same settings: the table in Section 3.3.1, for example, gives 0.785, 0.826 and 0.853 for the first three rows on hospitals the method has seen.

Read it from the top.

• The default normalization is the biggest step. Going from splitting on spaces to CountVectorizer() adds 5 points for hospitals the method has seen and multiplies the score by two and a half for hospitals it has not. After that, every change is worth a few points either way.
• The two columns disagree about what is best. For hospitals it has seen, the winner is word presence without stopwords. For hospitals it has never seen, that same setting falls below the default (0.340 against 0.421), and the winner is word counts without stopwords. The changes that do worse on new hospitals (presence, word pairs) give more say to rare tokens, and a hospital’s rarest tokens are its own: its headings, stock phrases and spellings. The changes that help most on new hospitals (lowercasing, removing stopwords) erase part of a hospital’s house style.
• Each winner, tested once. Refitted on training and validation reports together and run on the matching test set, word presence without stopwords scores 0.881 on hospitals it has seen (the default: 0.853), and word counts without stopwords scores 0.489 on hospitals it has never seen (the default: 0.411).
• The second column is the less certain one. It rests on whichever 137 hospitals happened to land in the validation pile. Over five different random choices of test hospitals, word counts without stopwords scored anywhere from 0.395 to 0.562, while beating the default every time.

So there is no best tokenizer in general. There is a best tokenizer for a question, and the validation set has to ask the question the method will really face. And even the best choice leaves nearest neighbors below 0.5 on new hospitals: it compares whole reports, and a hospital’s template is most of a report. Chapter 4 replaces it with a model that learns which words matter, and that reaches 84.6% on hospitals it has never seen.

3.7 Exercises

3.7.1 Predict before you run

# Two rules explain almost everything you will see:
#   1. text is lowercased and split on anything that is not a word character
#   2. only runs of TWO OR MORE word characters survive
# So single digits and single letters disappear entirely.

3.7.2 Write a tokenizer that keeps what matters

# Build one pattern with | alternatives, and put the SPECIFIC patterns first,
# because re.findall tries them left to right:
#     ratios       \b\d+/\d+\b
#     measurements \b\d+(?:\.\d+)?(?:cm|mm|g)\b
#     hyphenated   \b\w+(?:-\w+)+\b
#     plain words  \b\w{2,}\b

3.7.3 Count a corpus

# For 5: build a set of tokens per report, then intersect them all.
#     sets = corpus["text"].apply(lambda t: set(analyzer(t)))
#     common = set.intersection(*sets)

3.8 Summary

• Tokenization defines the units everything downstream is built from. It is a modeling decision, not a mechanical preliminary.
• Clinical text breaks ordinary tokenizers. Hyphenated biomarkers, staging strings, measurements and node ratios each fail differently — and 2/9 disappears completely under the default rule.
• Normalization reduces variation but can remove information. Lowercasing merges Carcinoma with carcinoma, and also ER with er.
• Stopword removal is dangerous here. no evidence of metastasis and evidence of metastasis become identical, and they mean opposite things.
• A corpus induces a vocabulary, and that vocabulary is the set of columns in the document-term matrix. Change the tokenizer, change the matrix.
• Most of the vocabulary occurs once. Rare and specialised terms are precisely the informative ones, and a fixed whole-word vocabulary discards them.
• Subword tokenization removes the out-of-vocabulary problem by representing unseen words as sequences of seen pieces, learned from data by algorithms such as BPE.
• Character n-grams bring the same idea to CountVectorizer today: a misspelled word still shares most of its runs of letters with the correct one. On the real corpus they lift 1-nearest-neighbor from 0.827 to about 0.86 on validation.
• Word n-grams keep no evidence together, but for nearest neighbors they mostly add rare columns and hospital phrases: 0.827 becomes 0.823, and 0.421 becomes 0.377 on hospitals never seen.
• Tokenization determines input length, and input length runs into hard model limits. What does not fit is truncated, and truncation is not neutral.
• There is no best tokenizer in general, only a best tokenizer for a question. On hospitals it has seen, word presence without stopwords wins (0.881 on test); on hospitals it has never seen, word counts without stopwords win (0.489, against the default’s 0.411). Choose on a validation set that asks the question the method will really face.

The next chapters take these representations and start building models on them.