2 Nearest Neighbors on the Real Corpus
Can the simplest classifier read real reports — and does its score mean what we think?
Chapter 1 labelled a new report by copying the diagnosis of its most similar known report, on six made-up sentences. This chapter does the same with the 9,523 real pathology reports. The method stays exactly as simple as it was; what changes is that, for the first time, we have to judge it honestly — and that turns out to take most of the chapter.
Cells run in your browser and share one session, so run them in order from the top. The first one takes fifteen to thirty seconds while Python loads.
2.1 Splitting the data
In Section 1.4.1 we knew the diagnosis of six reports, and one report was new. On the real corpus we know the diagnosis of every one of the 9,523 reports, so there is nothing new to try the method on. We make something new: we set some of the labelled reports aside, hide their diagnoses from the method, and treat them as if they had just arrived. Afterwards we compare the method’s guesses with the diagnoses we hid.
Why not simply ask the method to label the reports it already knows? Because nearest neighbors labels a report by finding the most similar known report — and the most similar known report to any known report is that report itself. Every answer would be right, and the score would say nothing about reports the method has not seen, which are the only ones anyone needs it for.
The two piles have standard names. The reports the method learns from are the training set; the reports held back to measure it are the test set. A common choice is 80% for training and 20% for testing. (A third pile, the validation set, appears later in this chapter.)
Splitting sounds trivial: shuffle the reports and cut the pile. On this corpus it is not, because the cancer types are very unevenly represented — 1,034 breast reports, but only 43 of cholangiocarcinoma, a cancer of the bile ducts. A careless split can quietly ruin the measurement for the rare types, sometimes for a whole type at once.
2.1.1 A random split can lose a class
Start small, with 40 imaginary reports: 24 breast, 12 kidney and 4 lung. To study how a split behaves we do not even need the reports’ text — only their labels.
train_test_split shuffles a list and cuts it in two. With test_size=0.25, a quarter of the 40 labels — 10 of them — go to the test set. random_state fixes the shuffle, so you see exactly what we see; change it and you get a different random split.
There is no lung report in the test set. The method would be trained on all four lung reports and then never tested on lung at all: whatever its score turned out to be, it would tell us nothing about how it handles lung cancer.
That was not one unlucky shuffle. Here are a thousand different random splits, counting the lung reports that land in the test set each time:
Read the table row by row. A fair share would be one lung report in the test set — a quarter of four — and fewer than half of the splits manage it. In almost three splits out of ten the test set has no lung report at all. And in two of them all four lung reports went to the test set, which leaves none in the training set: the method could never predict lung, because it would never have seen it.
Before reading on: how would you divide these 40 reports so that every type is sure to get its fair share on both sides?
2.1.2 Stratification
The fix is to split each class separately, in the same proportion, and then put the pieces back together. With a quarter for testing, that means 6 of the 24 breast reports, 3 of the 12 kidney reports and 1 of the 4 lung reports. Every class then appears on both sides, in the same proportion as in the whole dataset. This is called a stratified split — the data is divided into layers, or strata, one per class, and each layer is split on its own.
In scikit-learn it is one extra argument: stratify=labels tells train_test_split which labels to keep in proportion.
Six, three and one, exactly. Change random_state and different reports are chosen, but the counts stay the same every time.
2.1.3 What nearest neighbors does with a split
Before the real corpus, here is what the split is for. A real report is a vector with thousands of features, one per vocabulary word, and nobody can picture that. So imagine a vocabulary of only three words — breast, renal and lung — so that every report is an arrow from the origin in ordinary three-dimensional space. A breast report leans toward the breast axis, a kidney report toward renal, a lung report toward lung, but none lies exactly on its axis, because real reports use some of each.
In the widget, the dots are the training set: arrows whose labels the method knows. The diamonds are the test set: their labels are hidden from the method, and are coloured here only so that we can check its answers. To label a test report, nearest neighbors computes its cosine similarity with every training report and copies the label of the one with the largest cosine — the arrow pointing in the most similar direction. The arc marks the angle θ between the test arrow and its nearest neighbor; the smaller the angle, the larger cos θ.
Drag the picture to turn it. Click a diamond (or a T button) to classify a different test report, and click any dot to measure its angle too. Draw a new training set and watch which predictions change.
Things to notice:
- Length does not matter. Tick make every arrow the same length and every arrow is shrunk or stretched onto one sphere. The pictures change; not one prediction does, because cosine similarity only ever looked at directions. A long report and a short report that use their words in the same proportions point the same way.
- Nearest by angle is not nearest by distance. Press Reset and select T5, a short lung report. The arrow it points most like is #15, a long lung report, so the method answers correctly. Now click the kidney dots near T5’s tip: one of them, #9, is closer in straight-line distance than #15 is, but at a wider angle. Measured by distance, T5 would have been called kidney.
- The score depends on the split. Press New training set a few times. Most training sets label all six test reports correctly, but some get one or two wrong: a training report of another class happens to point closer. Nothing about the method changed — only which reports it was given to learn from. That is the same luck stratification was protecting us from, and it is why a score from a small test set should never be taken at face value.
2.1.4 The real corpus
Now the real thing. Again we need only the labels, so the cell below rebuilds all 9,523 of them from the number of reports of each cancer type. (These are TCGA’s short codes: BRCA is breast cancer, CHOL cholangiocarcinoma. Lab 1 builds the table they come from.)
Follow the rarest type, cholangiocarcinoma, through a thousand random 80/20 splits. A fair share of the test set would be 20% of 43 — between 8 and 9 reports.
The test set gets anywhere from 2 to 17 cholangiocarcinoma reports, depending only on how the shuffle fell. Neither end is harmless. With 2, the method’s accuracy on this cancer rests on two reports, so each one is worth 50 percentage points. With 17, only 26 are left to learn from. Either way the measurement for this type depends on luck — and so, to a smaller degree, does the measurement for every other type.
A stratified split removes the luck:
Thirty-four to learn from and nine to test on — and the same proportion for every one of the 32 types.
2.1.5 When a class is too small
Stratification keeps the proportions as closely as whole reports allow. With very small classes, whole reports do not allow much. Go back to the toy example, but now with only two lung reports among twenty:
A quarter of two lung reports is half a report, and half a report rounds to none: even the stratified split leaves lung out of the test set. Shrink the class to a single report and the split cannot be done at all — one report cannot be on both sides — so scikit-learn refuses:
The real corpus is safe from both problems — its smallest type has 43 reports — but it is not safe from the milder version. Nine test reports measure cholangiocarcinoma only coarsely: each one is worth about 11 percentage points of that type’s accuracy. Stratification guarantees that every type is measured; it cannot make the measurement precise. The only real remedies are more reports, reporting each type’s result together with how many test reports it rests on, and repeating the whole experiment with several different splits to see how much the numbers move.
2.1.6 Checking a split
Whatever split you make, check it before trusting it.
1. Are the classes in proportion on both sides? Compare each type’s share of the training set with its share of the test set. (The cell repeats the stratified split first, because train and test were last set by the toy examples above.)
A gap of a few hundredths of a percentage point: every type has essentially the same share on both sides. A random split of the same corpus typically leaves a gap of about one percentage point, and sometimes more than two — small in the table, but enough to tilt a score.
2. Is any patient on both sides? Lab 1 checked that our corpus has exactly one report per patient, so splitting reports is the same as splitting patients. In many medical datasets that is not true: one patient can have several reports, written in the same words by the same people. Split those by report, and the same patient can end up in both piles — the method then “recognises” a test report because it has already seen that patient’s other reports. The fix is to split by patient. Later in this chapter the same problem turns up one level higher, with whole hospitals.
3. Is the split fixed and saved? Use a fixed random_state and keep the result, so that every method in the following chapters is measured on exactly the same test reports. A comparison between two methods tested on different splits is partly a comparison between two shuffles.
Lab 2 makes exactly this split, with stratify=, in its first puzzle.
2.2 Building the vocabulary from the training reports only
With the split made, the steps of Chapter 1 apply unchanged — with one rule about which reports each step may look at. The vocabulary is built from the training reports alone, with fit_transform; the test reports are then counted with that same vocabulary, using transform. Here is the rule on a tiny scale, with a misspelling in the “test” report:
The vocabulary has no column for carcinma, so the test report’s row simply has nothing there: the word is dropped (Chapter 1 called this an out-of-vocabulary word).
Why not build the vocabulary from all the reports, so that nothing is dropped? Because the test set stands in for the future. When the method is really used, the new reports do not exist yet when it is built, so nothing about them — not even which words they contain — can have shaped it. Anything learned from the data must be learned from the training reports only. That rule will matter more and more as the methods learn more from the data; here it already decides what the vocabulary is.
On the real corpus the effect is easy to see:
| Vocabulary built from | Words |
|---|---|
| the training reports (80%) | 38,839 |
| all 9,523 reports | 43,516 |
The test reports contain 4,677 words that appear in no training report. A random sample of them: oysts, hyalnized, synaptophasin, intraopeiative, leftseminal, periostoma, specifical, prepent. Almost all are scanning errors — the scanned-and-OCR’d origin of these reports, showing through. Each is dropped from its test report, exactly like carcinma.
Is that a serious loss? Usually not. A misspelled word is one of hundreds in its report, and the words around it still carry the match: carcinma is gone, but invasive, ductal and breast are not. It hurts when the damaged word is the one that matters most. Nothing in this chapter repairs it, because the rule above forbids the obvious repair. The remedy is to change what counts as a word: Section 3.5.3 compares reports by short runs of letters instead of whole words, so that carcinma still shares most of its pieces with carcinoma.
2.3 The baseline, then nearest neighbors
From here on the chapter works on the full corpus. Building a 9,523 × 38,839 table and comparing thousands of reports is more than a browser page should be asked to do, so the real-corpus code below is shown with its output rather than run live. It runs as written in Google Colab, and Lab 2 runs most of it — the split, the vocabulary, the classifier and the investigation later in this chapter.
First the number to beat. Always guessing the most common cancer type — breast, 1,034 of 9,523 reports — is right 10.9% of the time. Then the method of Section 1.4.1, unchanged: one nearest neighbor, cosine similarity, word counts.
known, new = train_test_split(reports, test_size=0.2,
stratify=reports["cancer_type"], random_state=0)
vec = CountVectorizer()
X_known = vec.fit_transform(known["text"])
X_new = vec.transform(new["text"])
knn = KNeighborsClassifier(n_neighbors=1, metric="cosine")
knn.fit(X_known, known["cancer_type"])
accuracy = (knn.predict(X_new) == new["cancer_type"]).mean()Nearest-neighbour accuracy: 0.853
Baseline: 0.109About 85% right — eight times the baseline — from a method with no learning in it at all, just “find the most similar report and copy its label”. Word counts clearly carry a great deal of information about cancer type. How much of that information is about the disease is a question the rest of the chapter takes seriously.
2.4 Choosing k with a validation set
2.4.1 More than one neighbor
One neighbor is a fragile witness: if the single most similar report happens to be unusual, the prediction goes wrong with it. The obvious remedy is to ask several. A k-nearest-neighbor classifier finds the k most similar training reports and lets them vote. Back to the six reports of Chapter 1, and a new kidney report:
With one neighbor or three, the answer is Kidney. With five, two kidney reports, two breast reports and a lung report vote — a tie between Kidney and Breast, which scikit-learn breaks by taking the label that comes first alphabetically. Breast wins, for no medical reason at all.
The fix is to let closer neighbors count for more. With distance-weighted voting, each neighbor’s vote is weighted by how close it is, so the two kidney reports — by far the most similar — outweigh the distant breast reports:
Push k further and a different problem appears. If k equals the whole training set, every report votes on every prediction, and the answer is always the same: the most common class. A huge k turns the classifier back into the baseline. Somewhere between “one fragile witness” and “everyone votes” is a best k — and the only way to find it is to try values and measure.
2.4.2 Why the test set cannot choose k
Measure on what? The tempting answer is the test set: try k = 1, 3, 5, … and keep whichever scores best there. But then the test set has been used to make a decision about the method. Its score is no longer a measurement of how the method does on unseen reports — the method has, in effect, been fitted to those reports by choosing the k that suits them best. Try enough settings and one will look good on the test set by luck alone.
So decisions get a pile of their own. The training reports are split once more, and a part of them becomes the validation set:
- the training set is for learning — here, the reports the neighbors are drawn from;
- the validation set is for development decisions — which k, which voting rule, which distance measure;
- the test set is for the final verdict, used once, after every decision is made.
Think of homework, practice exams and the final exam. You learn from homework; you check your study habits on practice exams, and change them if they are not working; the final exam is taken once, and the grade counts because you never saw it before. If you had studied the final exam’s questions, the grade would say nothing about what you know.
We keep the same test set as before and split the training 80% into 60% for training and 20% for validation — 5,713, 1,905 and 1,905 reports, each split stratified.
train_all, test = train_test_split(reports, test_size=0.2,
stratify=reports["cancer_type"], random_state=0)
train, val = train_test_split(train_all, test_size=0.25,
stratify=train_all["cancer_type"], random_state=0)2.4.3 What to tune
Four choices, each tried on the validation set:
- k — how many neighbors vote;
- the voting rule — every vote equal, or weighted by closeness;
- the distance measure — cosine similarity, or plain straight-line distance between the rows of counts;
- the features — word counts, or just word presence (1 if the word occurs at all, 0 if not).
validation accuracy votes equal distance-weighted
word counts, cosine k = 1 0.827 0.827
k = 3 0.834 0.837
k = 5 0.835 0.843
k = 9 0.821 0.833
k = 25 0.786 0.816
k = 101 0.729 0.762
k = 301 0.659 0.707
word counts, straight-line k = 1 0.803
k = 5 0.789
word presence, cosine k = 1 0.856 0.856
k = 3 0.847 0.857
k = 5 0.850 0.855
k = 9 0.838 0.849Read it from the top.
- Small k is best, and large k collapses. With word counts, accuracy peaks around k = 5 and falls steadily after that, to 66% at k = 301 — on its way back to the baseline. (One sign of the drift: at k = 301 the classifier labels 12.5% of the validation reports as breast cancer, against 10.8% at k = 1. The biggest class is starting to win votes it should not.)
- Distance-weighted voting is never worse, and helps more as k grows. It keeps the distant neighbors from outvoting the close ones.
- Straight-line distance is worse than cosine. It is affected by report length — a long report is far from a short one even when they say the same thing — which is exactly why Chapter 1 preferred cosine.
- Word presence beats word counts. Counts let a word repeated thirty times — usually the or of — dominate the comparison. Presence counts each word once, so the rarer, more informative words get a fairer say. Keep that in mind for Chapter 3.
The best setting on validation is word presence, cosine, k = 3, distance-weighted: 0.857. Only now, with every decision made, does the test set come out. The chosen setting is refitted on all the training reports — training and validation together, since the decisions are done — and run on the test set, once:
Test accuracy, chosen setting (presence, k = 3, distance-weighted): 0.870
Test accuracy, plain 1-nearest neighbor (counts): 0.853
Baseline: 0.109Two cautions in reading those numbers. First, the top few settings on validation differ by a few tenths of a percentage point, and with 1,905 validation reports a difference that small is within the noise — the choice among them is closer to a coin toss than the table suggests. Second, 87% is an average over 32 cancer types, and they are far from equal:
| Cancer type | Test reports | Accuracy | Most often mistaken for |
|---|---|---|---|
| Uterine carcinosarcoma (UCS) | 11 | 18% | endometrial cancer (UCEC) |
| Rectal adenocarcinoma (READ) | 32 | 47% | colon adenocarcinoma (COAD) |
| Cholangiocarcinoma (CHOL) | 9 | 67% | liver cancer (LIHC) |
| Kidney papillary (KIRP) | 56 | 68% | kidney clear cell (KIRC) |
| Lung squamous cell (LUSC) | 94 | 70% | lung adenocarcinoma (LUAD) |
| Breast (BRCA) | 207 | 99% |
The mistakes are medically sensible — rectum for colon, one kind of kidney or lung cancer for another, bile duct for liver — which is reassuring about what the method has picked up. And the rare types are both the hardest and the least precisely measured: the 18% for UCS rests on 11 reports, the 67% for CHOL on 9. That is the warning of Section 2.1, in practice.
2.5 What nearest neighbors costs
Most classifiers spend their effort ahead of time: they study the training data once and condense it into something small — logistic regression condenses it into a table of weights. Nearest neighbors does the opposite. “Fitting” it only stores the training reports; all the work happens at prediction time, when each new report is compared with every stored one.
fit (store 7,618 reports): 0.01 seconds
predict 1,905 reports against 1,904 stored: 0.14 seconds
predict 1,905 reports against 3,809 stored: 0.30 seconds
predict 1,905 reports against 7,618 stored: 0.65 secondsDouble the training reports and every prediction takes twice as long, and the whole training set has to be kept around to predict anything. For 9,500 reports that hardly matters. For a hospital’s millions of documents it does, and large systems use approximate nearest-neighbor search: indexes that find almost the closest reports very quickly, giving up a little accuracy for a great deal of speed. The same idea — find the most similar documents, fast — returns in later chapters, as search over word and document vectors.
2.6 Looking inside the matches
About 87% right is a strong number, but a number cannot say why. Nearest neighbors, unlike most classifiers, can show its reasons: for any prediction, here are the reports it was copied from. A small find-similar tool shows the most similar known reports, and the words that contributed most to each match. A word’s contribution is its count in one report times its count in the other — cosine similarity is built from exactly those products — so the words with the biggest products are the ones doing the matching.
On six clean sentences the shared words are exactly what a pathologist would point to. On the real reports, the same tool shows something less comfortable. Here is a kidney cancer (chromophobe type, KICH) from the test set and its three closest training reports:
NEW: TCGA-UW-A7GN KICH "Diagnosis: Kidney, left, laparoscopic nephrectomy. - Renal cell
carcinoma, chromophobe type, grade 2 (of 4, Fuhrman classification)..."
0.85 TCGA-UW-A7GP KICH "Diagnosis: Kidney, left, laparoscopic nephrectomy. Renal cell
carcinoma, chromophobe type, grade 3 (of 4, Fuhrman classification)..."
top shared words: of, tumor, not, renal, the, negative, is, kidney
0.83 TCGA-B8-5546 KIRC "Diagnosis: Kidney, right, laparoscopic nephrectomy. Histologic
tumor type/subtype: Renal cell carcinoma, clear cell type..."
top shared words: tumor, of, not, renal, the, grossly, negative, cm
0.83 TCGA-B8-A54F KIRC "Diagnosis. Kidney, left, radical nephrectomy. Histologic tumor
type/subtype renal cell carcinoma, clear cell. type..."
top shared words: not, of, tumor, renal, negative, kidney, ureter, doesTwo things stand out. Half of the top shared words — of, the, is, not — would be in almost any report about anything; across all 1,905 test reports, 79% of the top shared words in each match are among the 50 most common words in the training reports. Much of what “similar” means here is “uses the same everyday words”. That is the problem Chapter 3 takes up: what should count as a word, and should every word count?
The second thing is in the barcodes, and it needs a section of its own.
2.7 Data leakage
Data leakage is information getting into the training or the evaluation of a method that would not be available when the method is really used. It makes the measured score better than the real one, and it rarely announces itself: nothing errors, the number is simply too good. This corpus has three candidates.
2.7.1 Leak 1: preprocessing that learns from the test reports
Imagine a teacher who writes the study guide after reading the final exam. The class studies the guide and everyone scores well — but the scores say little about what the students learned. They mostly say that the guide and the exam were written by someone who had seen both. Give the class a different exam on the same subject, and the scores fall.
A method can be handed the same unfair advantage before any “learning” starts. Before a method is trained, the data is usually prepared: words are counted, rare words are dropped, numbers are rescaled, missing values are filled in, the most useful columns are picked out. None of this feels like learning, but each step looks at the data and makes a decision — which words exist, what the average is, which columns seem to matter. If that look includes the test reports, the test reports have helped write the study guide. It usually happens innocently: the code cleans the whole table first and splits it afterwards, because that is the order the steps come to mind.
The question to ask of every preparation step is: could this step have been done before the test reports existed? When the method is really used, the new reports arrive later, so nothing about them can have shaped it.
2.7.1.1 When the leak is harmless
Section 2.2 built the vocabulary from the training reports only. What if it had been built from all of them? For this particular method, it turns out to change nothing — test accuracy is 0.853 either way — and the reason is worth seeing:
Adding carcinma to the vocabulary makes the test report’s row longer, which lowers its cosine similarity with every training report by the same factor. The order does not change, so the nearest neighbor does not change. Here the leak happens to be harmless — but only because of a property of this one method, not because leaks usually are.
2.7.1.2 When the leak is not harmless
The damage shows when a preparation step decides which columns the method gets to see. A common, reasonable-sounding step: “there are thousands of words; keep only the 20 that differ most between the classes.” The experiment below makes the effect easy to see. It invents 200 “reports” whose 5,000 word counts are pure random numbers, and gives each one a label by coin flip. There is nothing to learn, so an honest method should be right about half the time.
The first number is 0.70: 70% on coin flips. Among 5,000 random words, a few happen, by pure chance, to turn up a little more often in the A reports than in the B reports. Chosen using all 200 reports, the winning words are the ones whose chance pattern also lines up with the coin flips of the test reports — the test reports have quietly voted on what the method gets to see. Chosen from the training reports alone, the same step gives 0.48: the honest answer, that there was nothing to find.
The size of that gap is the point. The leaky number is not slightly optimistic; it is a good-looking result from data with no information in it at all.
2.7.1.3 Not just a classroom mistake
This exact leak — choosing the useful columns before splitting — has reached published research. In 2002, Ambroise and McLachlan showed that published cancer classifiers claiming zero errors from only a handful of genes had chosen those genes using the very samples they were tested on; with the choice made fairly, the errors were no longer zero. Two decades later, a survey by Kapoor and Narayanan found leakage of one kind or another in 294 published papers across 17 fields, from medicine to predicting civil wars. Medicine is where it matters most: a leaked score is the one a hospital reads before trusting a method with patients.
The remedy is the pattern of Section 2.2, applied to every step that learns anything: split first; fit each step on the training reports (fit or fit_transform); then apply it, unchanged, to the test reports (transform). The vocabulary was the first such step. Later chapters add more — which words to keep, how much weight each word gets — and each follows the same rule.
2.7.2 Leak 2: duplicates across the split
If the same report sits in both the training and the test set, the method is “tested” on a report it has already memorized. Lab 1 found 18 exact duplicates in the corpus. After our split, 1 test report has an exact copy in the training set, and 10 test reports have a training neighbor at distance zero — the same words in the same numbers. They are revealing:
- six are fill-in-the-blank templates from a single hospital (“Sex: Female. Diagnosis: Breast Cancer. Histological description: …”), each matching another patient’s report filled in on the same template;
- two more are pairs of reports from one hospital written almost word for word alike;
- one is a lung checklist matched to another checklist with the same boxes ticked — filed under the other kind of lung cancer, so the prediction is wrong;
- one is not a report at all but a blank “TCGA Missing Pathology Report Form”, matched to an equally blank form filed under a different cancer — wrong again.
Ten reports out of 1,905 cannot move the overall score much, and here they do not. But the lesson generalizes: wherever the same document can be filed twice — a report and its addendum, a template filled in twice — check for duplicates before trusting a score.
2.7.3 Confounding: The Laboratory
Every barcode has three parts, and the middle one names the tissue source site: the hospital or laboratory that sent the sample.
TCGA-UW-A7GN
^^
tissue source siteLook back at the find-similar output. The new report came from site UW, and its closest match, TCGA-UW-A7GP, came from site UW too — and the two reports open with almost exactly the same words, because the same department wrote them on the same template. That turns out to be the rule, not the exception:
Site codes in the corpus: 686
Most cancer types sent under any one site code: 1
Test reports whose nearest neighbor is from the same site: 78.9%
Correct predictions whose nearest neighbor is from the same site: 92.5%
...if the neighbor were a random training report of the right type: 12.6%Every one of the 686 site codes covers exactly one cancer type — TCGA gave a hospital a separate code for each cancer study it contributed to. So a neighbor from the same site is automatically of the right type. When nearest neighbors gets a report right, 92.5% of the time it has matched a report from the same site, where chance would give 12.6%. The method is largely recognizing the paperwork: the template, the headings, the stock phrases of one department. And in this corpus that is enough to get the label, because the paperwork and the diagnosis travel together.
Statisticians call this confounding: the site is tied both to the text of a report and to its label, so it can stand in for the diagnosis. Nothing about it is special to text. A well-known study of chest X-rays, Zech et al. (2018), “Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study”, PLoS Medicine 15(11): e1002683, found that its models could tell which hospital system an X-ray came from almost perfectly, and partly used that to detect pneumonia; at a hospital they had not been trained on, their performance dropped.
Is that leakage? It depends on how the method will be used. If it will only ever see new reports from the same hospitals, recognizing their templates is a real — if unglamorous — skill. But no hospital deploying it wants to hear “accurate, provided your reports look like ones we have seen”. The way to find out is to split by site instead of by report: every report from a site goes to one side, so the test set holds only hospitals the method has never seen. On a toy corpus of six sites with four reports each, compare an ordinary (stratified) split with a group split:
The ordinary split scatters five of the six sites across both sides; the group split keeps each site whole. (Notice the price: the group split’s test set has no breast reports at all, because whole sites moved together. With 686 sites rather than 6, that problem mostly disappears — but it is why group splits and stratification pull in different directions.)
On the real corpus, with a fifth of the sites held out:
New reports from sites the method has seen (ordinary split): 0.853
New reports from sites it has never seen (split by site): 0.411
Baseline on the site-held-out test set: 0.100Accuracy falls by more than half. The method still does four times better than the baseline, so it has learned something about the words of each cancer — but most of its 85% came from recognizing hospitals. The honest number to quote a hospital that has never sent a report is 41%, not 85%.
This is the lesson of Section 2.1, one level up. Split so that the test set resembles the situation the method will actually face: by patient when a patient can have several reports, by hospital when the method must work for hospitals it has never seen. A test set that shares patients, hospitals or templates with the training set measures memory, not reading.
2.8 Boundaries that bend
Section 1.4.3 showed that a linear classifier splits the space of reports with a straight boundary. Nearest neighbors has no boundary written down anywhere — but it has one all the same: the line between the region where new points would be labelled one class and the region where they would be labelled the other. That line bends around the training examples.
The widget below uses 29 made-up reviews, each placed by two numbers — how positive and how negative its wording is — and coloured by its true label. The shading shows what k-nearest neighbors would predict at every point of the plane, using straight-line distance so the picture is easy to read. Move k and compare with the straight line of a linear classifier.
At k = 1 the boundary wraps tightly around every review, even the odd ones out, carving little islands for them: every training review lands on its own side, which only proves that the classifier remembers what it was shown. Leave each review out and predict it from the others — a miniature test set — and the mistakes appear. As k grows, the islands disappear and the boundary smooths out; at k = 29, every review votes on every point and the whole plane takes the majority label. The straight line cannot bend around anything: it is the same line whatever the data’s quirks, which is both its weakness and, as Section 1.4.2 begins to show, its strength.
2.9 Summary
- A classifier must be judged on reports it has not seen, so labelled reports are split into a training set and a test set, and decisions such as k get a validation set of their own. The test set is used once.
- Stratification keeps every class in proportion on both sides of a split. Without it, a rare class can vanish from the test set or the training set by luck. With it, rare classes are measured — but, with a handful of test reports, only coarsely.
- Anything the method learns — the vocabulary first of all — comes from the training reports only.
- k-nearest neighbors lets the k most similar training reports vote. On the real corpus it reaches 87% on the test set, against a baseline of 11%, with word presence, cosine similarity, k = 3 and distance-weighted votes chosen on validation.
- It learns nothing ahead of time and does all its work at prediction, comparing each new report with every stored one.
- Data leakage inflates scores without any warning. In this corpus the big one is the hospital: reports are matched largely by their template, and accuracy falls from 85% to 41% on hospitals the method has never seen. Split by the unit the method will face in use.
The find-similar tool showed one more thing: much of the matching runs on words like the and of. Chapter 3 asks what should count as a word in the first place.