Introduction
Human language is one of the richest forms of data we produce. It records observations, decisions, explanations, uncertainty, and knowledge that may never appear in a spreadsheet. Yet language is not naturally arranged for computation. A computer does not initially see a sentence as an expression of meaning; it sees a sequence of symbols. Before a program can compare documents, discover patterns, or make predictions from text, language must somehow be translated into a mathematical form.
Natural language processing, or NLP, studies how to make that translation useful. Its goal is not merely to count words or search for exact phrases, although both can be valuable. The larger goal is to design representations and models that can extract information from language while respecting the complexity, ambiguity, and context that language carries.
This book develops those ideas through one sustained problem: using pathology reports to predict a cancer type. A pathology report is a particularly revealing kind of text. It contains specialized medical vocabulary, measurements, abbreviations, recurring sections, and descriptions of evidence observed at different stages of examination. At the same time, it is written in natural language by a physician for other physicians. Its structure varies across institutions, important facts may be repeated or scattered throughout the report, and a single word such as no can reverse the clinical meaning of an otherwise familiar phrase.
This makes pathology reports an unusually good setting in which to learn NLP. The text is meaningful enough that the task matters, structured enough that patterns can be found, and messy enough that simple methods quickly reveal their limitations.
From language to a model
The first challenge is representation. A statistical model cannot operate directly on a sentence such as renal cell carcinoma involving the kidney. It needs numbers. We begin by constructing a vocabulary and representing each report as a vector whose components record which vocabulary terms appear. This bag-of-words representation discards grammar and most word order, but it makes something important possible: reports can now be compared mathematically. Similarity between two documents can be measured using the angle between their vectors, and even a simple nearest-neighbor rule can use those similarities to make a preliminary classification.
That first success raises a deeper question: what exactly should count as a word? Real clinical text contains capital letters, punctuation, numbers, abbreviations, misspellings, and artifacts introduced when scanned reports are converted into text. Tokenization divides a report into units that a model can use, while normalization attempts to make superficially different forms more consistent. Choices such as lowercasing text, removing punctuation, or discarding common words may appear mechanical, but each choice changes the information available to the model. Removing a frequent word may reduce noise; removing a word such as no may destroy the distinction between the presence and absence of disease.
Once the reports have been represented numerically, we can train a classifier. Logistic regression will serve as our first serious model because it is mathematically accessible, effective with sparse text data, and comparatively easy to interpret. Each feature receives a learned coefficient, allowing us to examine which words push a prediction toward one cancer class or another. Before training, however, the corpus must be divided into training, validation, and test sets. A model should learn from the training data, be refined using validation data, and be evaluated on test reports that have remained outside the development process. This separation is what allows performance on known documents to become evidence about performance on new ones.
A working classifier is only a beginning. Its representation can be improved by including short sequences of words, or n-grams, so that phrases such as no evidence can become features rather than being broken entirely into isolated words. Raw word counts can be replaced with TF–IDF weights, which reduce the influence of terms that occur throughout the corpus and emphasize terms that distinguish one document from another. Pipelines can join preprocessing, vectorization, and classification into a reproducible procedure. Hyperparameter search can then compare alternative settings systematically, while additional classifiers such as random forests provide useful points of comparison.
No model should be judged by a single accuracy number. Cancer classes may not be equally common, and a model can perform well overall while failing badly on a rare class. Precision, recall, F1 scores, confusion matrices, and class-specific results reveal different aspects of performance. Feature importance and model coefficients help us investigate what the system has learned. Error analysis brings us back to the reports themselves: Which cases were misclassified? Did the text genuinely contain ambiguous evidence, or did the model rely on formatting, institutional language, duplicated information, or some other artifact of the dataset? In a medical setting, understanding such failures is not an optional supplement to prediction; it is part of determining what the prediction means.
The final step in the present arc moves beyond sparse count-based representations. In a bag-of-words matrix, every vocabulary term occupies a separate dimension, and words with related meanings remain unrelated unless they occur in similar documents in ways the classifier can exploit. Word embeddings instead represent words with dense learned vectors. Terms used in similar linguistic contexts tend to receive nearby representations, introducing a geometric notion of meaning. This transition — from sparse vectors that record occurrence to dense vectors that encode learned relationships — prepares the ground for the more context-sensitive language models that follow later.
One corpus, many questions
Throughout these chapters, the methods will change but the underlying application will remain largely fixed. Holding the task steady allows us to ask meaningful comparative questions. What does tokenization change? Do bigrams improve predictions over individual words? Does TF–IDF outperform raw counts? Which cancer types are easiest to identify, and why? Do more sophisticated representations correct earlier errors, or do they merely introduce different ones?
The objective is therefore broader than producing the most accurate classifier. The aim is to understand the entire path from a clinical document to a prediction:
\[ \text{raw report} \longrightarrow \text{tokens} \longrightarrow \text{numerical representation} \longrightarrow \text{model} \longrightarrow \text{evaluation and interpretation}. \]
Each arrow hides choices, assumptions, and possible failure points. By making those choices visible, the book treats NLP not as a collection of library commands but as a sequence of mathematical and scientific decisions.
The reports used for this continuing investigation come from The Cancer Genome Atlas, or TCGA. They were created as part of actual medical care, collected alongside a major cancer-research program, scanned, de-identified, and later recovered as text. They were not written as clean examples for a machine-learning course. Their clinical origins explain both their value and their disorder.
Before turning words into vectors, then, we should understand the documents themselves: where they came from, who wrote them, what information they contain, and why predicting an already documented cancer type is still a useful place to begin.
Where the reports come from
The Cancer Genome Atlas
The Cancer Genome Atlas — TCGA, said as its letters — was a research program run jointly by the National Cancer Institute and the National Human Genome Research Institute, both part of the United States National Institutes of Health. It began as a small pilot in 2006 covering three cancers, expanded over the following decade, and concluded in 2018.
Its purpose was molecular. TCGA set out to catalog, in detail, the genetic changes that occur in human cancers: which genes are mutated, which are amplified or deleted, which are switched on or off, and how those patterns differ from one cancer to another. To do that it collected tumor samples — and matched normal tissue from the same patients — from roughly 11,000 people spanning 33 cancer types, then ran a battery of molecular assays over each one. The resulting data is public, and it remains one of the most heavily used resources in cancer research.
Notice that nothing in that description mentions text. The pathology reports we are about to work with are a byproduct of how the collection worked.
TCGA did not treat patients. Tumor samples were contributed by hospitals and research centers — the tissue source sites — from patients who were being diagnosed and treated in the ordinary course of care. A site that contributed a specimen also contributed the clinical documentation that came with it, and that documentation included the pathology report written when the specimen was first examined. Those reports were deposited as scanned pages, released publicly, and later converted back into text.
So the corpus is not a benchmark someone assembled to test language models. It is a pile of real clinical documents that happen to sit beside a research dataset. That is what makes it valuable, and it is also what makes it messy.
One report, verbatim
Here is a single report from the corpus, exactly as it comes out of the pipeline — nothing corrected, nothing tidied.
Date of Recelpt: Clinical Diagnosis & History: Incidental 3 cm left upper pole renal mass. Specimens Submitted: 1: Kidney, Left Upper Pole; Partial Nephrectomy. DIAGNOSIS: 1. Kidney, Left Upper Pole; Partial Nephrectomy: Tumor Type: Renal cell carcinoma - Conventional (clear cell) type. Fuhrman Nuclear Grade: Nuclear grade II/IV. Tumor Size: Greatest diameter is 2.4 cm. Local Invasion (for renal cortical types): Not Identified. Renal Vein Invasion: Not identified. Surgical Margins: Free of tumor. Non-Neoplastic Kidney: shows focal chronic inflammation and focal superficial glomerulosclerosis. Adrenal Gland: Not identified. Lymph Nodes: Not identified. Staging for renal cell carcinoma/oncocytoma: pT1 Tumor <= 7.0 cm in greatest dimension limited to the kidney. JATTEST THAT THE ABOVE DIAGNOSIS IS BASED UPON MY PERSONAL EXAMINATION OF THE SLIDES (AND/OR OTHER MATERIAL), AND THAT IHAVE. REVIEWED AND APPROVED THIS REPORT. Gross Description: 1). The specimen Is received fresh for frozen section consultation, labeled “left upper pole renal tumor stitch marks deep margin”. It consists of a 6.6 X 3.7 x 2.9 cm wedge shaped portion of kidney with a suture marking the deep margin and attached 12.2 x 8,5 x. 2.7 cm of perirenal. Also received in the same container Is a separate 11.6 x 8.4 x 4.6 cm aggregate of unremarkable. fibroadipose tissue. The margin is inked black and the specimen is serially sectioned to reveal a well-circumscribed cortically. based yellow focally hemorrhagic 2.4 x 2.0 x 1.9 cm tumor. The tumor appears grossly confined to the renal parenchyma, no. extension into perirenal fat is identified. The clearance from the resection margin is 1.1 cm. A representative section of the. nearest margin is submilted for frozen section diagnosis. Representatively submitted. A photograph Is taken. Portions of the. tumor are submitted for TPS. Summary of sections: FSC - frozen section control. T tumor. M margin. RS - representative sections. Summary of Sections: Part 1: Kidney, Left Upper Pole; Partial Nephrectomy. Block. Sect. Site. PCs. 1. fsc. 1. 2. m. 2. 1. rs. 1. 2. t. 2. Intraoperative Consultation: Note: The diagnoses given in this section pertain only to the. tissue sample examined at the time of the intraoperative. consultation. 1). FROZEN SECTION DIAGNOSIS: LEFT UPPER POLE RENAL TUMOR. RENAL CORTICAL NEOPLASM. WITH BENIGN MARGIN. PERMANENT DIAGNOSIS: SAME.
That is 370 words — a short report; the corpus median is nearer 600. Every claim made abstractly above is visible in it.
A physician wrote this, and you can tell. The pathologist received a wedge of kidney, described it by eye, inked its margin, sliced it, put it under a microscope, and recorded a conclusion. The prose is addressed to a colleague who already knows the patient.
It has been through OCR, and the damage is still there. Recelpt for Receipt. JATTEST for I ATTEST. IHAVE for I HAVE. submilted for submitted. 8,5 where 8.5 was meant. Stray capitals in mid-sentence: The specimen Is received. And line breaks in the scan became sentence-ending periods in the text — cortically. based, Portions of the. tumor, renal parenchyma, no. extension. Any tool that splits this into sentences will split it wrongly.
The date field is empty. Date of Recelpt: is followed immediately by the next heading. No name, no record number, no date anywhere — that is de-identification, done before release.
It is semi-structured, which is harder than either extreme. There are 22 heading-like labels, from Tumor Type to PERMANENT DIAGNOSIS. Enough structure to tempt you into writing a parser, not enough consistency for one to survive contact with the next hospital’s format.
The cancer type is stated outright — Renal cell carcinoma, Kidney, repeatedly. This is exactly why cancer-type classification is the easy task, and why a good score on it means less than it appears. We come back to that below.
The report contains eight measurements in centimeters. Suppose you wanted tumor size — the quantity that decides T stage — and reached for the first number followed by cm.
You would get 3 cm, from Incidental 3 cm left upper pole renal mass. That is the estimate from imaging before surgery. The measured size is 2.4 cm, further down under Tumor Size.
The rest are traps of their own: 6.6 x 3.7 x 2.9 is the specimen, not the tumor; 12.2 x 8,5 x 2.7 and 11.6 x 8.4 x 4.6 are fat; 1.1 cm is the margin clearance. And 7.0 cm is not a measurement of anything here at all — it is part of the printed definition of pT1.
One document, one quantity, seven wrong answers within easy reach of a regular expression.
Who writes a pathology report, and why
A pathology report is written by a pathologist: a physician who specializes in diagnosing disease by examining tissue and cells, rather than by seeing patients in clinic. This is worth stating plainly, because it is easy to assume the surgeon or the oncologist decides what a tumor is. Neither does. The surgeon removes the tissue. The oncologist treats the patient. Both wait for the pathologist.
The sequence runs roughly like this. A surgeon or radiologist obtains tissue, either by biopsy or by removing the tumor outright. The specimen goes to the pathology laboratory, where it is first examined with the naked eye — measured, described, cut, and sampled. This is the gross examination, and it is where details such as tumor size and surgical margins are established. Selected pieces are then set in wax, sliced thin enough for light to pass through, mounted on slides, and stained so the structures become visible. The pathologist examines these under a microscope and, where appearance alone is not decisive, orders additional stains or molecular tests.
Then they write it down. That document is what establishes the diagnosis, and the rest of the patient’s care is built on it.
A report typically records the type of specimen and where in the body it came from, the gross description, what was seen under the microscope, the results of any additional tests, and — usually set apart at the end — a final diagnosis. For cancer specimens it also carries the measurements that determine stage: how large the tumor is, how deeply it has invaded, whether it extends to the cut edges of the specimen, and how many lymph nodes were examined and found to contain tumor.
Two properties of that document shape everything we will do with it.
It was written for a human reader. The intended audience is one treating clinician who already knows the patient. The pathologist writes in prose, using the vocabulary of their specialty and the conventions of their institution, and can rely on the reader to supply context. Nobody was trying to make the document machine-readable, and nobody anticipated that a program would one day try to parse it. That is precisely why this is an NLP problem and not a database query.
It varies. The corpus spans many contributing institutions and a span of years. Different hospitals use different templates, different section headings, different abbreviations, and different house style. A finding that one pathologist states in a sentence, another records in a checklist. There is no single format to write a parser against — which, again, is the reason a statistical approach is worth reaching for.
Why predict a diagnosis that is already written down
Here is a fair objection, and one you should raise: if the pathologist already stated the diagnosis in the report, what exactly is a model adding by reading the report and predicting the diagnosis?
It is the right question, and it deserves a straight answer rather than a slogan.
For this course, the honest answer is that it is a task where we know the right answers. Every report can be matched to a structured clinical record for the same patient, and that record independently states the cancer type. So we have thousands of documents paired with trustworthy labels. That is what makes it possible to measure whether a text model works at all — to train on some reports, test on reports the model has never seen, and get a number that means something. You cannot learn to evaluate an NLP system on a task where nobody knows the truth. You practice on one where everybody does.
Outside a classroom, the motivating problem is scale. Hospitals and cancer registries hold decades of narrative reports. Turning those into structured, searchable data is real work that is currently done by trained human abstractors reading documents one at a time. Anything that automates part of it — or flags the cases a human should look at — makes it possible to assemble research cohorts that would otherwise be too expensive to build. “Find me every patient in twenty years of records with this tumor type” is a question that free text cannot answer and a structured field can.
The answer is also less explicitly present than you might expect. The same diagnosis can be written a dozen ways. Reports are redacted before release, and ours were recovered by imperfect optical character recognition from scanned pages, so text is sometimes garbled or missing. Mapping the surface language of a report onto a fixed set of categories is genuine work even when a human would find the answer obvious.
And it is the first rung on a ladder. Cancer type is the easiest thing to recover from a report, which is exactly why we start there. Stage is harder: it must be assembled from measurements and node counts scattered through the document, and is often not stated outright. Patient outcome — whether this person was alive five years later — appears nowhere in the report at all. Those are the tasks where a model is doing something a reader cannot simply do by looking, and they are within reach later in this book.
Because the diagnosis usually is stated somewhere in the report, a model that scores well on this task has not necessarily done anything impressive. Before believing any accuracy figure in this book, you will need to know what a model would score by guessing the most common class every single time. We build that habit in the next chapter and never abandon it.