1 From Words to Vectors

A toy pathology-report classifier.

1.1 Introduction

How can a computer take a sentence written in ordinary language and turn it into a diagnosis label?

This chapter develops a miniature natural language processing (NLP) pipeline using six synthetic pathology-style reports. The example is deliberately small enough that every intermediate object can be inspected directly. We will not treat the model as a black box. Instead, we will follow the full path from language to mathematics:

sentence
   ↓
word counts
   ↓
vector
   ↓
similarity or learned decision rule
   ↓
predicted diagnosis

The purpose is pedagogical. This toy classifier is not a clinical diagnostic system, and its outputs should never be interpreted as medical conclusions. Its value is that it lets us see, in a few lines of code, the same fundamental ideas that appear in much larger NLP projects.

These code cells are live

Every Python cell below runs in your browser — there is nothing to install and nothing is sent to a server. The first cell you run takes fifteen to thirty seconds while Python, pandas, and scikit-learn download; after that each cell is instant, and the download is cached for your next visit.

The cells share a single Python session, exactly like a notebook, so run them in order from the top. If a cell reports that a name is not defined, you have skipped one of the cells above it. Edit any cell and run it again to see what changes.

1.1.1 Learning objectives

By the end of the chapter, you should be able to:

Explain how a bag-of-words model represents a document numerically.
Interpret the rows, columns, and dimensions of a document-term matrix.
Identify what information bag of words preserves and what it discards.
Use cosine similarity to compare documents.
Transform a new document using an existing vocabulary.
Build simple nearest-neighbor and logistic-regression text classifiers.
Inspect which words influence a linear classifier.
Construct examples that expose weaknesses in a model.
Explain how bigrams partially restore information about word order.

1.2 From language to a matrix

Everything in this chapter rests on a single move: replacing each report with a row of numbers. This section builds that representation, then asks what it keeps and what it throws away. It is the same move the project makes on nine and a half thousand real reports — performed here on six sentences, so that every intermediate result stays small enough to check by hand.

1.2.1 The toy corpus

1.2.1.1 What the real project looks like

This book accompanies AI Campus Project 7, whose corpus is roughly 9,500 pathology reports from The Cancer Genome Atlas. Those reports began as scanned paper: photographed pages, run through optical character recognition to recover their text. Each report carries a patient barcode of the form TCGA-XX-XXXX, and that barcode is the hinge of the whole project — it links the free text of the report to the structured clinical record for the same patient, which is where the diagnosis, stage, and outcome are stored. The Introduction describes that corpus and where it came from.

The task we build toward is cancer-type classification: given the text of a report, predict which of 33 cancer types the patient was diagnosed with.

1.2.1.2 Why we start with six sentences instead

A real report runs several hundred to a few thousand words — the median is around 600 — laid out in sections, thick with abbreviations, and carrying whatever errors the OCR made along the way. A document-term matrix built from the full corpus has roughly 9,500 rows and tens of thousands of columns. You cannot look at a matrix that size. You cannot check its arithmetic by hand, and when a result surprises you, you cannot tell whether the surprise came from your code, the representation, or the data.

So we begin with six synthetic pathology-style reports, one clean sentence each, belonging to three classes: breast, kidney, and lung. Everything in this chapter fits on screen.

What is not simplified is the method. CountVectorizer followed by LogisticRegression — the pipeline this chapter builds — is the actual baseline used on the real corpus in the reference notebooks, where it reaches roughly 95% accuracy against a majority-class baseline of about 35%. You are not learning a toy technique that gets replaced by a serious one later. You are learning the real technique on data small enough to see through. Verifying those published numbers against the data yourself is a later exercise, not a footnote.

The toy corpus is, however, easier than the real thing in ways that matter — the organ name appears in nearly every report, the classes are perfectly balanced, and nothing is misspelled. Section 1.6.3 returns to that gap deliberately, once you have something concrete to compare against.

1.2.1.3 The corpus itself

The correspondence between the two lists matters. The report at position i in texts has the diagnosis at position i in labels.

Two parallel lists are the whole of it here. In the real corpus the same correspondence exists, but it has to be earned: the report text lives in one table and the diagnosis in another, and they are matched on the patient barcode. Getting that join right — and confirming that no patient’s text ended up beside another patient’s label — is the first real task of the project.

Mathematically, the corpus consists of labeled examples

\[ (d_1,y_1),(d_2,y_2),\ldots,(d_6,y_6), \]

where \(d_i\) is a document and \(y_i\) is its known class.

1.2.1.4 Think before coding

Read each report as a human. Which words make the likely diagnosis apparent? You may notice clues such as:

breast, ductal, and estrogen;
renal, kidney, and clear cell;
lung and pulmonary.

Humans bring biological knowledge and language comprehension to the task. The computer initially has neither. We must decide what measurable information to extract from the text.

1.2.2 Turning language into a matrix

Scikit-learn’s CountVectorizer builds a vocabulary from the corpus and counts how many times each vocabulary word appears in each document.

To inspect the result, convert the sparse matrix to a pandas DataFrame:

This is the first “magic trick”:

Language has become a matrix.

The output is called a document-term matrix.

Each row represents one document.
Each column represents one vocabulary term.
Each entry records the number of times that term appears in that document.

If \(X_{ij}=2\), for example, then vocabulary term \(j\) occurs twice in document \(i\).

1.2.3 What does one row mean?

Consider the report:

clear cell renal carcinoma involving the kidney

Its row can be viewed conceptually as follows:

breast	carcinoma	cell	clear	kidney	lung	renal	…
0	1	1	1	1	0	1	…

The sentence is no longer stored as a sentence. It has become a vector:

\[ x=(0,1,1,1,1,0,1,\ldots). \]

If the vocabulary contains \(V\) terms, then every document is represented by a vector in \(\mathbb{R}^V\):

\[ x_i\in\mathbb{R}^V. \]

This allows us to apply linear algebra, statistics, and machine-learning algorithms to language.

1.2.3.1 What did we preserve?

The representation preserves:

which vocabulary words occur;
how often each word occurs.

1.2.3.2 What did we lose?

It discards or largely ignores:

word order;
grammar;
sentence structure;
context;
many relationships between words;
the meanings of unfamiliar words.

The name bag of words captures this limitation. It is as though the words were placed into a bag and shaken: we can count the contents, but their original order is gone.

1.2.4 Dimensions and vocabulary size

We can inspect the learned vocabulary:

Its size is:

The matrix dimensions are:

In general,

\[ X\in\mathbb{R}^{n\times V}, \]

where:

\(n\) is the number of documents;
\(V\) is the vocabulary size.

For this toy corpus, \(n=6\). In a real pathology-report corpus, there may be thousands of reports and tens of thousands of vocabulary terms. The resulting matrix can therefore be extremely large.

Most entries are nevertheless zero: any one report contains only a small fraction of the complete vocabulary. This is why CountVectorizer returns a sparse matrix rather than an ordinary dense NumPy array.

Instructor prompt

If a corpus contains 9,523 reports and a vocabulary of 30,000 terms, what are the dimensions of its document-term matrix? How many entries would the corresponding dense matrix contain?

1.2.5 Examining individual features

A column of the matrix is a feature. We can inspect a single feature:

Or we can focus on several potentially useful terms:

The result has a simple pattern:

document	breast	renal	kidney	lung	pulmonary
0	1	0	0	0	0
1	1	0	0	0	0
2	0	1	1	0	0
3	0	1	0	0	0
4	0	0	0	1	0
5	0	0	0	1	1

Even before training a classifier, we can see that these columns contain information about the diagnosis.

Discussion question

Could you almost classify these six reports using only the five displayed columns? What information would be lost by discarding all the other columns?

1.2.6 Attaching the labels

So far, the matrix contains only input features. Let us attach the known output labels for inspection:

This gives us the central supervised-learning structure:

\[ X\longrightarrow y, \]

where:

\(X\) is the matrix of word-count features;
\(y\) is the vector of diagnosis labels.

A classifier attempts to learn a rule

\[ f:X\longrightarrow y \]

that can also be applied to documents whose labels are unknown.

1.2.7 Looking for words associated with diagnoses

Because the features are numerical, we can compute the mean word count within each diagnosis class:

To focus on our five example terms:

Conceptually, we obtain a table like this:

diagnosis	breast	renal	kidney	lung	pulmonary
Breast	1.0	0.0	0.0	0.0	0.0
Kidney	0.0	1.0	0.5	0.0	0.0
Lung	0.0	0.0	0.0	1.0	0.5

Words characteristic of each class begin to emerge. This is an informal form of feature selection: we are looking for terms whose distributions differ between classes.

In probability language, the data suggest that

\[ P(\text{“renal”}\mid\text{Kidney}) > P(\text{“renal”}\mid\text{Breast}). \]

With only two examples per class, these estimates should not be trusted statistically. The point is to expose the logic that a classifier will later formalize.

1.3 Comparing documents geometrically

Once reports are vectors, how alike two of them are becomes a question of arithmetic. That alone is enough to label a new report, before we have trained anything.

1.3.1 Measuring similarity between reports

Once documents are vectors, we can compare them geometrically. A common NLP measure is cosine similarity:

\[ \operatorname{cosim}(x,z) = \frac{x\cdot z}{\lVert x\rVert\lVert z\rVert}. \]

For nonnegative word-count vectors, cosine similarity ranges from 0 to 1:

a value near 1 means the vectors point in similar directions;
a value near 0 means they share little vocabulary.

Compute all pairwise similarities:

The diagonal entries equal 1 because every document is perfectly similar to itself.

The pedagogically interesting question is whether reports with the same label tend to have higher similarity than reports with different labels.

Linear-algebra interpretation

We are measuring linguistic resemblance using the angle between document vectors.

1.3.1.1 Why not simply use the dot product?

Longer documents usually contain more words and therefore tend to have larger vector magnitudes. The cosine normalizes by both magnitudes, emphasizing direction rather than document length.

1.3.2 Transforming a new report

Now consider a report that was not in the original corpus:

Transform it using the vectorizer that was fitted on the original reports:

Then inspect its vector:

The distinction between fit_transform and transform is essential:

fit_transform(texts) learns a vocabulary from the training corpus and transforms that corpus.
transform(new_text) applies the already learned vocabulary to new text.

We do not use a second fit_transform call on the new report. Doing so would create a different feature space, so the new vector would no longer be comparable to the training vectors.

1.3.2.1 Out-of-vocabulary words

Try this example:

If nephrectomy never occurred in the training corpus, the current vectorizer has no column for it. The word is ignored. This is called an out-of-vocabulary problem.

1.3.3 Finding the most similar report

We can compare the new report to every report in the corpus:

Find the index of the largest value:

This creates a primitive diagnostic rule:

Assign the new report the label of the most linguistically similar old report.

It is already a form of classification, even though we have not yet introduced a formal classifier.

1.3.3.1 A useful caution

Similarity is only as meaningful as the representation. Two documents can share many words and still express opposite conclusions. Consider a report that reuses the exact wording of one of our kidney cases in order to rule the diagnosis out:

The third entry is 1.0. The new report is not merely similar to report 2 — under this representation it is indistinguishable from it:

So the rule we just built assigns a label with the highest score it can produce:

The single word carrying the entire clinical meaning, no, never occurred in the six training reports. The vectorizer has no column for it, so transform discards it without warning — the out-of-vocabulary problem from the previous section. What remains is precisely the vocabulary of a positive kidney finding.

Note that the failure would survive even if no were in the vocabulary. One extra count in one column cannot outweigh seven matching ones, and nothing in the representation records that no modifies what follows it. Word order was discarded when we built the matrix. We take up that deeper problem in Section 1.5.2.

1.3.4 Features

Everything so far has rested on one particular choice, made back in Section 1.2.1 and never revisited: one feature per vocabulary word, valued by how many times that word occurred. That gave us a \(6 \times 26\) matrix, and every result since — the similarities, the nearest neighbor, the vector that stood in for a new report — followed from it.

It is worth saying plainly that this was a choice, not a law. Nothing about classification requires it.

1.3.4.1 What a feature actually is

A feature is any function that takes a document and returns a number.

\[ f_j : \text{document} \longrightarrow \mathbb{R} \]

Pick \(V\) such functions and every document becomes a vector \(x = (f_1(d), f_2(d), \ldots, f_V(d))\) in \(\mathbb{R}^V\). Bag of words is the special case where \(f_j(d)\) counts occurrences of vocabulary word \(j\). That is one family among many.

The point that makes this powerful is that the classifier cannot tell the difference. LogisticRegression receives a matrix of numbers and learns a weight for each column. It has no idea whether column 3 means “how many times the word renal appeared”, “how many words are in this report”, or “the price of tea”. It sees a column, and it learns how much that column should count toward each class.

So the columns are ours to design.

1.3.4.2 The same pipeline, different columns

Here are three features for the same six reports that have nothing to do with per-word counts:

Three columns instead of twenty-six, and not one of them is a word count in the bag-of-words sense. The first measures length. The second is binary — presence, not frequency. The third consults a list we wrote by hand, which means it carries a small amount of human knowledge about anatomy that no amount of counting could have supplied.

This matrix could be handed to exactly the same classifiers. Nothing else in the pipeline would need to change.

1.3.4.3 Things a feature can measure

Once you accept that a feature is just a number extracted from a document, the space of options opens up considerably. A rough map:

Variations on counting words. Use presence instead of frequency, so a word that appears nine times counts the same as one that appears once. Weight rare words more heavily than common ones, which is what TF–IDF does. Count pairs of adjacent words rather than single words, which is what Section 1.5.3 introduces. These stay within the bag-of-words family but change what it emphasizes.

Properties of the document as a whole. Its length. Its average sentence length. Whether it is longer than usual for its type. Readability scores.

Presence of a pattern. Whether a measurement in centimetres appears anywhere. Whether a date appears. Whether the document contains a percentage. These are usually found with regular expressions rather than word lookup.

Counts from a curated list. How many words come from a sentiment lexicon, an anatomy list, a drug list, a list of hedging words like possible and suggestive of. This is where a domain expert’s knowledge enters the model most directly.

Structure. Whether the report has a section headed FINAL DIAGNOSIS. How many sections it has. Whether the diagnosis section is at the top or the bottom. Real pathology reports are full of this kind of structure, and it is invisible to a model that only counts words.

Things that are not the text at all. Which institution produced the report. What year it was written. What kind of specimen it describes. These are often the most predictive features available, and often the most dangerous — a model that learns “hospital B mostly sends kidney cases” has learned something true about the dataset and useless about pathology.

1.3.4.4 Why this is where the thinking happens

A model can only learn from what it is shown. If a distinction is not visible in the features, no amount of training data or clever optimization will recover it — the information simply is not in the matrix.

That is the real reason feature design matters. It is the channel through which everything a human knows about the problem reaches the model. Our organ-word list above is a tiny example: a bag-of-words model would have to learn from examples that renal and kidney both point at the same organ, and with six reports it cannot. Writing them into one list hands it over for free.

The trade-off runs in both directions. Hand-designed features are few, interpretable, and cheap to compute, but they demand expertise and they cap what the model can notice at whatever their designer thought of. Bag of words needs no expertise and lets the data speak, but it is enormous, mostly zeros, and blind to everything that is not a word. Later chapters introduce a third option — features that are learned rather than designed or counted — which is what makes modern language models work.

1.3.4.5 Next

The next section builds a classifier from the ground up on a set of four hand-designed features, each one a deliberate hypothesis about what makes a piece of writing positive or negative. Watching weights get attached to features you chose yourself makes the mechanism far easier to see than watching them attached to twenty-six word counts.

1.4 Learning a decision rule

Nearest neighbors predicts by comparison with stored examples. Logistic regression instead learns a weight for every word — and those weights can be read back.

1.4.1 A nearest-neighbor classifier

Jumped straight to this section? Run this first

The cells below need the corpus, the fitted vectorizer, and the new report from earlier in the chapter. This one cell rebuilds all of it, so you can start here without scrolling back.

Already worked through the chapter in order? Skip this — your session already holds these objects, and re-running it simply rebuilds them identically.

The preceding rule is precisely the idea behind a one-nearest-neighbor classifier:

The expected prediction is:

['Kidney']

The full logic is visible:

new sentence
     ↓
word-count vector
     ↓
similarity to known vectors
     ↓
label of the nearest report

1.4.1.1 Why this model is useful pedagogically

Nearest neighbors provides a classifier without hiding the reasoning. A student can inspect the matrix, calculate the similarities, locate the nearest report, and reproduce the prediction manually.

1.4.1.2 Why it is not enough

With only six examples, the model is fragile. It memorizes individual examples rather than learning reliable population-level patterns. In a serious application we would need:

many more labeled reports;
a principled train/validation/test split;
evaluation on unseen patients;
careful analysis of errors and class imbalance.

1.4.2 Logistic regression

Nearest neighbors predicts by comparison with stored examples. Logistic regression does something different: it learns a weight for every feature, and a prediction is a weighted sum of those features pushed through one function.

That mechanism is worth building by hand before turning it loose on pathology reports. So this section steps away from the corpus entirely and works a smaller problem end to end — four sentences, four features, and weights we choose ourselves rather than fit. Once every arithmetic step is visible, we come back to the reports at the end.

1.4.2.1 A smaller problem: four product reviews

Suppose we are reading customer reviews of a product and want to decide whether each one is positive or negative. That is a binary classification: two classes rather than the three we have been using.

We write \(y^{(i)}\) for the label of review \(i\), so \(y^{(1)}=1\) and \(y^{(2)}=0\). Throughout this section a superscript in parentheses indexes the review and a subscript indexes the feature: \(x^{(3)}_2\) is the second feature of the third review.

1.4.2.2 Choosing features by hand

Chapter 1 has so far used one feature per vocabulary word. Here we do the opposite: four hand-designed features, each one a guess about what signals sentiment.

Two of them count words from small sentiment lexicons:

The four features for a review \(x\) are:

feature	meaning	why we might expect it to help
\(x_1\)	how many words appear in the positive lexicon	praise words suggest a positive review
\(x_2\)	how many words appear in the negative lexicon	complaint words suggest a negative one
\(x_3\)	\(1\) if the word `no` appears, else \(0\)	negation often accompanies a complaint
\(x_4\)	\(\ln(\text{number of words})\)	a hunch: long reviews are more often rants

The last two are worth pausing on. \(x_3\) is binary — it records only presence, not how many times. And \(x_4\) is a hypothesis in numerical form: that an annoyed customer writes at greater length. We take the logarithm because the difference between a 5-word and a 50-word review matters far more than the difference between 500 and 550.

Laid out as a table, with each row a review and each column a feature:

review	\(x_1\)	\(x_2\)	\(x_3\)	\(x_4\)	\(y\)
\(x^{(1)}\) great little speaker and no complaints about the sound	1	0	1	2.197	1
\(x^{(2)}\) the case broke after a week and the seller was awful	0	2	0	2.398	0
\(x^{(3)}\) works well and the price is nice	2	0	0	1.946	1
\(x^{(4)}\) poor quality and there is no way to return this terrible thing	0	2	1	2.485	0

Four sentences have become four points in \(\mathbb{R}^4\) — the same move as Chapter 1, but with features we designed instead of counted.

Notice that \(x^{(1)}\) is already awkward. It is a positive review, but it contains the word no — from the phrase no complaints, which is praise. The \(x_3\) feature will push it the wrong way. Keep an eye on that review; it comes back.

1.4.2.3 The logit: one weighted sum

Logistic regression assigns a weight \(w_j\) to each feature and one bias \(b\), then computes a single number:

\[ z = w_1x_1 + w_2x_2 + w_3x_3 + w_4x_4 + b = \mathbf{w}\cdot\mathbf{x} + b . \]

This number is called the logit. Large and positive means “confidently class 1”; large and negative means “confidently class 0”; near zero means undecided.

Normally the weights are learned from data. To keep every step visible we will simply choose them, picking values that encode the intuitions in the table above:

Read those signs: positive words push toward positive (\(w_1>0\)), negative words push away (\(w_2<0\)), no pushes mildly negative (\(w_3<0\)), and length pushes mildly negative (\(w_4<0\)), which is the “long reviews are rants” hunch.

Take review 1, with \(x^{(1)} = (1,\,0,\,1,\,2.197)\):

\[ \begin{aligned} z^{(1)} &= (2.5)(1) + (-2.5)(0) + (-1.0)(1) + (-0.5)(2.197) + 0.5\\ &= 2.5 - 0 - 1.0 - 1.0985 + 0.5\\ &= 0.9015 . \end{aligned} \]

Positive, so the model leans “positive review” — but only just. The praise word earned \(+2.5\), and then no and the review’s length gave back most of it.

1.4.2.4 The sigmoid: from a score to a probability

A logit can be any real number, but we want a probability between 0 and 1. The sigmoid (or logistic) function does that:

\[ \sigma(z) = \frac{1}{1 + e^{-z}} . \]

It has exactly the properties we need:

it is squeezed into \((0,1)\) for every input, so the output can be read as a probability;
\(\sigma(0) = 0.5\) — a logit of zero is a coin flip;
it is increasing, so a larger logit always means a larger probability;
it saturates: once \(|z|\) is past about 5, moving further changes almost nothing.

We define \(\hat{y} = \sigma(z)\) as the model’s estimate of \(P(y=1\mid x)\), and predict class 1 when \(\hat{y} > 0.5\) — which, because \(\sigma(0)=0.5\), is the same as predicting class 1 when \(z > 0\).

All four come out right. But look at the spread: reviews 2, 3, and 4 are decided emphatically, while review 1 scrapes through at \(0.71\) — exactly the review whose no worked against it.

Here is the curve, with our four reviews marked on it:

The flatness at the edges is the saturation point made visual: reviews 2 and 4 sit so far left that a large change in \(z\) would barely move their probability at all.

1.4.2.5 Softmax: the same idea for more than two classes

The sigmoid handles two classes. For three or more, the generalization is the softmax, which takes one logit per class and normalizes them:

\[ P(y=c\mid x) = \frac{e^{z_c}}{\sum_k e^{z_k}} . \]

These are not two unrelated functions. The sigmoid is the two-class softmax. If we treat our single logit \(z\) as the score for class 1 and fix the score for class 0 at \(0\), the softmax gives

\[ \frac{e^{z}}{e^{z} + e^{0}} = \frac{e^z}{e^z + 1} = \frac{1}{1 + e^{-z}} = \sigma(z), \]

where the last step divides top and bottom by \(e^z\). Check it on review 1:

Section 1.4.3 applies the softmax in its three-class form to the pathology model.

1.4.2.6 Cross-entropy loss: how wrong is a prediction?

We chose the weights by hand. To learn them we need a score for how good a set of weights is — a loss that is small when predictions are right and large when they are wrong.

Accuracy will not do. It only asks whether \(\hat{y}\) landed on the correct side of \(0.5\), so it cannot tell a lucky \(0.51\) from a confident \(0.99\), and it gives no signal about which direction to nudge a weight.

The right loss asks: what probability did the model assign to the answer that actually occurred? For a true label of 1 that is \(\hat{y}\); for a true label of 0 it is \(1-\hat{y}\). Both cases fold into one expression:

\[ L(\hat{y}, y) = -\Big[\, y\log \hat{y} + (1-y)\log(1-\hat{y}) \,\Big]. \]

The trick is that \(y\) is 0 or 1, so one of the two terms always vanishes. When \(y=1\) the loss is \(-\log\hat{y}\); when \(y=0\) it is \(-\log(1-\hat{y})\). This is the cross-entropy (or negative log-likelihood) loss. It is \(0\) when the model puts probability 1 on the truth, and grows without limit as the model becomes confidently wrong.

For review 1, with \(y^{(1)}=1\) and \(\hat{y}^{(1)}=0.7113\):

\[ L^{(1)} = -\log(0.7113) = 0.3407 . \]

For review 2, with \(y^{(2)}=0\) and \(\hat{y}^{(2)}=0.0033\):

\[ L^{(2)} = -\log(1 - 0.0033) = -\log(0.9967) = 0.0033 . \]

Review 1 carries almost all of it — its loss of \(0.34\) against roughly \(0.001\) to \(0.01\) for the others. Every one of the four predictions is correct, yet the loss is far from zero, because being right by a narrow margin is still counted as a shortcoming. That is the property accuracy lacks and the reason this is the quantity worth minimizing.

1.4.2.7 Stochastic gradient descent: how the weights are found

We picked \(\mathbf{w}\) and \(b\) out of the air. Fitting means searching for the values that make the average loss as small as possible — and the loss gives a direction to search in.

For cross-entropy over a sigmoid, the derivative works out to something strikingly simple:

\[ \frac{\partial L}{\partial w_j} = (\hat{y} - y)\, x_j , \qquad \frac{\partial L}{\partial b} = (\hat{y} - y). \]

Read it in words: the gradient for a feature is the prediction error times that feature’s value. If the model is right, \(\hat{y}-y\) is near zero and nothing moves. If it is confidently wrong, the error is close to \(\pm 1\) and every weight moves in proportion to how strongly its feature was present.

Stochastic gradient descent turns that into an algorithm. Repeat many times:

take one training example (that is the stochastic part — one at a time, in random order, rather than the whole corpus at once);
compute \(z\), then \(\hat{y} = \sigma(z)\);
compute the error \(\hat{y} - y\);
nudge every weight against its gradient, \(w_j \leftarrow w_j - \eta\,(\hat{y}-y)\,x_j\), and likewise for \(b\).

The learning rate \(\eta\) sets the size of the step: too small and the search crawls, too large and it overshoots and thrashes. One pass over the whole corpus is an epoch, and training runs for many.

For review 1, \(\hat{y}-y = 0.7113 - 1 = -0.2887\). The error is negative, so each weight is pushed up in proportion to its feature — most of all \(w_4\), because \(x_4 = 2.197\) is the largest feature value. The model would learn to lean less on review length.

We stop here rather than run the loop, because scikit-learn’s fit does this for us. It uses a more sophisticated optimizer than plain SGD, but the principle — follow the gradient of the loss downhill until it stops improving — is the one described above, and it is what Section 1.4.3 means when it says the fit has no closed-form solution.

1.4.2.8 Back to the pathology reports

That is the entire mechanism: features, a weighted sum, a squashing function, a loss, and a search. The pathology classifier is the same machine with two changes — the features are word counts rather than four hand-picked signals, and there are three classes rather than two, so softmax replaces sigmoid.

Jumped straight to this part? Run this first

The cells below need the corpus, the fitted vectorizer, and the new report from earlier in the chapter. This one cell rebuilds all of it, so you can start here without scrolling back.

Already worked through the chapter in order? Skip this — your session already holds these objects, and re-running it simply rebuilds them identically.

To see the model’s scores in probability-like form:

The predicted class is the class with the largest value.

1.4.2.9 Are these clinical probabilities?

No. A value such as 0.63 is a model-generated class probability under this fitted toy model. It is not automatically:

the true probability that the patient has that cancer;
a measure of medical certainty;
a calibrated risk estimate;
evidence that the model is safe for clinical use.

Prediction confidence and clinical certainty are not the same thing.

1.4.3 Looking inside the linear classifier

Section 1.4.2 called predict_proba and read three numbers off the output. This section takes that apart. There are two separate questions, and it is worth keeping them apart:

Given coefficients, how does the model turn a document into probabilities?
Where did those coefficients come from in the first place?

1.4.3.1 One score per class

The model holds one weight per word per class, plus one offset per class. For class \(c\) and document vector \(x\) it computes a single number:

\[ s_c(x)=\beta_{c0}+\beta_c^\mathsf{T}x =\beta_{c0}+\sum_{j=1}^{V}\beta_{cj}\,x_j . \]

Read that sum literally. Every vocabulary word \(j\) contributes \(\beta_{cj}x_j\) — its weight for class \(c\), times how many times it occurred. Words absent from the document have \(x_j=0\) and contribute nothing. So for the kidney class,

\[ s_{\text{Kidney}}(x) = \beta_{\text{Kidney},0} +\beta_{\text{Kidney},\text{renal}}\,x_{\text{renal}} +\beta_{\text{Kidney},\text{kidney}}\,x_{\text{kidney}} +\cdots. \]

With 3 classes and \(V=26\) words, that is a \(3\times 26\) table of weights and 3 offsets. Those are exactly the arrays scikit-learn fitted:

1.4.3.2 From scores to probabilities

Three scores are not yet probabilities: they can be negative, and they need not sum to one. The fix is the softmax, which exponentiates each score and divides by the total:

\[ P(y=c\mid x)=\frac{e^{s_c(x)}}{\sum_{k}e^{s_k(x)}} . \]

Exponentiating makes every value positive; dividing by the sum makes them add to one. The ordering is untouched — the largest score is still the largest probability — so predict is just the arg-max of the scores.

This is the whole of predict_proba, and you can check that claim rather than take it on faith. Compute the scores by hand from the fitted arrays:

Now apply the softmax by hand:

Nothing is hidden. predict_proba is one matrix multiplication and one softmax over the coefficients the model already showed you.

Where each object enters

X and labels are used once, during fit, to choose the coefficients. After that the training data is not consulted again. A prediction on new_X touches only coef_, intercept_, and new_X itself — which is exactly how logistic regression differs from the nearest-neighbor model of Section 1.4.1, where every prediction compares against all six stored documents.

1.4.3.3 Where the coefficients came from

Fitting means choosing \(\beta\). The criterion is this: pick the coefficients that make the training labels as probable as possible.

Each training document \(x_i\) has a known class \(y_i\). Any candidate \(\beta\) assigns some probability \(P(y_i\mid x_i)\) to the correct answer. A good \(\beta\) makes those probabilities large across all six documents. Multiplying many small numbers is numerically awkward, so we take logarithms and flip the sign, which turns “maximize” into “minimize”:

\[ \mathcal{L}(\beta) = -\sum_{i=1}^{n}\log P(y_i\mid x_i) \;+\; \frac{1}{2C}\sum_{c,j}\beta_{cj}^{2}. \]

The first term is the cross-entropy loss: it is zero when the model assigns probability 1 to every correct label, and grows without bound as the model becomes confidently wrong.

The second term is a penalty on large coefficients, and it is not optional here. Our six documents are perfectly separable — a model can classify all of them correctly. Left alone, the fit would keep enlarging the weights to push the training probabilities closer and closer to 1, and the coefficients would run away to infinity without ever reaching a best value. The penalty stops that by charging for size. C sets the exchange rate: small C means an expensive penalty and timid coefficients, large C means a cheap one and bold coefficients.

You can watch that happen:

At C=0.01 the model is squeezed so hard it barely commits — near one-third each. At C=100 it is emphatic. The data did not change; only the price of confidence did. C=1.0 is scikit-learn’s default and the value used everywhere else in this chapter.

Unlike a line of best fit, this minimization has no closed-form solution — there is no formula to plug numbers into. The value is found by iterative numerical search: start from zero, compute the loss and the direction that decreases it, take a step, repeat until the improvement becomes negligible. That is what max_iter=1000 bounds. If a model warns that it failed to converge, it ran out of steps before settling, and the coefficients you get are wherever it happened to stop.

1.4.3.4 Reading the coefficients

Because the score is a plain sum, a large positive \(\beta_{cj}\) means word \(j\) pushes a document toward class \(c\). So the weights can be read directly:

The exact numbers depend on the fitted model, but we expect patterns such as:

Breast
breast
ductal
estrogen

Kidney
renal
kidney
clear

Lung
lung
pulmonary
adenocarcinoma

This is the second “magic trick”:

Words became numbers.
The model learned which words support each class.

1.4.3.5 Interpret with care

Large coefficients reveal how the model uses a feature; they do not establish a medical or causal relationship. In this synthetic corpus, obvious organ words dominate because we deliberately wrote the reports that way.

1.4.4 Test the classifier interactively

The following helper function displays a prediction and all class scores:

Try it:

Then invite students to write new reports and predict the outcome before running the cell.

1.5 Where bag of words breaks down

The limits of the representation teach as much as its successes. Here we go looking for them deliberately.

1.5.1 Break the model intentionally

A good way to understand a model is to construct examples on which it should struggle.

1.5.1.1 Case 1: Too little distinguishing information

The sentence contains terms that occur in several classes but lacks a strong organ-specific clue.

1.5.1.2 Case 2: Contradictory clues

The report contains high-value terms for all three classes.

1.5.1.3 Case 3: A paraphrase using unseen vocabulary

A human with medical knowledge might connect nephrectomy with the kidney. The model cannot make that semantic connection if the word never appeared in its training vocabulary.

1.5.1.4 Questions to ask

Which words in the input does the vectorizer recognize?
Which words are ignored?
Which recognized words favor each class?
Does the predicted label make sense given the available features?
Is the failure caused by the representation, the training data, the classifier, or some combination?

The quality of a prediction is constrained by the information present in both the input and the training data.

1.5.2 The central weakness of bag of words

Consider two sentences:

Their meanings are nearly opposite, but their word-count vectors are very similar:

Compute their similarity:

The representation sees extensive vocabulary overlap. It does not understand that not reverses the central claim.

1.5.2.1 A clinically sharper example

Compare:

No evidence of metastatic carcinoma.

with:

Evidence of metastatic carcinoma.

The difference is one word—no—but the clinical meanings are opposite.

This exposes the key limitation:

Bag of words knows which words occurred. It does not genuinely understand what the sentence means.

1.5.3 Bigrams as a first improvement

One response is to preserve short sequences of words. A unigram is a single word; a bigram is a sequence of two adjacent words.

Change:

CountVectorizer()

to:

CountVectorizer(ngram_range=(1, 2))

Now the representation includes both unigrams and bigrams:

The vocabulary can now contain phrases such as:

no evidence
evidence of
metastatic carcinoma
renal carcinoma
ductal carcinoma

The phrase no evidence is distinct from the individual words no and evidence. The model has gained a limited amount of word-order information.

This does not solve language understanding. Bigrams still cannot capture long-range context, subtle negation, ambiguity, or background knowledge. But they are a natural first improvement.

Discovery prompt

Before introducing the term bigram, ask: “How could we change the representation so that the computer can distinguish evidence of cancer from no evidence of cancer?” Students may independently suggest keeping neighboring words together.

1.6 Beyond the toy example

What six synthetic documents can and cannot demonstrate, and what a real corpus forces a project to add.

1.6.1 The complete classroom progression

The chapter can be taught as a 45–60 minute guided investigation.

Step	Guiding question	Concept introduced
1	Can language become numbers?	Vectorization
2	What does a row represent?	Document vector
3	What does a column represent?	Feature
4	What are the matrix dimensions?	Documents × vocabulary
5	Which words distinguish diagnoses?	Feature association
6	Which reports are similar?	Cosine similarity
7	Can we classify a new report?	Generalization to new input
8	Which old report is closest?	Nearest neighbors
9	Can a model learn word weights?	Logistic regression
10	Which words did it learn?	Coefficient interpretation
11	Can we fool it?	Error analysis
12	What information did bag of words lose?	Representation limits
13	How might we improve it?	Bigrams and richer NLP

The questions should come before the terminology whenever possible. Students first encounter a problem, then learn the name of a method that addresses it.

1.6.2 Suggested exercises

1.6.2.1 Exercise 1: Read a vector

Select one row of bow and reconstruct the words with nonzero counts.

Which parts of the original sentence cannot be reconstructed from this output?

1.6.2.2 Exercise 2: Compare same-class and different-class similarity

Calculate:

the mean cosine similarity between pairs sharing a label;
the mean cosine similarity between pairs with different labels.

Does same-class similarity appear larger in this corpus? Why would six examples be insufficient for a reliable conclusion?

1.6.2.3 Exercise 3: Remove the obvious organ word

Compare predictions for:

breast tissue demonstrates invasive ductal carcinoma

and:

invasive ductal carcinoma with estrogen receptor positivity

Can the model still infer the class after breast is removed? Which remaining words matter?

1.6.2.4 Exercise 4: Create an adversarial report

Write a short report that a human would associate with one class but that the model predicts as another. Explain why the bag-of-words features mislead it.

1.6.2.5 Exercise 5: Compare unigram and bigram vocabularies

Fit two vectorizers:

Compare their vocabulary sizes and identify the new phrase features.

1.6.2.6 Exercise 6: Add training examples

Add two more synthetic reports per class. Refit both models and inspect how the learned coefficients change.

Questions:

Do the same terms remain important?
Does the prediction for an ambiguous report change?
Did you accidentally insert the label directly into every report?

1.6.3 Where the toy example stops being trustworthy

The example makes the mechanics visible, but it omits nearly every difficulty of real clinical NLP:

The corpus has only six documents.
The reports are synthetic and unusually short.
The classes are perfectly balanced.
Organ names often reveal the answer directly.
There are no misspellings, abbreviations, formatting artifacts, or contradictory sections.
There is no separation into training, validation, and test sets.
There is no patient-level grouping or protection against data leakage.
There is no evaluation of rare classes.
There is no analysis of calibration, fairness, or clinical consequences.
The labels and reports were constructed to align cleanly.

The model may classify its tiny training world successfully while learning very little that transfers to real reports.

That is not a flaw in the lesson. It is the lesson’s final transition:

Once we understand the miniature pipeline, what must change before we can trust results on a real corpus?

1.6.4 From the toy example to a real NLP project

The larger workflow is:

Raw reports and clinical metadata
              ↓
Build labeled examples (text, diagnosis)
              ↓
Create train, validation, and test sets
              ↓
Learn a vocabulary from training text only
              ↓
Transform language into numerical features
              ↓
Fit a classifier on the training set
              ↓
Make modeling choices using validation data
              ↓
Evaluate once on untouched test data
              ↓
Analyze errors and improve the representation

This toy example focuses on the middle of that pipeline: representation, similarity, classification, and interpretation. A real project must also establish where the text and labels came from, how the data were split, and whether the final evaluation measures genuine performance on unseen cases.

1.7 Conclusion

The central idea is simple but powerful:

\[ \text{document} \longrightarrow x\in\mathbb{R}^V. \]

Once a document becomes a vector, we can:

inspect individual features;
compare documents using geometry;
find nearest neighbors;
learn a classification rule;
examine learned coefficients;
test the system on new text;
diagnose failures in the representation.

But converting words to numbers is not the same as understanding language. The near-equivalence of evidence of metastatic carcinoma and no evidence of metastatic carcinoma under a unigram bag-of-words representation makes that limitation impossible to ignore.

That failure motivates the next stage of NLP. Bigrams preserve short phrases; TF–IDF changes how terms are weighted; embeddings represent similarity of meaning; contextual models account for surrounding words; and transformer-based language models capture relationships that a simple count matrix cannot.

The tiny matrix is therefore more than a demonstration. It is a map of the entire subject: representation makes computation possible, and the limitations of the representation determine what the model can and cannot learn.