Natural Language Processing

With applications in Medical Diagnosis from Pathology Reports

Preface

This book teaches natural language processing — getting computers to work with human language — through one real problem: reading cancer pathology reports and predicting what the patient was diagnosed with.

The reports are real. About 9,500 of them come from The Cancer Genome Atlas, a large public cancer-research collection. They were written by doctors during ordinary patient care, scanned, stripped of identifying details, and turned back into text by software. They are messy in all the ways real text is messy, and every technique in this book is tried against them.

You do not need to be a programmer, a statistician, or a medical professional to read this book. Each idea is built up from a small example you can check by hand before it is applied to the full collection, and the medical background is explained as it comes up.

What you will learn

• How a paragraph of text becomes a row of numbers a computer can work with.
• How to pair each report with its correct diagnosis, and why getting that pairing right matters more than any model.
• Classic methods that still work remarkably well: word counts, word weighting, and simple classifiers such as logistic regression.
• How modern language models approach the same task, and what they add.
• How to tell whether a result is genuinely good — starting with what you would score by simply guessing the most common answer every time.

How to read this book

Every Python cell in the book runs in your web browser. There is nothing to install and nothing is sent to a server; you can edit any cell and run it again to see what changes.

If you have never programmed before, start with the two appendices. Appendix A — Python Fundamentals covers the Python you need, and Appendix B — NumPy, pandas, and Data Wrangling covers NumPy and pandas, the tools used to handle tables of data. Readers who already know Python can go straight to the Introduction.