Learned it, or memorised it?
Overfitting and underfitting, explained without the noise. Machine learning, simply · 05
Last time a curve with eight bends passed through every point we had, then said a plant would be below the ground one week later. This time we look at that trap head on. Every model can fail in two opposite ways: it can be too simple to see the pattern, or so clever it learns the noise. Same three chapters: where the idea came from, how the math shows it, and how you catch it the same day.
Chapter 1 · The story
Long before computers, people worried about rules that explain too much.
In the 1320s the English friar William of Ockham argued for the simplest explanation that does the job. The idea later got his name: Occam’s razor. Shave off whatever you do not need.
In 1901 Carl Runge showed the math version, which we met in #04. Give a polynomial too many bends and it wobbles wildly between the points it fits perfectly.
The turning point came in 1931. An American psychologist, S. C. Larson, fitted a prediction rule on one group of people, then tried the same rule on a fresh group. It got clearly worse. He called it shrinkage. That is the first published version of the habit this whole article is about: test on data the model has never seen.
After that, the fixes kept coming. In 1970 Arthur Hoerl and Robert Kennard published ridge regression: give the model a small fine for every big number it uses, so it cannot twist itself around the noise. In 1974 Mervyn Stone set out cross-validation: hold back a slice, test, rotate, repeat. In 1992 Stuart Geman, Elie Bienenstock and René Doursat gave the two failures the names statisticians still use, bias and variance. In 2014 a Toronto team led by Geoffrey Hinton published dropout, which switches off random parts of a neural network while it trains so no single part can memorise.
Seven hundred years, one worry: a rule that fits the past too well can fail the future. So how do you tell?
Chapter 2 · The math
Ten students. For each one, the hours they studied and their exam score out of 100:
Hours: 0.5, 1.5, 2.5, 3.5, 4.5, 5.5, 6.5, 7.5, 8.5, 9.5
Score: 29, 46, 67, 69, 86, 80, 91, 95, 96, 94
More study helps, then it levels off. And there is noise: the student on 5.5 hours scored lower than the one on 4.5. Maybe they were ill that day.
Now five new students take the same exam. We keep them aside and do not fit on them:
Hours: 1, 3, 5, 7, 9
Score: 40, 66, 88, 94, 94
Fit three models on the first ten, then check all three on the new five. To keep it readable, “misses” below means the average number of marks the model is off by.
Too simple: a straight line. It misses the old class by 8.7 marks and the new students by 8.3. Bad on both, and about equally bad. It cannot bend, so it cannot see that scores level off. It even predicts more than 100 marks for 10 hours of study. That is underfitting.
Just right: one bend. The curve from #04, with an hours² column. It misses the old class by 3.8 and the new students by only 1.8. It found the shape and ignored the bumps.
Too clever: eight bends. It goes through all ten old students exactly. Misses on the old class: 0.0. Misses on the new students: 33.8. It says a student who studies 9 hours gets 138 out of 100, and one who studies 1 hour gets −21. That is overfitting: it learned the noise, the ill student and all.
The U-shaped curve
Try every size from a straight line up to eight bends and plot both kinds of misses.
The black line, misses on the class you fitted, only goes down. Every extra bend lets the curve chase the dots a little closer. If that was all you looked at, you would always pick the most complex model.
The pink line, misses on new students, drops first, bottoms out at one bend, and then climbs. At eight bends it shoots up to 33.8.
The black line tells you how well the model remembers. The pink line tells you how well it learned. Only the pink one matters.
Too stiff, or too jumpy
There is one more way to see the two failures. Imagine six different classes of ten students, all drawn from the same real pattern, and fit each class separately.
The straight lines come out almost identical, and all of them miss the real curve in the same places. That steady, repeatable error is bias. The model is too stiff to reach the truth, however much data you give it.
Get Karthick Raja Mohan’s stories in your inbox
Join Medium for free to get updates from this writer.
The eight-bend curves come out completely different each time, swinging wildly at the edges. That instability is variance. The model is so flexible that it copies whichever ten students it happened to see.
Good models sit in between: bendy enough to follow the pattern, stiff enough to ignore the noise.
Chapter 3 · The algorithm
You never know the real pattern. You only have your data. So you fake “new students” by hiding some of your own.
- Split. Put 20 to 30% of your rows aside. Do not look at them while building.
- Fit. Try a few model sizes on the rest: a line, one bend, two bends.
- Score. Measure each one on the rows you put aside.
- Pick and refit. The lowest score on the hidden rows wins. Refit that size on all your data and use it.
Little data? Use cross-validation, Stone’s 1974 idea. Cut the data into five slices. Fit on four, test on the fifth, and repeat until every slice has been the test once. Average the five scores. Every row gets to be “new” exactly once.
Read the result
- Bad on old data, bad on new data: underfit. Add bends, better columns, or use a more flexible model.
- Good on old data, bad on new data: overfit. Use fewer bends, get more data, add a fine on big numbers (ridge), or stop training earlier.
- Good on both, and close to each other: good fit. Ship it.
“Stop training earlier” links back to #03. A model learning by gradient descent usually gets better on new data for a while, then worse. Watch the hidden-data error at each step and stop when it turns up.
Use it today
Put the ten old students in columns A and B, and the five new ones in D and E. TREND fits a line on the old class and predicts the new one. SUMXMY2 adds up the squared misses, and SQRT(... / 5) turns that into average marks off.
The second formula adds ^{1,2} to give the curve one bend, exactly as in #04. Straight line: 8.3 marks off on new students. One bend: 1.8. The third formula scores the one-bend curve on its own class: 3.8.
It works as is in Excel 365. In Google Sheets, wrap the two ^{1,2} formulas in ARRAYFORMULA( ), or you get #REF!. I checked all three numbers in a live Sheet.
When the check itself lies
The hold-out check is honest only if the hidden rows stay hidden.
Three ways it goes wrong:
- Peeking. If you try fifty models and keep the one that scores best on the hidden rows, you have slowly fitted to those rows too. For anything serious, keep a third slice you touch only once, at the very end.
- Time. For sales, prices or anything that changes over time, hide the latest rows, not random ones. Otherwise the model is “predicting” Tuesday from Wednesday.
- Too few hidden rows. Five students is enough to show the idea, not to bet a business on. The fewer rows you hide, the more luck there is in the score.
The takeaway
Ockham said keep it simple. Larson showed why: a rule always looks better on the people it was built on. The test that tells learning from memorising is old and plain. Hide some data, build without it, then see how you do on it. If the model shines on what it has seen and stumbles on what it has not, it memorised.
Next in the series: decision trees, the model that asks yes-or-no questions, and why it overfits so eagerly.
