Knowing When You Know Enough
What learning the ukulele taught me about forecasting, overfitting, and knowing when to move on.
A few months ago, I decided to learn to play the ukulele.
I imagined myself sitting on the patio on a warm summer evening, effortlessly strumming familiar songs. Reality turned out to be a little different.
One of the first chords I learned was C. It was almost disappointingly easy — one finger on one string. Within a few minutes I could make it ring clearly.
Then came G7.
Three fingers.
Three strings.
Three opportunities to get something wrong.
What surprised me wasn’t that G7 was harder. It was that I could switch easily from G7 back to C, but going the other direction — from C to G7 — felt like trying to solve a tiny puzzle every single time. My fingers knew where they wanted to go individually, but they refused to arrive there together.
I practiced.
I adjusted my hand position.
I even wondered whether I was holding the ukulele incorrectly.
Some evenings I spent fifteen minutes working on nothing but that one transition.
The strange part was that the harder I tried to make G7 perfect, the less progress I seemed to make.
Then one evening I stopped chasing perfection.
Not because I had mastered G7.
I hadn’t.
But I had learned it well enough.
So I turned the page in my lesson book.
I learned another chord.
Then another.
Before long I wasn’t just practicing isolated chords anymore.
I was playing songs.
Something unexpected happened.
Learning became fun again.
Looking back, I realized my mistake wasn’t practicing too little.
It was confusing perfecting a chord with learning an instrument.
That realization stayed with me because I’ve begun noticing the same pattern in places that have nothing to do with ukuleles.
One of my favorite books is Algorithms to Live By, by Brian Christian and Tom Griffiths.
In one chapter, the authors describe a study of life satisfaction during the first ten years of marriage. Researchers in Germany collected years of survey data from newly married couples and asked a deceptively simple question:
Given these observations, what mathematical model best predicts what will happen next?
At first glance, the answer seemed obvious.
Draw the model that best fits the data.
The original observations looked like this.
Figure 1. Life satisfaction during the first ten years of marriage.
A simple one-factor model captured the overall trend, but it clearly missed some of the rises and falls.
Adding a second factor produced a much better fit.
Then the researchers built a much more sophisticated model using nine different factors.
At first glance, it looked brilliant.
It hugged almost every point on the graph.
Every bump.
Every dip.
Every little wiggle.
Get Steven Baru’s stories in your inbox
Join Medium for free to get updates from this writer.
If your goal was simply to explain the past, you probably would have declared it the winner.
But that’s not why we build models.
We build them to predict the future.
When the researchers extended the models beyond the ten years of observed data, something remarkable happened.
Figure 2. Comparing one-, two-, and nine-factor models.
The elaborate nine-factor model predicted that marital satisfaction would eventually collapse, plunging toward zero after about ten years.
The much simpler two-factor model predicted something quite different. Happiness still rose and fell — as life inevitably does — but gradually returned toward a person’s normal baseline. That prediction fit much better with what psychologists and economists already understood about how people adapt to life’s major events.
The simpler model made the better forecast.
The extra seven variables hadn’t uncovered a deeper truth.
They had simply mistaken noise for signal.
That idea stopped me in my tracks.
Like many people, I had always assumed that more information naturally produced better predictions.
Sometimes it does.
Sometimes it simply makes us more confident.
Forecasts are often misunderstood for the same reason.
Before the 2016 U.S. presidential election, many forecasting models gave Donald Trump roughly a 30 percent chance of winning. When he won, many people concluded that the forecasts had failed.
But a 30 percent chance isn’t zero.
If tomorrow’s weather forecast says there’s a 30 percent chance of rain and it rains, we don’t accuse the meteorologist of making a bad forecast.
We simply happened to live in the 30 percent world.
Good forecasters don’t promise certainty.
They estimate probabilities.
The more I read about forecasting, the more I realize it isn’t really about mathematics.
It’s about humility.
It’s about recognizing that the world is more complicated than we’d like it to be.
It’s about understanding that every additional variable doesn’t necessarily bring us closer to the truth.
Sometimes it carries us farther away.
Learning the ukulele has quietly taught me the same lesson.
When I spent all my time trying to perfect G7, I wasn’t really making progress.
I was polishing one small piece of a much larger puzzle.
The goal wasn’t to master one chord.
The goal was to learn how chords connect together into songs.
Forecasting isn’t much different.
The goal isn’t to explain every wiggle in the past.
The goal is to understand the pattern that is most likely to continue into the future.
There will always be another variable to add.
Another statistic to collect.
Another chord to practice.
The difficult part isn’t deciding whether more information exists.
It always does.
The difficult part is recognizing the moment when more information stops adding understanding and starts adding noise.
I’m still learning the ukulele.
My transitions still aren’t smooth.
My fingers still miss the occasional string.
But these days, I think less about perfecting the next chord.
I think more about learning the next song.
