Bias and Variance, Part 1: Your machine learning model can go wrong in Two Different Ways
Part 1 of a two-part series on bias and variance for data science interviews.
Every model you ship is wrong. The useful question is how it’s wrong, because there are exactly two ways, they have opposite cures, and applying the wrong cure makes things worse.
1. Two ways to be wrong
Imagine two credit risk teams. Both built a probability-of-default model. Both report an AUC of 0.71 on the test set. Identical performance, identical dashboards.
Team A’s model is a simple logistic regression on five variables. It’s stable; refit it on last quarter’s data instead of this quarter’s and the coefficients barely move. But it consistently underestimates risk for self-employed applicants, every single time, on every dataset. It is reliably, predictably wrong in the same direction.
Team B’s model is a deep gradient boosting ensemble. It handles self-employed applicants beautifully. But refit it on a different sample of the same portfolio and the predictions for individual borrowers swing wildly. One applicant scores 4% default probability in Monday’s model and 19% in Tuesday’s. It is unreliably wrong in unpredictable directions.
Team A has a bias problem. Team B has a variance problem. The fixes are opposites: Team A needs a more flexible model, Team B needs a less flexible one. If either team applies the other’s remedy, their model gets worse.
Interview questions from this section
“What is the bias-variance tradeoff?” is the balance between a model being too simple (high bias) and too sensitive to the training data (high variance). Usually, reducing one can increase the other, so the goal is to find a model that balances both and generalizes well to unseen data.
2. The Risks of a Perfect Memory: Variance as the “Memorization” Penalty
Variance is how much your fitted function would change if you had been handed a different training set.
You have one dataset. But imagine you could reach into the population and draw a hundred different training samples of the same size. You fit your model a hundred times. You now have a hundred slightly different functions. Variance measures how spread out those hundred functions are.
Picture the same set of dots on a chart, with two different lines drawn through them. The first is a flexible curve that bends up and down so it passes close to nearly every dot. The second is one straight line.
Now move a single dot upward. The flexible curve changes shape to follow it; it bends toward the new position, and the curve looks different afterwards. The straight line just tilts a little. Nothing else about it changes.
The flexible curve has high variance. The straight line has low variance. Same dots, same small change, completely different sensitivity.
Rule of thumb: the more flexible the method, the higher the variance.
High-complexity models tend to overfit, meaning they learn the “noise” in the training data rather than the actual patterns. There is a profound irony here: “perfect” training scores often signal a model that is no longer applicable to different datasets. Instead of learning the true trends, the model has simply memorized the noise.
Real-world version: A retail bank’s churn model was retrained weekly on a rolling 12-month window. The model was a deep random forest with unlimited depth. Every week, the marketing team received a list of “high-risk churners” to target with retention offers. Every week, roughly 40% of the list was different and not because customer behaviour had changed. The same customers cycled on and off the list depending on which week’s noise the model happened to latch onto. Marketing lost trust in the model long before anyone diagnosed the variance.
Interview questions from this section
“Is variance a property of the model or of the data?” Both, and saying so is the strong answer. Model flexibility sets the sensitivity; sample size and noise level set how much there is to be sensitive to. The same tree is high-variance on 500 rows and much less so on 5 million.
3. The Cost of Simplicity: Bias as a “Lack of Imagination”
Bias is the systematic error introduced by the assumptions your model makes about the data.
Return to those hundred training sets. Fit your model on each one, then average the hundred predictions for a given input. Bias is the gap between that average prediction and the truth.
The averaging is the part most explanations skip, and it’s what makes bias different from ordinary error. Bias is what remains after you’ve averaged away all the randomness. It’s the error the model would still make if you gave it infinite data; because it comes from the model’s structure, not from sampling luck.
Linear regression is the textbook case. It assumes the relationship between features and target is a straight line. If the truth is a curve, no amount of data fixes this. Give linear regression ten billion rows and it will still fit a line through a curve, still overshoot in the middle, still undershoot at the ends. Consistently. Forever.
Real-world version: Income-to-default relationships are rarely linear. Default risk falls steeply as income rises from very low levels, then flattens out; above a certain income, extra income barely reduces risk. A linear model forced through this shape will systematically overestimate risk for high earners and underestimate it for the working poor. Not sometimes. Structurally. This is why credit scorecards bin continuous variables into ranges and apply weight-of-evidence transformations; the binning is a deliberate purchase of flexibility to buy down bias, while keeping the model simple enough to stay low-variance and auditable.
Interview questions from this section
“What is bias in machine learning?” Systematic error from the model’s structural assumptions; the error that remains after averaging over all possible training sets.
“Does collecting more data reduce bias?” No, and this is a trap question. More data reduces variance. Bias comes from the model’s functional form, so a linear model fit on infinite data still fits a line through a curve. Candidates who say yes have memorized “more data is better” without understanding the decomposition. More/better features: changes the model class itself. That’s a capacity increase, so bias falls and variance rises.
4. Where models sit and how to measure it
Every model makes a trade between assuming and listening. Here is the whole landscape in one table.
Two things drive the whole table.
Get narmhabil’s stories in your inbox
Join Medium for free to get updates from this writer.
Low-bias models assume little, so they can express fine detail. A decision tree doesn’t presume the relationship is linear, or polynomial, or anything in particular. Give it a portfolio and it will find that default risk spikes for applicants who are under 25 and recently relocated and self-employed, a three-way pocket a linear model can only capture if someone thought to hand-engineer the interaction term.
That same ability is why they overfit. In the same portfolio, the tree will also find that every applicant who applied on a Tuesday defaulted. Both patterns are entirely true of your training rows. The difference is that the first would reappear in a fresh sample and the second wouldn’t; it’s a coincidence involving a handful of applicants who were risky for other reasons. But nothing inside the data says so. There is no column marking which patterns are real, and training rewards learning both equally, because both reduce training error. Only fresh data separates them, and the model never sees fresh data while it is training. Flexibility is the freedom to be misled.
High-bias models are harder to mislead, for exactly the reason they underfit. Logistic regression cannot express “Tuesday applicants specifically,” so it never falls for the coincidence, but it cannot express the three-way pocket either. That rigidity protects the model from chasing random patterns, but it also prevents it from learning real patterns that are more complex. That is the trade, and more rows will not change it.
Real-world version: Property valuation. In most cities the price per square metre falls as properties get bigger: a 30 m² studio might go for €1,800/m² while a 300 m² house goes for €1,275/m², because a kitchen and a bathroom cost roughly the same whether the flat around them is small or large. A straight line cannot represent that. It has to commit to one number for the extra square metre, and it lands on about €1,280: too cheap for small flats, too expensive for large ones. The result is a signature you can predict before you fit it: the model overvalues both ends of the market and undervalues the middle. On simulated data with that shape, a linear model comes out +9% on the studio, −1% at 75 m², and +6% on the 300 m² house. Feed it a million more sales and those percentages barely move, because the problem is the straight line, not the sample size. That is bias.
Now the other failure. A deep tree handles the curve without being told about it; but it also notices that the three Kirov Street properties in your training set each sold about 15% above what their size and condition would predict. There was a reason: someone was buying up that street to assemble a plot, and overpaid on all three. The tree doesn’t know that. It creates a leaf for street = Kirov and stores "+15%". The developer finishes assembling; a perfectly ordinary Kirov Street flat comes to market next spring; your model still adds 15% to it. If that valuation backs a mortgage, you have just lent against value that isn't there. And the tell is exactly the bootstrap check above: refit on a resample that happens to exclude those three sales and the +15% rule disappears entirely. It was never a fact about the street, it was a fact about three transactions. That is variance.
Interview questions from this section
“Why do flexible models overfit?” The single best question in this topic, and most candidates answer it circularly (“because they fit the noise”). Give the mechanism: signal and noise both reduce training error, they are indistinguishable from inside the training set, and the model has no information that would let it prefer one. Flexibility is the freedom to be misled.
“Is a low-bias model always better?” No. Low bias is bought with variance. On small samples, a high-bias model frequently wins on test error despite being structurally wrong, because there isn’t enough data to estimate a flexible model stably.
“Your model performs well on aggregate metrics but stakeholders complain about individual predictions. What’s happening?” High variance at the row level being masked by averaging. This is a favourite among interviewers at insurers, banks, and anywhere predictions are consumed one at a time. The answer is the bootstrap check plus regularization or ensembling.
“Name a situation where you’d deliberately choose the higher-bias model.” Small sample, regulatory scrutiny, need for stable coefficients over time, or a requirement to explain individual decisions to a customer.
5. The archery quadrants
Four arrows, one target. This is the standard picture and it earns its place.
High bias, low variance. Your arrows land in a tight cluster, well away from the bullseye. The bow’s sights are misaligned. You are consistent and consistently wrong. In practice: a logistic regression that will not fit a non-linear relationship, no matter how much data you give it. Reliable, auditable, and structurally incorrect.
Low bias, high variance. Your arrows scatter across the whole target. Some are dead centre. You know where to aim, but a crosswind takes each shot somewhere different. In practice: an unpruned decision tree. It has the capacity to be right and no discipline about it.
High bias, high variance. Arrows scattered and centred on the wrong spot. Worst of both. In practice: a high-capacity model trained on a badly specified feature set, or a complex model fit on far too few rows. Complexity does not automatically buy you low bias.
Low bias, low variance. Tight cluster on the bullseye. This is the goal, it is bounded by the irreducible error in section 7, and you reach it through the tradeoff rather than by ignoring it.
Interview questions from this section
“Can a model have both high bias and high variance? Give an example.” Yes, and many candidates wrongly say no, assuming the tradeoff means one must be low. A high-capacity model trained on too few rows, or on a badly specified feature set, is both scattered and centred on the wrong answer. Complexity buys low bias only if the added flexibility points in a useful direction.
Quick recap of Part 1
- There are two ways to be wrong: systematically (bias) and erratically (variance). They have opposite cures.
- Variance is how much your fitted function moves if you swap in a different training set.
- Bias is the error left after averaging over all possible training sets.
- Flexibility is the single dial. Low-bias models assume little, so they can express fine detail, and that same ability is what lets them learn coincidences.
- Signal and noise both look like patterns that hold in your data. Nothing inside the training set separates them.
- Both can be high at once: a complex model on too few rows, or on bad features.
Quick check before Part 2
If you can’t answer one of these in two sentences, reread the section it came from.
- Define bias. Define variance.
- Does adding more rows reduce bias? Does adding more features?
- Why can’t a flexible model tell signal from noise?
- Can a model have both high bias and high variance? Give an example.
Coming in Part 2: the cures for bias and variance.
