Understanding the Bias–Variance Tradeoff in Machine Learning
One of the most common questions beginners face in Machine Learning is:
Why does a model perform extremely well on training data but poorly on unseen data?
The answer often lies in two important concepts: Bias and Variance.
Understanding these concepts helps us identify whether a model is underfitting, overfitting, or generalizing well.
What is Bias?
Bias is the error introduced when a model makes overly simplistic assumptions about the underlying data.
In simple terms, bias measures how far a model’s average predictions are from the actual values.
A model with high bias struggles to learn the true patterns present in the dataset and often oversimplifies the problem.
Low Bias
- Learns important patterns from data.
- Makes fewer incorrect assumptions.
- Captures relationships more accurately.
High Bias
- Makes strong simplifying assumptions.
- Misses important patterns.
- Often leads to underfitting.
For example, using a straight line to model a highly non-linear relationship will likely result in high bias.
What is Variance?
Variance measures how sensitive a model is to the specific training data it receives.
A high-variance model changes significantly when trained on different subsets of data, while a low-variance model remains relatively stable.
Low Variance
- Produces consistent predictions.
- Less sensitive to training data changes.
- Generalizes more reliably.
High Variance
- Learns training data too closely.
- Captures noise along with patterns.
- Performs poorly on unseen data.
High variance is typically associated with overfitting.
Underfitting vs Overfitting
Before understanding the tradeoff, let’s connect bias and variance to two common machine learning problems.
Underfitting
Underfitting occurs when a model is too simple to capture the underlying patterns in the data.
Characteristics:
- High Bias
- Low Variance
- Poor training performance
- Poor testing performance
Example:
A linear model trying to fit a complex non-linear relationship.
Overfitting
Overfitting occurs when a model learns not only the true patterns but also the noise present in the training dataset.
Characteristics:
- Low Bias
- High Variance
- Excellent training performance
- Poor testing performance
Example:
A very deep decision tree that memorizes the training dataset.
The Bias–Variance Tradeoff
At this point, we know:
- High Bias → Underfitting
- High Variance → Overfitting
Naturally, we would like to reduce both.
Get Prateek Agrahari’s stories in your inbox
Join Medium for free to get updates from this writer.
Unfortunately, reducing one often increases the other.
As model complexity increases:
- Bias generally decreases.
- Variance generally increases.
As model complexity decreases:
- Variance generally decreases.
- Bias generally increases.
This relationship is known as the Bias–Variance Tradeoff.
The goal is not to minimize bias or variance individually.
The goal is to minimize the overall prediction error on unseen data.
Four Possible Scenarios
1. High Bias + High Variance
This is the worst-case scenario.
Characteristics:
- Poor training performance
- Poor testing performance
- Model fails to learn meaningful patterns
- Predictions are unstable
2. High Variance + Low Bias
Characteristics:
- Model is overly complex
- Excellent training accuracy
- Poor testing accuracy
- Overfitting
3. Low Variance + High Bias
Characteristics:
- Model is overly simple
- Stable predictions
- Misses important relationships
- Underfitting
4. Low Variance + Low Bias
Characteristics:
- Learns useful patterns
- Produces stable predictions
- Generalizes well to unseen data
This is the ideal situation.
Visualizing the Tradeoff
Colab Notebook : Link
Key Takeaways
- Bias measures errors caused by overly simplistic assumptions.
- Variance measures sensitivity to training data.
- High Bias leads to underfitting.
- High Variance leads to overfitting.
- Increasing model complexity generally decreases bias but increases variance.
- The objective is not minimizing bias or variance independently, but it is achieving the best generalization performance on unseen data.
- Understanding the Bias–Variance Tradeoff is fundamental because nearly every machine learning model can be viewed as a search for the right balance between simplicity and complexity.
