Why Bagging Reduces Variance — A Mathematical Intuition

Dr. Nitish Sagar

4 min read

·

Apr 10, 2026

--

Listen

Share

In machine learning, one of the most important as well as popular techniques is Ensemble Method. What I am writing today is not a new concept rather it is often repeated so frequently that we accept it without truly understanding it:

Bagging reduces Variance

The idea behind this blog is to provide a simple mathematical understanding behind this statement.

Before we begin diving into this, lets just take a minute for the sake of complete picture to understand Ensemble methods are so important in machine learning.

Background

In machine learning, it is a common problem of a model being overfit or underfit. Overfitting refers to the scenario when the model performs really well on training data but not so well in real test samples. On the other hand, in underfitting model has large error on training data itself due to flawed assumption. To give an example, if decision tree is trained which has large depth and many leaf nodes, it generally tends to overfit thus leads to high variance. Just to give another perspective, this happens in neural networks as well when there are multiple layers with many neurons. People use dropout method to avoid overfitting.

If the tree is too short, it tends to underfit and leads to large bias errors in prediction. This means having one model does not solve the problem. What researchers have found out that if you use multiple models instead of one, it can solve these problems to much extent. This led to the use of ensemble models such as bagging, boosting, stacking, etc. Here I will just talk about bagging.

Now let’s ask our first important question of the day:

If each model is noisy, why does averaging them make things better?

Answer lies with the technique of Bagging.

Bagging reduces Variance

In bagging approach, multiple models are trained but each model learns different set of features and different subset of training data. The idea behind is to have models with uncorrelated errors and thus taking average of the prediction could reduce the variance. It is like instead of having a judge, will have jury members with different thought process but reasonable.

Let’s assume a random variable Xi that denotes the prediction of a model.

In bagging we take the average of the predictions from different models. Let’s try to write in the form of equation using basics of statistics.

Variance of an average = sum of individual variance and how variable interact with each other

For the sake of simplicity, we will assume the variance of each model prediction be \sigma2, correlation between each models to be r and number of models trained to be n.

With this, we will get the first term in RHS of above equation to be

Similarly, covariance with n(n-1) terms will be:

With minor arrangement of terms, we get:

The first term in RHS refers to the variance due to correlation between model predictions and the second term is a reducible variance if the number of models are increased. Thus in bagging technique such random forest if the individual models are low correlation, the variance of the aggregate model will be lower than that of individual models.

• If r = 0 (uncorrelated models)

variance reduces as we add more models

• If r = 1 (fully correlated):

Thus, no improvement with ensemble approach.

In real scenarios, we will r ranging between 0 and 1. Lower the correlation value, more reduction in the overall variance.

Conclusion

Bagging reduces variance not by making individual models better, but by combining multiple imperfect models in a smart way.

From the mathematical derivation, we see that the variance of the averaged model depends on two key components: the number of models and the correlation between them. While increasing the number of models helps reduce variance, the real power of bagging lies in ensuring that these models are as uncorrelated as possible.

In the ideal case, where model errors are independent, variance decreases proportionally to 1/n. However, if models are highly correlated, the benefit of averaging diminishes significantly.

This explains why techniques like bootstrap sampling and random feature selection (as in Random Forests) are critical — they introduce diversity among models, reducing correlation and maximizing variance reduction.

Interestingly, the core idea behind bagging — reducing variance through aggregation of diverse predictions — continues to play an important role even in modern large-scale AI systems like Large Language Models (LLMs).

While LLMs such as GPT models are not explicitly trained using classical bagging, similar principles appear in practice. Techniques like mixture-of-experts architectures relies on generating diverse outputs and aggregating them to improve reliability and stability.

Averaging cancels out noise — but only when the noise is different across models.