Statistical Machine Learning: Episode 1 — An Introduction to Statistical Learning
What is Statistical Learning?
Statistical Learning is a field concerned with learning from data. It provides a framework for analyzing data to uncover hidden patterns and relationships, enabling inference and the construction of models for predicting future observations.
Statistical machine learning is a subfield of machine learning that builds models based on probabilistic and statistical principles.
To understand Statistical Learning, consider the following example:
Suppose our task is to improve the sales of a particular product. We are given a dataset consisting of 200 observations of product sales across different markets, along with the advertising budgets in each market for three media: TV, radio, and newspaper.
The figures below illustrate the relationship between sales and each advertising medium’s budget. Each blue line represents a simple model that can be used to predict sales based on the different marketing media.
By analyzing the data using statistical principles, we can uncover the relationship between sales and advertising budgets. This enables better budget planning for advertising, which can help increase sales.
Variables and Terminology
Predictors and Responses
The advertisement budgets in the Example above are called input variables, and the sales are called the output variable.
In most textbooks, the input variables are denoted by X, and the output variable is denoted by Y. In the Example above, X_1 represents the TV budget, X_2 represents the radio budget, and X_3 represents the newspaper budget.
Input variables can be referred to by different names depending on the context. In statistical learning, X is referred to as predictors (variables used to predict Y) or features, while Y is referred to as the response.
Quantitative and Qualitative Variables
Variables can generally be divided into two categories:
- Quantitative variables take numerical values, such as a person’s age, weight, or income.
- Qualitative variables, also referred to as categorical variables, take non-numerical values that represent categories, such as a person’s sex (female or male).
In the Example we covered earlier, the output variable sales is a quantitative variable.
Even though qualitative variables take non-numerical values, they are often encoded using numerical values. For example, if the response variable belongs to only two classes, “successful” or “unsuccessful,” these classes can be encoded using a single binary digit, such as 1 and −1, respectively.
If the response variable is quantitative, the task is called regression, since the goal is to predict a numerical value. If the response variable is qualitative (categorical), the task is called classification, since the goal is to predict the category or class to which an observation belongs.
Why Do We Estimate f?
Suppose we observed a quantitative response Y and p different predictors:
We assume that there is some relationship between X and Y
We assume that f is some fixed unknown function of X, and ε is some random error that is independent of X and has a mean of zero
If we have N observations, then the ith observed value of X is denoted as x_i
A dataset is often denoted as D
→ Our goal is to estimate f.
There are two reasons why we estimate f: prediction and inference
prediction
Given a training dataset, we want to build a model that can learn the relationship between the input variables and the response variable, and then use that model to predict the response for future observations.
Because of the presence of the error term ε, it is almost impossible to determine the exact form of f. Therefore, the objective is to estimate f as closely as possible so that the resulting model performs well in predictions.
The quantities \hat{f} and \hat{Y} represent the estimated values of f and Y, respectively.
To evaluate the performance of a model, we need to measure to what extent the prediction matches the observed response
We want to find an estimate of f that minimizes the prediction error.
Given an estimated f and a set of observations X, we assume both are fixed. So let a be:
Then the expected value (or the average) of the square difference between the predicted and actual value of Y is given as
We got the variance from
The accuracy of a prediction depends on two sources of error: Reducible error and Irreducible error
- Reducible error, which arises because our estimate of f is not exactly equal to the true function.
- Irreducible error, which arises from the random error term ε. Since this randomness is inherent in the data-generating process, it cannot be eliminated, regardless of how well we estimate f.
The goal of prediction is to estimate f in a way that minimizes the reducible error. This can be achieved by selecting a more appropriate model, improving the estimation procedure, or using more training data.
Get Shaima’s stories in your inbox
Join Medium for free to get updates from this writer.
The error term ε may contain unmeasured variables that affect the response, measurement errors, or other sources of random noise. In the statistical learning framework, ε is assumed to be independent of X. Therefore, regardless of how accurately we estimate f, we cannot eliminate the variability introduced by ε. This variability is known as the irreducible error.
However, we can view ε as placing an upper limit on the accuracy of our predictions for Y → it determines the smallest possible prediction error.
Inference
Given a dataset, we may be interested in understanding how the response variable Y changes as the predictors X change. In this setting, the goal is to understand the relationship between the predictors and the response.
Inference seeks to answer questions such as:
- Which predictors are associated with the response? Not all predictors necessarily affect the response variable.
- What is the nature of the relationship between the response and each predictor? For example, does an increase in the predictor lead to an increase in the response (a positive relationship), or does it lead to a decrease in the response (a negative relationship)? We may also be interested in the magnitude of this effect.
- Can the relationship between Y and the predictors be adequately summarized by a linear model, or is it more complex and better described by a non-linear model?
How Do We Estimate f?
There are two methods to estimate f: parametric and non-parametric
Parametric Methods
Parametric methods consist of two steps:
Step 1: Make an assumption about the form or shape of f.
For example, we may assume that f is a linear function of X.
By assuming that f is linear, the problem of estimating the unknown function f is reduced to estimating the p+1 coefficients β0,β1,…,βp.
Step 2: Train the model to fit the data.
Once the form of f has been selected, the model is trained using the available data. In the case of a linear model, this involves finding the coefficients β0,β1,…, βp that produce the best fit to the data and yield the most accurate predictions.
The model is typically fitted by minimizing a loss function, which measures how well the model’s predictions match the observed responses.
Non-Parametric Methods
Unlike parametric methods, non-parametric methods do not make strong assumptions about the form or shape of f.
Instead, the data is used to estimate the form of f directly. There are various non-parametric techniques for estimating f, including K-Nearest Neighbors (KNN), Decision Trees, Random Forests, Smoothing Splines, and Kernel Regression
The Flexibility Trade-Off
Even though parametric methods simplify the estimation problem and, in some cases, the assumed model performs well, they are inherently restrictive. If the true relationship is substantially different from the assumed form, the model may perform poorly because it cannot adequately capture the underlying pattern in the data.
Non-parametric methods, on the other hand, are generally more flexible and can capture complex relationships between the predictors and the response. However, unlike parametric methods, the problem is not reduced to estimating a small number of parameters. As a result, non-parametric methods typically require a much larger number of observations to obtain an accurate estimate of f.
In addition, being too flexible can lead to overfitting, while being too restrictive can lead to underfitting.
Categories of Statistical Learning
Statistical learning can be categorized into two categories:
- Supervised learning
- Unsupervised learning:
Supervised learning
For each observation x_i, there is an associated response y_i.
The goal is to fit a model that describes the relationship between the predictors and the response.
In this case, learning is “supervised” because we are given the response variable Y, which guides the learning process.
Unsupervised learning
We are given observations x_i, but there are no associated response values. The goal is to understand the structure or relationships within the data, such as relationships between observations or among predictors.
An example of unsupervised learning is clustering, where the objective is to determine whether the observations can be grouped into distinct clusters or groups based on similarity.
The Bias–Variance Trade-Off
There are two important concepts that we need to understand: variance and bias.
Variance
Variance refers to the amount by which the estimated function f would change if it were trained on a different training dataset.
Since different training datasets produce different estimates of f, some variation is expected. However, ideally, the estimated function should not vary too much from one training dataset to another.
A method with high variance is very sensitive to the training data. As a result, small changes in the training dataset can lead to large changes in the estimated function.
Models with high variance are typically too flexible, allowing them to fit not only the underlying relationship but also random noise in the training data. Consequently, high variance often leads to overfitting.
Bias
Bias refers to the error that results from approximating f as a model that is too simple to capture the complex relationships within the data.
A method with high bias tends to make strong assumptions about the form of f. Hence, it is too restrictive, preventing the learning of the true relationship between the predictors and the response. Consequently, high bias often leads to underfitting.
The goal is to find a balance between bias and variance so that the model generalizes well to new, unseen data. This balance is known as the bias–variance trade-off.
