Logistic Regression: Why Is It Called Logistic, and What Is the Math Behind It?
When we hear the word logistics, we usually think about planning, organizing, and managing the movement of goods or resources. But what does logistic mean in Logistic Regression? Interestingly, the meaning is quite different in machine learning. Here, logistic refers to a mathematical function that transforms any real number into a value between 0 and 1, making it useful for predicting probabilities. Just as logistics helps organize complex operations into manageable processes, the logistic function transforms a linear mathematical score into a meaningful probability. Although this is only an analogy, it provides an intuitive way to remember the concept.
Logistic Regression is one of the most widely used machine learning algorithms for classification. Despite having the word regression in its name, it is primarily used to predict categories rather than continuous numerical values.
But why do we need a logistic function for classification? And how does it work mathematically?
Let’s explore the intuition, mathematics, and practical implementation of Logistic Regression.
Why Is It Called Logistic Regression?
The term logistic comes from the logistic function, also known as the sigmoid function.
In Linear Regression, we predict a continuous value using a linear equation:
The problem is that the output of this equation can be any number, from negative infinity to positive infinity.
For classification, however, we often want to estimate the probability of an event occurring, which must lie between 0 and 1.
This is where the logistic function comes in.
The Sigmoid Function
The sigmoid function transforms any real number into a value between 0 and 1.
Where e is Euler’s number, approximately 2.718. Remember Euler’s number? ( flashback 11th grade)
For example,
Notice that:
- Large positive values approach 1.
- Large negative values approach 0.
- When (z=0), the output is exactly 0.5.
The sigmoid function creates an S-shaped curve, making it useful for modeling probabilities in binary classification.
This is why the algorithm is called Logistic Regression: it uses a linear model together with the logistic function to estimate class probabilities.
2. How Does Logistic Regression Make Predictions?
Suppose we want to predict whether a student will pass an exam based on the number of hours studied.
And the target variable is:
First, Logistic Regression calculates a linear score:
Next, it applies the sigmoid function:
Suppose the learned model has:
For a student who studies for 5 hours:
Applying sigmoid:
The model estimates an 88.1% probability of passing.
Using a classification threshold of 0.5, the predicted class is Pass (1).
The threshold can also be adjusted depending on the problem. For example, medical screening systems may use a lower threshold when missing a positive case is particularly costly.
3. The Mathematics Behind Logistic Regression
Understanding Odds and Log-Odds
The mathematical foundation of Logistic Regression becomes clearer when we understand odds.
If the probability of an event is p, its odds are:
For example, if the probability of passing is 0.8:
This means the odds of passing are 4 to 1.
Taking the natural logarithm gives us the log-odds, also called the logit:
Logistic Regression assumes that the log-odds are a linear function of the input features:
This is an important distinction: Logistic Regression does not assume that probability itself is linear. Instead, it models the log-odds as a linear function.
Rearranging this equation gives:
Solving for p:
Which is equivalent to:
And we arrive back at the sigmoid function!
This also explains the “regression” part of the name: the model performs a linear regression-like operation on the log-odds.
Understanding the Weights
Each weight tells us how a feature influences the predicted outcome.
For a one-unit increase in xj, while holding other features constant, the log-odds change by wj.
Equivalently, the odds are multiplied by:
For example, if a feature has a weight of 0.7:
A one-unit increase in that feature multiplies the odds by approximately 2.01, holding other features constant.
This makes Logistic Regression relatively interpretable compared with many more complex machine learning models.
4. How Does Logistic Regression Learn? The Cost Function
In Linear Regression, we often use Mean Squared Error to measure prediction errors.
Logistic Regression typically uses a different loss function called Binary Cross-Entropy, or Log Loss.
Get Swati Kaushik’s stories in your inbox
Join Medium for free to get updates from this writer.
For one training example:
Where:
- y is the actual label, either 0 or 1.
- p^ is the predicted probability of class 1.
If the actual label is 1, the loss becomes:
If the actual label is 0:
The loss becomes large when the model is confidently wrong.
For m training examples, the average loss is:
The goal of training is to find the weights and bias that minimize this loss.
Gradient Descent
One common optimization technique is Gradient Descent.
The weights are updated iteratively:
Where 𝜶 is the learning rate.
For binary Logistic Regression, the gradients are:
During training, the optimizer adjusts the parameters to reduce the difference between predicted probabilities and actual labels.
In practice, libraries such as scikit-learn provide efficient numerical optimization algorithms, so we rarely need to implement the optimization process manually.
5. From Binary to Multiclass Logistic Regression
So far, we have discussed binary classification, where the target has only two possible classes.
But what if we have more than two?
For example, suppose we want to classify flowers into three species: Setosa, Versicolor, and Virginica.
This is a multiclass classification problem.
One common approach is Multinomial Logistic Regression, which uses the Softmax function.
The Softmax Function
For each class k, the model calculates a score:
The Softmax function converts these scores into probabilities:
Where K is the number of classes.
Suppose the model calculates three scores:
Applying Softmax gives approximately:
The probabilities sum to 100%, and the model predicts Class A because it has the highest probability.
The predicted class is:
For multiclass classification, the model commonly minimizes categorical cross-entropy:
Here, yik is 1 if example i belongs to class k and 0 otherwise.
Another approach is One-vs-Rest (OvR), which trains a separate binary classifier for each class. Multinomial Logistic Regression, however, learns the classes jointly through Softmax.
6. Why Use Logistic Regression?
Despite being relatively simple, Logistic Regression remains an important baseline algorithm in machine learning.
Advantages
- Easy to understand and implement.
- Computationally efficient.
- Provides probability estimates.
- Model coefficients are interpretable.
- Supports both binary and multiclass classification.
- Often performs well when classes are approximately linearly separable in the feature space.
Limitations
- Learns linear decision boundaries in the original feature space.
- May struggle with complex nonlinear relationships without feature engineering.
- Can be sensitive to highly correlated features.
- Benefits from appropriate feature scaling, especially when regularization is used.
- May require class weighting or other techniques for highly imbalanced datasets.
Although Logistic Regression is simpler than neural networks and ensemble methods, it is often a strong starting point for classification tasks.
7. Practical Example: Multiclass Classification Using Logistic Regression
Now that we understand the mathematics behind Logistic Regression, let’s see how it works in practice.
In this example, I implemented a multiclass Logistic Regression model using Python and scikit-learn to classify wines into three different categories based on their chemical properties.
Dataset
I used the built-in Wine dataset from scikit-learn, which contains 178 wine samples belonging to three classes: class_0, class_1, and class_2.
Each sample has 13 numerical features, including alcohol content, malic acid, magnesium, flavanoids, color intensity, hue, and proline.
Implementation Steps
- Load and explore the dataset: Loaded the Wine dataset using
load_wine()and examined its features, target classes, and descriptive statistics using Pandas. - Check missing values: Used
isnull().sum()to check for missing data. All 13 features had zero missing values. - Split the dataset: Divided the data into 80% training and 20% testing using
train_test_split(), resulting in 142 training samples and 36 testing samples. - Train the model: Created a Logistic Regression classifier using
LogisticRegression()and trained it withmodel.fit(). - Evaluate predictions: Tested the trained model on unseen test data and generated a classification report containing precision, recall, and F1-score for each class.
- Visualize performance: Created a bar chart comparing precision, recall, and F1-score across the three wine classes.
Model Results
The Logistic Regression model achieved an accuracy of 91.67% on the test dataset, correctly classifying 33 out of 36 wine samples.
In addition to accuracy, I used precision, recall, and F1-score to evaluate classification performance for individual classes. These metrics provide a more detailed understanding of how well the model distinguishes between different wine categories.
An Important Observation
During training, the model generated a convergence warning indicating that the optimizer reached its maximum number of iterations before convergence.
Since the Wine dataset contains features with different numerical scales, applying StandardScaler and increasing max_iter could help the optimizer converge more effectively and potentially improve model performance.
This example demonstrates how Logistic Regression can be extended beyond binary classification to predict multiple categories using real-world numerical features.
Complete implementation: Logistic Regression — Multiclass Classification Example (GitHub)
8. Conclusion
Logistic Regression is a great example of how a relatively simple mathematical idea can become a powerful machine learning algorithm.
The key concept is that Logistic Regression models the log-odds of an outcome as a linear combination of input features. The sigmoid function then converts the result into a probability between 0 and 1.
For problems involving more than two classes, the Softmax function extends this idea to multiclass classification.
Understanding the mathematics behind Logistic Regression also builds a strong foundation for learning more advanced concepts, including neural networks, classification loss functions, and optimization algorithms.
Final takeaway: Logistic Regression may have “regression” in its name, but its real strength lies in classification. And the word “logistic” comes from the mathematical function that makes probability-based predictions possible.
