Linear Regression
Cost Function, Gradient descent, Multivariate, Vectorization, Feature Scaling, Feature engineering & Polynomial Regression,
Intuition:
Imagine you’ve got a scatter plot, house sizes on the x-axis, prices on the y-axis. Dots all over. Linear regression is just: draw the straight line that best fits through those dots. Then to predict a new house’s price, you find its size on the line.
That’s basically it. The whole game is “what does best fit mean?”
Prediction: ŷ = w·x + b
(w = weight/slope, b = bias/intercept)
Cost Function — Mean Squared Error (MSE):
Cost function = how we measure “best fit.”
For each data point, look at the gap between the actual value and what your line predicts. That gap is the error (or residual).
Most common cost function for linear regression: Mean Squared Error (MSE).
- We square the errors (so positives and negatives don’t cancel, and big errors hurt more)
- We average across all m points
- The 1/2 is just a math convenience for when we take derivatives later
Lower J = better line. The whole optimization problem is: find w and b that make J smallest.
Convex guarantee One nice property of the MSE cost function for linear regression: it’s always a convex function. Think of it as a bowl shape — no local valleys, no traps. There is only one minimum, and it’s the global one. That means gradient descent will always find the bottom, no matter where you start. This is not true for all ML models (neural nets, for instance), so it’s worth appreciating here.
Question on my mind:
why do you think we square the errors instead of just taking absolute value?
1. Math friendliness — Squaring is smooth and differentiable everywhere. Absolute value has a sharp corner at zero, which makes gradient-based optimization messy. With squared error, you get clean derivatives → easy gradient descent.
2. Punishes big errors more — An error of 10 contributes 100 to the cost; an error of 2 contributes only 4. So the model really tries to avoid large mistakes, even if it means tolerating small ones. Absolute value treats all errors proportionally.
I like it!
Gradient descent:
Gradient descent = how we actually find the w and b that minimize J.
Analogy: Walking downhill blindfolded — take small steps in the direction of steepest descent until you reach the bottom.
The intuition: Imagine J plotted as a 3D bowl-shaped surface over w and b. You’re standing somewhere on this surface and want to reach the bottom. You look around, find the steepest downhill direction, take a small step. Repeat until you can’t go lower.
Gradients:
dw = (1 / m) * np.sum(error * x)
db = (1 / m) * np.sum(error)
# Update
w = w — alpha * dw
b = b — alpha * db
The update rule:
w := w − α · (∂J/∂w)
b := b − α · (∂J/∂b)
Learning rate behavior α (learning rate) controls your step size — and getting it right matters.
- Too small → convergence is painfully slow, you’ll be crawling toward the minimum
- Too large → you overshoot the minimum and the algorithm can diverge entirely
Here’s what’s elegant though: you don’t need to manually shrink α over time. As gradient descent approaches the minimum, the slope of J naturally flattens out. Smaller slope × same α = smaller step. The step size decreases automatically. The math does the work for you.
α (learning rate): controls step size.
• Too small → slow to converge
• Too large → overshoots, diverges
Batch GD: uses all m examples per step. Guaranteed to converge on convex (bowl-shaped) cost.
Monitoring convergence A good habit: plot J after every iteration. If gradient descent is working, J should decrease on every step and eventually flatten out. If J is going up or oscillating wildly, your α is too large — pull it back. Watching this curve is how you know whether you’re converging or diverging, before you waste hours of training time.
Key thing: you update w and b simultaneously using the old values — not one then the other. This is non-negotiable. If you update w first and then use the new w to compute db, you’ve broken the math. Both gradients must be computed from the same snapshot of (w, b), then both updated together.
Multiple Features & Vectorization
Intuition:
So far we’ve been predicting house prices using just one feature — size. But real houses have bedrooms, bathrooms, age, location, floors. The real world has n features, not one.
Multiple linear regression is just: more inputs, same idea. Instead of one slope w, you have a slope for every feature.
That’s it. The line becomes a hyperplane, but the intuition stays the same.
From Univariate to Multivariate:
Univariate (what we had):
ŷ = w·x + bMultivariate (what we have now):
ŷ = w₁x₁ + w₂x₂ + ... + wₙxₙ + bEach wⱼ is the model’s bet on how much feature xⱼ matters. Big w on “square footage” means size drives price. Small w on “floor number” means the model learned it barely matters.
But writing that sum out for thousands of features? That’s where things get ugly fast.
The Vectorized Form:
Instead of a long sum, collapse everything into vectors:
Without Vectorization ✗
Looping manually over every feature:
f = 0
for j in range(n):
f += w[j] * x[j]
f += bProblem: Python loops are slow. With thousands of features or training examples, this becomes a bottleneck. You’re asking Python to manage every single multiplication one at a time.
With Vectorization ✓
Use NumPy’s dot product:
f = np.dot(w, x) + bOne line. Same result. Orders of magnitude faster.
Why? NumPy calls BLAS routines under the hood — these run in parallel on CPU hardware, not sequentially like a Python loop. The CPU processes multiple operations simultaneously. Python loops can’t touch that.
Question on my mind:
Does vectorization change what the model learns, or just how fast it learns?
Get Akshitha Kukudala’s stories in your inbox
Join Medium for free to get updates from this writer.
Just how fast. The math is identical — np.dot(w, x) computes the exact same number as the loop. The difference is purely execution speed. Your gradients, your cost, your final weights — all the same. You're just not waiting around for Python to finish counting.
Rule of thumb:
In ML code, always prefer matrix and vector operations over loops. Vectorization is not optional at scale. The moment you write
for j in range(n)over features or training examples, ask yourself if NumPy can do it instead. It almost always can.
Feature Scaling
Intuition:
Say you’re predicting house prices with two features — number of bedrooms (range: 1–5) and square footage (range: 500–5000). Those two features live on wildly different scales.
Gradient descent sees the cost function as a surface and takes steps downhill. But when features are on different scales, that surface becomes a stretched-out oval — like a narrow canyon instead of a bowl. Steps that are good in one direction wildly overshoot in the other. Slow, bouncy, inefficient.
Feature scaling squashes everything into the same range. Now the surface looks like a proper bowl. Gradient descent walks straight to the bottom.
The three methods:
Min-max scaling (normalization) — rescales everything into [0, 1]:
x_scaled = (x - x_min) / (x_max - x_min)Simple. But one outlier can compress everything else into a tiny range. Sensitive to extreme values.
Z-score standardization — rescales around mean = 0, std = 1:
x_scaled = (x - mean) / stdThis is the default you should reach for. Doesn’t bound to [0, 1], but handles outliers much better and plays nicely with gradient descent.
Mean normalization — similar to z-score but uses range instead of std:
x_scaled = (x - mean) / (x_max - x_min)Less common. Centers the data around zero, values roughly in [−1, 1].
One critical thing: fit the scaler on training data only. Then apply those same mean and std to your test data. Never fit on test — that’s data leakage.
Question on my mind:
Does feature scaling affect what the model learns, or just how fast it learns?
Just how fast — with a catch. The final weights w will be in scaled space, so they aren’t directly interpretable as “price per extra square foot” anymore. To get back to original units, you’d reverse the scaling. But the predictions ŷ are identical — the model converges to the same line either way, just much faster with scaling.
Rule of thumb:
If any of your features has a range much larger than the others — scale everything. A good target: every feature roughly in [−1, 1] or [0, 1]. If a feature is already in that range, scaling it won’t hurt. Not scaling when you should will.
Feature Engineering & Polynomial Regression
Intuition:
Linear regression draws a straight line. But what if the truth is curved?
Say you’re predicting how much hair damage builds up from heat styling. The relationship isn’t a straight climb — it accelerates.
Style once a week, your hair barely notices. Style every single day with a flat iron on high heat — the damage compounds. Split ends lead to breakage, breakage leads to dryness, dryness makes you style more to tame frizz, which causes more damage. By the time you’re at daily styling, each extra session does far more harm than the first few ever did.
The damage doesn’t grow by the same amount each day — it grows faster the further in you go. That shape is a curve, not a line.
A straight line will miss this completely. It’ll underestimate damage at high frequency and overestimate it at low frequency. The truth curves upward — and that’s exactly what polynomial regression captures.
Feature engineering is the move: instead of giving the model just x, you hand it x, x², x³ — powered-up versions of the same input. The model is still doing linear regression. It just has richer features to work with. The curve comes from your features, not from changing the algorithm.
That’s the key insight. Polynomial regression isn’t a new model. It’s linear regression with engineered features.
Part 1 — Feature Engineering:
Feature engineering = creating new input columns from existing ones to help the model capture patterns it couldn’t see before.
House price example — your raw features are size and number of floors. A useful engineered feature:
# raw
size = 2000 # sq ft
floors = 2
# engineered
size_per_floor = size / floors # 1000 sq ft per floor — new signalAnother classic — instead of giving the model frontage and depth separately, multiply them to get area:
area = frontage * depth # the model couldn't learn this product on its ownThe model only knows how to take weighted sums. It can’t discover nonlinear combinations by itself. You have to hand it the combination as a new column.
Part 2 — Polynomial Regression:
The most systematic form of feature engineering: add powers of your original feature.
Start with one feature x. Build a polynomial by adding x², x³, and so on:
y-hat = w1·x + w2·x² + w3·x³ + bIn code, this is just creating new columns and running the same linear regression:
import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression# original feature - shape (m, 1)
X = np.array([[1],[2],[3],[4],[5],[6]])
# engineer x², x³ automatically
poly = PolynomialFeatures(degree=3, include_bias=False)
X_poly = poly.fit_transform(X)
# X_poly now has columns: [x, x², x³]
model = LinearRegression()
model.fit(X_poly, y)The weights w1, w2, w3 are what gradient descent learns — same algorithm, same cost function, same update rule. You changed nothing about the learning process. You just changed what you handed it.
Question on my mind:
If polynomial regression is still linear regression, what does “linear” actually mean?
Linear in the parameters — not in the features. The model is always computing a weighted sum of its inputs. What counts as “linear” is how the weights enter the equation, not how the features do. w1·x² is linear in w1 even though it's quadratic in x. That's why the same gradient descent math works perfectly. You're not bending the algorithm — you're bending the input space.
The degree dial — and why it matters:
- Degree 1 → model sees
x→ draws a straight line. Simple, but misses anything curved. - Degree 2 → model sees
x, x²→ draws a parabola. Good for U-shaped or accelerating patterns like our hair damage example. - Degree 3 → model sees
x, x², x³→ one inflection point. More flexibility to follow the data's natural shape. - Degree 9 → model sees
x, x², x³, x⁴, x⁵, x⁶, x⁷, x⁸, x⁹→ nine bends available. At this point the curve stops learning the pattern and starts memorising the data — including all the noise.
The higher the degree, the more the curve wiggles to hit every point — including the noisy ones. That’s overfitting. It looks great on training data and falls apart on anything new.
Drag the slider — watch what happens as degree climbs past 5 or 6. The red dashed line starts contorting to chase individual points. That’s overfitting in action.
Feature scaling + polynomial regression:
One thing that will bite you silently: x² and x³ explode in scale compared to x. If x is in the hundreds, x² is in the tens of thousands. Feature scaling becomes non-negotiable here.
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LinearRegression
pipe = Pipeline([
('poly', PolynomialFeatures(degree=3, include_bias=False)),
('scale', StandardScaler()), # scale AFTER expanding features
('model', LinearRegression())
])
pipe.fit(X_train, y_train)Always scale after expanding. Scaling before misses the engineered columns.
When to use polynomial features:
- Your residual plot shows a curve — the errors aren’t random, they have a pattern
- Domain knowledge says the relationship isn’t linear (growth rates, wear, diminishing returns)
- A low-degree polynomial fits better than linear with no meaningful overfitting
When not to: you have few data points and a high degree. The model will memorize, not generalize. Start with degree 2, validate, go higher only if needed.
That’s linear regression — fully dressed. You now have the intuition, the math, the code, and the traps. Next up: what happens when the model gets too clever for its own good. That’s regularization, and it’s where things get interesting.
