⚡ Gradient Descent — Part 1
How Does a Computer Know Which Way to Move?
In the previous articles of our ML Handbook, we started with one of the simplest machine-learning problems:
How can a computer learn the best line from data?
We began with a simple model:
ŷ = wx + b
where:
- w is the slope
- b is the intercept
- x is the input
- ŷ is the prediction
But simply drawing a line is not learning.
The computer needs to answer a much more important question:
Which values of (w) and (b) make this line the best possible line for our data?
To answer that, we introduced the idea of loss.
The loss tells us how wrong our predictions are.
For Mean Squared Error:
L(w, b) = (1/n) Σᵢ₌₁ⁿ (yᵢ − wxᵢ − b)²
So our goal becomes:
Find w, b that minimize L(w, b)
In the previous article, we saw that Linear Regression can solve this using Least Squares.
But now we encounter an important question:
What if we cannot directly calculate the best parameters?
That is where our next idea begins.
🚧 When There Is No Direct Answer
For simple Linear Regression, we can derive a mathematical solution.
But imagine that our model becomes more complicated.
Instead of two parameters:
w,b
we might have:
θ₁, θ₂, θ₃, …, θₘ
where (m) could be thousands, millions, or even billions.
Now finding the exact mathematical solution can become extremely difficult.
So instead of asking:
“Can I calculate the answer directly?”
we ask:
“Can I start somewhere and gradually improve the answer?”
This is a completely different way of thinking about optimization.
And it leads us to:
Gradient Descent
🏔️ Imagine Standing on a Mountain
Let’s forget machine learning for a moment.
Imagine that you are standing somewhere on a mountain.
Your goal is simple:
Reach the bottom of the valley.
But there is a problem.
It is completely dark.
You cannot see the mountain.
You don’t know where the valley is.
You can only feel the ground immediately around you.
What could you do?
You could ask:
Which direction slopes downward?
Then take a small step in that direction.
After taking the step, check the slope again.
Then take another step.
And repeat.
Eventually, you may reach the bottom.
This is the basic intuition behind Gradient Descent.
🧠 The Mountain Is Actually the Loss Function
In machine learning, the “mountain” is not a physical mountain.
It is the loss landscape.
Remember:
L(w,b)
Our loss depends on the parameters (w) and (b).
Different values of (w) and (b) produce different losses.
For example:
The numbers are only illustrative, but the idea is important.
Some parameter combinations produce:
High Loss
while others produce:
Low Loss
Our objective is to find the parameter combination at the bottom.
Conceptually:
High Loss ↓ Lower Loss ↓ Minimum Loss
🔍 But How Does the Computer Know Which Way to Move?
This is the central question.
The computer cannot simply “look” at the entire loss landscape and know where the minimum is.
Instead, it calculates something called a gradient.
But before understanding the gradient, we need to understand something simpler:
What Is a Derivative?
Suppose we have a function:
f(x)=x²
Its derivative is:
df/dx = 2x
What does this derivative tell us?
It tells us the rate at which the function changes as (x) changes.
For example, at:
x=3
we have:
df/dx = −6
The positive value tells us that the function is increasing as (x) increases.
At:
x=-3
we get:
df/dx = −6
Now the function is decreasing as (x) increases.
So the derivative gives us information about the local direction and steepness of a function.
🧭 The Derivative Is Like a Local Compass
Imagine that you are walking along a hill.
The derivative tells you something about the slope underneath your feet.
If the slope is:
+5
the terrain rises strongly in one direction.
If the slope is:
-5
the terrain falls in that direction.
If the slope is:
0
you are at a locally flat point.
This is why derivatives are so useful for optimization.
They give us local information about the landscape.
➡️ But Our Model Has More Than One Parameter
Our Linear Regression model has two parameters:
w
and
b
Therefore, our loss depends on two variables:
L(w,b)
We need to know:
How does the loss change when (w) changes?
and:
How does the loss change when (b) changes?
So we calculate two partial derivatives:
∂L/∂w
and:
∂L/∂b
Together, these form the gradient:
∇L = [ ∂L/∂w ; ∂L/∂b ]
The gradient is essentially a collection of derivatives.
🧭 What Does the Gradient Tell Us?
This is the key idea:
The gradient points in the direction in which the loss increases most rapidly.
Notice the word:
increases.
Gradient Descent wants to decrease the loss.
Therefore, we move in the opposite direction:
−∇L
So:
Gradient → direction of greatest increase
while:
−Gradient → direction of greatest decrease
That is where the name comes from:
Gradient + Descent
We use the gradient to determine how to move down the loss landscape.
🏔️ Think About Walking Downhill
Imagine you are standing on a slope.
The gradient tells you:
“The steepest uphill direction is over there.”
You want to go downhill.
So you move in the opposite direction.
In machine learning:
Gradient → uphill direction
and:
−Gradient → downhill direction
The computer doesn’t need to know the entire landscape.
It only needs to know the local slope.
Then it takes a step.
And checks again.
👣 Gradient Descent Is a Repeated Process
Suppose our initial parameters are:
w₀, b₀
The computer calculates the current loss.
Then it calculates the gradient.
Then it moves downhill.
We get:
(w₀, b₀) → (w₁, b₁)
Then it calculates the gradient again:
Get Anupama Bidargaddi’s stories in your inbox
Join Medium for free to get updates from this writer.
(w₁, b₁) → (w₂, b₂)
Then again:
(w₂, b₂) → (w₃, b₃)
And so on.
Conceptually:
Start → Check slope → Step downhill → Check again → Step again → ⋯
This is iterative optimization.
🎯 Why Not Take One Giant Step?
Here we encounter another important idea.
Suppose the computer knows which direction is downhill.
Should it take a tiny step?
A medium step?
Or a huge step?
This is controlled by something called the:
Learning Rate
We usually represent it by:
η
The learning rate determines how large a step the algorithm takes.
We will study the mathematics of this update in Part 2.
For now, the intuition is enough:
Small learning rate
Small steps:
→ slow but potentially stable
Large learning rate
Large steps:
→ faster but potentially unstable
So Gradient Descent is not simply:
“Move downhill.
It is:
“Move downhill by an appropriate amount.”
📉 What Happens to the Loss?
Suppose we begin with a poor line.
Its loss might be:
L=100
After one update:
L=72
After another:
L=48
Then:
L=27
Then:
L=13
Then:
L=7
The model is gradually improving.
Conceptually:
100 → 72 → 48 → 27 → 13 → 7 → ⋯
The parameters are changing because the computer is continuously trying to reduce the loss.
🔄 Learning Is Not a Single Calculation
This is an important shift in how we think about machine learning.
Learning is not necessarily:
Calculate answer once
Instead, it can be:
Guess → Measure → Adjust → Measure again → Adjust again → ⋯
Gradient Descent turns learning into a process of iterative improvement.
🧩 A Simple Example
Imagine that our current line is:
ŷ = 0.5x + 1
But the actual data suggests that the line should be steeper.
The computer calculates the loss.
Then the gradient indicates:
“Increasing (w) would reduce the loss.”
So the computer increases (w).
Perhaps:
w=0.5
becomes:
w=0.7
The new line is:
ŷ = 0.7x + 1
The computer calculates the loss again.
If the loss decreased, the update was useful.
Then it calculates the gradient again.
Perhaps the next update gives:
w=0.85
Then:
w=0.94
Then:
w=0.98
The line gradually approaches a better solution.
The important point is not the exact numbers.
The important point is the process:
Gradient → Direction → Update → Lower Loss
⚠️ What Happens Near the Minimum?
As we approach the bottom of the valley, the slope becomes smaller.
Therefore, the gradient becomes smaller.
So the updates become smaller too.
Eventually:
∇L ≈ 0
This means that the algorithm has reached a point where there is little useful movement left.
We call this convergence.
In simple terms:
The model has reached a point where further updates produce very little improvement.
We will examine convergence more carefully in Part 3.
🤔 Why Is This Idea So Important?
At this point, Gradient Descent might seem like just another algorithm.
It is much more important than that.
The fundamental idea is:
Use the error to determine how the parameters should change.
This idea scales.
For our simple line, we have only:
w, b
But a neural network may have millions of parameters:
θ₁, θ₂, θ₃, …, θₘ
Instead of asking:
“What should every parameter be?”
we ask:
“Given where I am now, how should every parameter change to reduce the loss?”
That is a much more practical problem.
🚀 From a Simple Line to Deep Learning
This is where our simple Linear Regression example becomes surprisingly powerful.
For Linear Regression:
ŷ = wx + b
we have two parameters.
For a neural network, we may have:
θ = [θ₁, θ₂, …, θₘ]
The loss becomes:
L(θ)
The gradient becomes:
∇θL
And the same fundamental idea remains:
θ ← θ − η∇θL
The model has become much more complicated.
But the optimization principle has not fundamentally changed.
🧠 The Big Picture
Let’s connect everything we have learned so far in the ML Handbook.
We started with data.
↓
We chose a model:
ŷ = wx + b
↓
The model made predictions.
↓
We measured prediction error using a loss function:
L(w, b)
↓
We asked:
How can we reduce this loss?
↓
For simple Linear Regression, Least Squares provides an analytical solution.
↓
But when an analytical solution is unavailable or impractical, we need an optimization strategy.
↓
That brings us to:
Gradient Descent
And its fundamental idea is:
Look at the local slope → move downhill → repeat
📌 The One Idea to Remember
If you remember only one thing from Part 1, remember this:
Gradient Descent is a method for minimizing a loss function by repeatedly moving the model parameters in the direction that decreases the loss.
Or even more simply:
Gradient tells us where the loss increases.
Therefore:
Negative Gradient tells us where to move to decrease it.
That is the intuition.
But intuition alone is not enough.
We now need to understand exactly how the computer calculates the gradient and updates the slope and intercept.
🔜 What’s Next? — Part 2
In Part 2, we will stop talking about the mountain metaphor and actually do the mathematics.
We will derive:
∂L/∂w
and:
∂L/∂b
Then we will use them to construct the actual update equations:
w ← w − η(∂L/∂w)
b ← b − η(∂L/∂b)
And most importantly, we will take a real numerical example and watch the computer update the line step by step.
Because the real magic begins when we move from:
“Gradient tells us which way to go.”
to:
“Here is exactly how the computer calculates that direction.”
That is where the mathematics of Gradient Descent begins.
