LoRA Fine Tuning: Explained from Scratch.
This blog explains LoRA (Low-Rank Adaptation) in very simple terms.
It is written for beginners like me. I wrote it with the intention that I should be able to get everything whenever I read it again.
Before going through the LoRA we need to brush up our skills on Matrices.
What is Rank in Matrices?
The rank of a matrix is the number of truly independent pieces of information inside it.
- Rank can be the number of linearly independent rows
- Rank can be the number of linearly independent columns
Both always give the same answer.
Below is the simple matrix A we have.
A =
[ 1 2 3 4 5 ]
[ 2 4 6 8 10 ]
[ 1 1 1 1 1 ]
[ 3 5 7 9 11 ]
- Row2 can be fetched by doing 2 × Row1
- Row3 is NOT a multiple of Row1
- Row4 can be fetched by doing Row1 + Row3
So here only Row 1, Row 3 cannot be fetched from any other row so, we called the rank of this matrix as 2 → Rank = 2
Example 2
A =
[ 1 2 3 4 ]
[ 2 4 6 8 ]
[ 3 5 7 9 ]
- Row2 = 2 × Row1
- Row3 is NOT a multiple of Row1
So, here only Row 1, Row 3 cannot be fetched from any other row so rank will be 2 for this matric so, Rank = 2
Important Rule About Rank
You will never see a situation where:
- rows give one rank &
- columns give another rank
This is a theorem in linear algebra: Row Rank = Column Rank
Also:
Rank ≤ min(number of rows, number of columns)
- If rows in matrix < columns in matrix → check rows
- If columns in matrix < rows in matrix → check columns
So, Rank = total number of independent rows (or columns)
So, we got a clear understanding of what Rank is and how it is calculated in matrices, so let’s move ahead now.
Incase if i am not clear still. I recommend you to watch the starting part of this video where he explained very very clearly on what is Rank in Matrix(P.S: I felt he didn’t explain LoRA fully though).
Now, moving ahead, Many might know the below topic but explaining just for those who are beginners.
How a Neural Network Learns
A neural network is a system that:
- Takes input
- Processes it using weights
- Produces an output
Input → Weights → Output (Forward Pass)
When input is given to a neural network:
- Input is converted into numbers
- These numbers are multiplied with weights
- The result becomes the output
This step is called the forward pass.
In simple form:
- output = weight × input
Weights are just numbers stored inside the model that represent what the model has learned.
What is Error?
After producing output:
- The model compares its output with the correct answer
- The difference is called error (or loss)
What is a Backward Pass?
The backward pass is the step where the model learns from its mistakes.
During the backward pass:
- The error is sent backward through the network
- The model calculates gradients for each weight
So, what is a Gradient?
A gradient tells the model:
How much and in which direction a weight should change to reduce the error
- Positive gradient → decrease the weight
- Negative gradient → increase the weight
- Large gradient → big change
- Small gradient → small change
Below is the process on how learning happens in a Neural Network.
- Forward pass → get output
- Calculate error
- Backward pass → compute gradients
- Update weights using gradients
This loop runs many times.
So,
- Training: forward pass + backward pass (weights change)
- Inference: only forward pass (weights do not change)
LoRA mainly changes which weights receive gradients.
Neural networks learn by producing output in a forward pass and correcting their weights during the backward pass using gradients.
Now let’s understand what Fine Tuning is and how it works.
What Does Full Fine-Tuning Do?
Let’s say if a weight matrix is W(By now we should be clear on what is weights in neural network):
W ∈ ℝ^(d × k)
- W = weight
- ℝ = Real Numbers
- d,k = matrix size
- ∈ = belongs to / equals to
Full fine-tuning allows every value in the weight matrix to change during training.
This is what we call maximum freedom in Full FineTuning. But it comes with maximum cost (memory, compute, storage).
Why Full Fine-Tuning is a Problem?
Research has shown that when large pre-trained models are fine-tuned:
- The model does not change everywhere
- Only a small number of directions / patterns actually change
THIS IS THE CRUX FOR WHY LoRA is invented……..
In other words: The required changes lie in few dimensional space
Please remember this point→ Only a small number of directions / patterns actually change even in full fine tuning.
Now let’s go ahead and understand LoRA. Then we will understand how LoRA leverages this?
What Is LoRA?
LoRA (Low-Rank Adaptation) is a way to fine-tune large models by learning small, low-rank updates instead of changing all model weights.
I know it doesn’t make sense much now but to understand it better, Let’s dive in.
In short:
- We freeze the original model
- We add a small trainable part
That small part learns the task (For example, when fine-tuning on medical data, the task-specific medical behaviour is learned in this small part while the original model weights remain unchanged.) → You can relate to this in the above image as well.
Core Idea Behind LoRA
To change the model’s behaviour:
- We don’t need to change everything.
- We only need to add a few new directions of change.(Research shows that even during full fine-tuning, many weights are updated, but the meaningful changes lie in a small number of coordinated directions rather than all dimensions changing independently.)
That is why LoRA exists.
How LoRA Changes the Weights
Instead of updating the full matrix W, LoRA says:
W_new = W + ΔW
- W is kept as is in LoRA, We only work on ΔW in LoRA.
But ΔW is not stored directly.
LoRA represents it as:
- ΔW = B × A
Where:
- B ∈ ℝ^(d × r)
- A ∈ ℝ^(r × k)
So, in LoRA we represent the new matrix ΔW as 2 different matrix A,B
Get praveenreddy_c’s stories in your inbox
Join Medium for free to get updates from this writer.
— A is the real number matrix of size d x r
— B is the real number matrix of size r x k
r → is the rank we pass as the hyperparameter.
Now, what should be the size of those A,B Matrices? As per above we can see they are d x r and r x k size.
How has this d,k size has come ? — This is the same size as the original weight of matrix W.
I recommend watching the YouTube video which I mentioned above by
Satyajit Pattnaik to get more understanding on matrices. In that video he has explained clearly on how to represent one matrix into 2 different matrices using the rank. P.S : LoRA I feel is not fully covered in that video.
So:
- W has size d × k
- ΔW = B × A = has size d × k → the d x k is represented as d x r & r x d
So, B × A ∈ ℝ^(d × k) → Meaning, B x A matrix is nothing but the real numbers(ℝ) matrix of size d x k.
So, B, A matrices will have:
— Same shape as W
Example: Let’s say If Rank(r) = 4
then,
- B ∈ ℝ^(d × 4)
- A ∈ ℝ^(4 × k)
ΔW = B × A ∈ ℝ^(d × k)
So, This can be understood as, We are allowing the layer to learn 4 new independent directions of change.
So, r is a key hyperparameter in LoRA. It controls how many independent directions the model is allowed to change during fine-tuning. A small r allows limited, focused adaptation, while increasing r makes LoRA behave closer to full fine-tuning. Even in full fine-tuning, although all weights can change, the meaningful changes usually lie in a low-dimensional space.
What Does Rank r Mean in Plain English?
Rank r = number of new behaviors you allow
- r = 1 → one new direction of change
- r = 4 → four independent changes
- r = 16 → more flexibility
But → Still far smaller than full fine-tuning
Very Important Clarification(Again)
Rank is NOT calculated from the matrix
Rank is chosen by us
- r is a hyperparameter
Its similar to hyperparameters we have in Deep Learning, Just like:
- learning rate
- batch size
- number of layers
LoRA says:
“I will allow at most r new independent directions of change, regardless of what W₀ looks like.”
What Are A and B Matrices Actually Doing in ΔW?
Are you still not clear with A, B matrices which are derived from ΔW? you can think of them like this:
- A → What part of this input matters for my task?
- B → How should I adjust the original output?
Together:
- They create small, focused corrections
- They do not replace original knowledge
What Happens During Training in LoRA?
Step 1: Forward Pass
x = input vector
X is applied to the actual weight + new ΔW we are creating.
We know ΔW = BA → BA or AB both are same(Don’t get confused)
- y = W x + B A x
Initially:
- B = 0
- BA = 0
So:
Model behaves exactly like the original model at the very first step in training process.
Step 2: Compare Output with Correct Answer
Loss is calculated.
Step 3: Backpropagation
Gradients flow like this:
- W → NO gradients — As in LoRA we are keeping this constant. We don’t update W at all.
- A, B → receive gradients
So the model learns:
“How should A and B change so that BA matches the output correctly?”
Step 4: After Many Steps
- BA becomes non-zero
- It consistently adjusts outputs
- Model behaviour changes
But:
- Original knowledge stays intact
- Only small, focused corrections are learned
Why Low Rank Is Enough?
LoRA works because:
- LLMs already know language
- New tasks rarely need new knowledge
- They need small behavioral shifts
LoRA captures those shifts efficiently.
How Rank r Actually Helps and what made LoRA go to option for Fine Tuning?
When we write:
- ΔW = B × A
This enforces(we decide the value of r [hyperparameter]):
- rank(ΔW) ≤ r
Meaning:
- You are allowed at most r independent update directions
- No matter how long you train
- No matter how much data you have
This is a hard constraint, not a regularizer.
Where Rank Helps Inside the Model(For now Let’s consider only Attention Block [Q,K,V] only)
Think about the Q matrix in attention.
Q controls:
- What features a token queries
- What it pays attention to
Full Fine-Tuning:
- Changes every query dimension independently
Where as LoRA with r = 4:
Allows 4 new attention patterns
- Learned across all tokens
- Shared consistently
Examples:
- One direction focuses on keywords
- Another on syntax
- Another on domain-specific terms
That is enough for most tasks.
as per the LoRA paper, they have even mentioned why LoRA works better than Adaptors.
Why Adapter Layers Didn’t Work Well
Adapter layers were introduced to reduce fine-tuning cost by adding small extra layers and training only them while freezing the main model.
While adapters reduce the number of trainable parameters, they introduce a key problem:
- Adapters add new layers to the model
- These layers run sequentially, not in parallel
- This increases inference latency, especially when batch size is small (like in real-time APIs)
Large models rely heavily on GPU parallelism, and adapter layers break this efficiency.
The problem becomes worse for very large models that are split across multiple GPUs.
So, although adapters save parameters, they:
- Slow down inference
- Scale poorly in latency-sensitive production systems
Which clearly explains the success reason behind LoRA.
As per the LoRA paper, the LoRA is applied only to the attention block of the transformer but not the FFN.
Why LoRA Is Applied to Q, K, V (and Not FFN)
LoRA is applied mainly to the attention matrices (Q, K, V) because:
- Attention controls what the model focuses on
- Small changes in attention can strongly change model behaviour
- This gives maximum impact with very few parameters
The FFN (MLP) layers are much larger and contain more parameters. Applying LoRA there would increase parameter count and reduce efficiency.
For simplicity and parameter efficiency, the paper focuses on attention weights and freezes FFN layers.
LoRA can be applied to FFN, but this was left for future work in the paper.
Adapters → add new layers (new computation path)
LoRA → modifies existing computation path
LoRA is:
- Inside the matrix, not beside it
So, I hope I made the LoRA explanation as clear as possible.
Now, You go ahead and look at the LoRA paper and dive more deeper.
