Classification

Deepak Mishra

13 min read

·

Aug 19, 2026

--

Listen

Share

← Part 2— End-to-End Machine Learning Project | Part 4 → Linear Regression from Scratch →

Press enter or click to view image in full size

1. High-Level Architecture

Thanks for reading! Subscribe for free to receive new posts and support my work.

CLASSIFICATION PROBLEM
                             │
                             ▼
                     Define positive class
                             │
                             ▼
                       Train classifier
                             │
                             ▼
                    Generate predictions
                             │
                             ▼
                 ┌───────────┴───────────┐
                 │                       │
             Hard labels             Scores
              0 / 1              probability/score
                 │                       │
                 ▼                       ▼
          Confusion Matrix        Threshold tuning
                 │                       │
       ┌─────────┼─────────┐             ▼
       │         │         │        Precision/Recall
       ▼         ▼         ▼             │
   Accuracy   Precision  Recall          ▼
                                      ROC / AUC
       │                                 │
       └──────────────┬──────────────────┘
                      ▼
                Error Analysis
                      │
                      ▼
              Model improvement
                      │
                      ▼
             Production decision

The key shift:

Regression asks “how far away was my prediction?”

Classification asks:

“What kind of error did I make, and what is the cost of that error?”

2. MNIST is only the vehicle

It starts with MNIST handwritten digits.

The raw problem is:

Image
 ↓
0,1,2,3,4,5,6,7,8,9

But instead of immediately solving 10-class classification, simplifies it:

Is this a 5?

YES → 1
NO  → 0

This creates a binary classifier.

That is a very important ML engineering pattern:

Simplify the problem first so you can understand the behavior of the model and evaluation system.

3. Coding: Creating the binary target

y_train_5 = (y_train == 5)
y_test_5 = (y_test == 5)

So:

Actual digit     Target
-----------------------
5                True
anything else    False

Then:

from sklearn.linear_model import SGDClassifier

sgd_clf = SGDClassifier(random_state=42)sgd_clf.fit(
    X_train,
    y_train_5
)

Chooses SGDClassifier partly because SGD can handle large datasets efficiently and is also suitable for online learning.

4. Interview insight: Why SGD?

If asked:

Why would you choose SGD for a classifier?

Good answer:

“SGD is useful when the dataset is large, because it can process examples incrementally rather than requiring the entire dataset in memory. It is also naturally compatible with online learning.”

This specifically highlights these properties.

You can think:

Small dataset
   → many algorithms possible

Huge dataset
   → SGD / distributed training / online learning

5. The first trap: Accuracy

Initially obtains roughly:

95%+

cross-validation accuracy for the 5 detector.

Sounds excellent.

But then it creates a ridiculous classifier:

class Never5Classifier(BaseEstimator):

    def fit(self, X, y=None):
        pass    def predict(self, X):
        return np.zeros((len(X), 1), dtype=bool)

This classifier always predicts “not 5.”

And it gets around:

90% accuracy

because only about 10% of MNIST examples are 5s.

6. This is a critical lesson

Suppose you’re building:

Fraud detector

Fraud rate:

0.1%

A classifier that always says:

NOT FRAUD

gets:

99.9% accuracy

but is completely useless.

Therefore:

Accuracy is not necessarily a useful metric when classes are imbalanced.

7. Confusion Matrix

Instead of just:

Accuracy = 95%

look at:

PREDICTED
              Negative Positive

ACTUAL
Negative       TN       FPPositive       FN       TP

MNIST example gives:

TN = 53,272
FP =  1,307

FN =  1,077
TP =  4,344

This gives you much more information.

8. Understand TP / TN / FP / FN deeply

This is absolutely interview-critical.

True Positive

Actual = Positive
Prediction = Positive

True Negative

Actual = Negative
Prediction = Negative

False Positive

Actual = Negative
Prediction = Positive

Also called:

Type I error

False Negative

Actual = Positive
Prediction = Negative

Also called:

Type II error

9. Real-world interpretation

Consider a fraud system:

Positive = fraud
Negative = legitimate

Then:

TP → correctly detected fraud

TN → correctly allowed legitimate transactionFP → legitimate customer blockedFN → fraud allowed through

Now the question becomes:

Which is more expensive?

If:

Cost(FN) >> Cost(FP)

you should design for high recall.

If:

Cost(FP) >> Cost(FN)

you should emphasize precision.

This is where classification becomes decision-system design.

10. Precision

[
Precision =
\frac{TP}{TP+FP}
]

Meaning:

When my model says positive, how often is it correct?

Example:

100 alerts
80 actually positive

Precision = 80%

11. Recall

[
Recall =
\frac{TP}{TP+FN}
]

Also called:

• sensitivity
• true positive rate

Meaning:

Of all actual positives, how many did my system detect?

Example:

100 fraud cases exist
90 detected

Recall = 90%

12. Precision vs Recall

This is probably the single most important concept.

Imagine:

Fraud detection

Very conservative model

Only flag extremely suspicious transactions.

Precision → high
Recall    → low

Aggressive model

Flag almost everything suspicious.

Precision → low
Recall    → high

This explicitly demonstrates this tradeoff. Raising the classification threshold generally increases precision while reducing recall; lowering it does the opposite.

13. F1 score

This combines precision and recall using:

[
F1 =
2\frac{Precision \times Recall}
{Precision + Recall}
]

It is the harmonic mean.

Why harmonic mean?

Because:

Precision = 1.0
Recall = 0.1

should not look like a great classifier.

F1 heavily penalizes imbalance between precision and recall.

14. But don’t blindly optimize F1

This is a very important level distinction.

This explicitly points out that F1 favors classifiers with similar precision and recall, but this isn’t always what you want.

For example:

Child-safety content

You might prefer:

Precision = extremely high
Recall = moderate

because false positives are acceptable but dangerous content slipping through is not.

Cancer screening

You may prefer:

Recall = extremely high
Precision = moderate

because missing a real case can be much worse than generating additional false alarms.

Therefore:

Metric selection should come from business risk, not mathematical convenience.

15. Coding the basic metrics

from sklearn.metrics import (
    accuracy_score,
    precision_score,
    recall_score,
    f1_score
)

accuracy = accuracy_score(
    y_true,
    y_pred
)precision = precision_score(
    y_true,
    y_pred
)recall = recall_score(
    y_true,
    y_pred
)f1 = f1_score(
    y_true,
    y_pred
)

And confusion matrix:

from sklearn.metrics import confusion_matrix

cm = confusion_matrix(
    y_true,
    y_pred
)

16. Cross-validation

from sklearn.model_selection import cross_val_score

scores = cross_val_score(
    sgd_clf,
    X_train,
    y_train_5,
    cv=3,
    scoring="accuracy"
)

It demonstrates implementing stratified K-fold manually using StratifiedKFold.

The important point is:

Fold 1
Train → folds 2,3
Validate → fold 1

Fold 2
Train → folds 1,3
Validate → fold 2Fold 3
Train → folds 1,2
Validate → fold 3

17. cross_val_score vs cross_val_predict

This is a good coding interview question.

cross_val_score

Returns:

evaluation scores

Example:

scores = cross_val_score(...)

cross_val_predict

Returns:

out-of-fold predictions

Example:

predictions = cross_val_predict(...)

It uses cross_val_predict() to generate clean predictions for every training example, where each prediction was produced by a model that did not train on that example.

This is particularly useful for:

Confusion matrix
Precision
Recall
Error analysis
Threshold analysis

18. Decision scores — extremely important

Many classifiers don’t simply think:

YES / NO

Internally they generate a score.

For SGD:

y_scores = sgd_clf.decision_function(X)

Then:

prediction = y_scores > threshold

This demonstrates changing the threshold from 0 to 200000 and shows that recall decreases as the threshold increases.

This is a very important architecture concept.

19. Prediction is really two stages

Think:

Model
                 │
                 ▼
          Score / Probability
                 │
                 ▼
             Threshold
                 │
                 ▼
            Final decision

Not:

Model → magically gives YES/NO

This distinction becomes very important in production ML.

20. Production example

Suppose:

Fraud probability = 0.72

You can choose:

threshold = 0.50

→ fraud

or:

threshold = 0.90

→ legitimate

Same trained model.

Different business policy.

Therefore:

The model and the decision policy can be separate components.

That’s a powerful architecture insight.

21. How do you select the threshold?

y_scores = cross_val_predict(
    sgd_clf,
    X_train,
    y_train_5,
    cv=3,
    method="decision_function"
)

Then:

from sklearn.metrics import precision_recall_curve

precisions, recalls, thresholds = \
    precision_recall_curve(
        y_train_5,
        y_scores
    )

This lets you analyze:

threshold
    ↓
precision
    ↓
recall

22. Threshold design

Don’t say:

“We use threshold 0.5.”

Ask:

What is the cost of FP?
What is the cost of FN?
What volume can downstream operations handle?
What is the human-review capacity?
What SLA exists?
What precision is required?
What recall is required?

For example:

Model score
    ↓
Threshold = 0.83
    ↓
High confidence
    ↓
Automatic action

0.50 - 0.83
    ↓
Human review< 0.50
    ↓
Reject

This becomes a decision architecture, not just an ML model.

23. ROC Curve

The ROC curve plots:

Y axis → TPR / Recall

X axis → FPR

where:

[
FPR =
\frac{FP}{FP+TN}
]

This explains that ROC represents sensitivity versus 1 - specificity.

24. ROC AUC

AUC = Area Under ROC Curve.

Roughly:

AUC = 1.0 → excellent/perfect separation
AUC = 0.5 → random

This gives exactly this interpretation.

Coding:

from sklearn.metrics import (
    roc_curve,
    roc_auc_score
)

fpr, tpr, thresholds = roc_curve(
    y_true,
    y_scores
)auc = roc_auc_score(
    y_true,
    y_scores
)

25. ROC vs PR curve

This is a very common interview question.

ROC

TPR vs FPR

Useful when classes aren’t extremely imbalanced.

Precision-Recall

Precision vs Recall

Prefer it when:

positive class is rare

or:

false positives matter significantly

This explicitly recommends the PR curve in such cases.

Interview answer

“For highly imbalanced classification, ROC-AUC can look deceptively good because the negative class dominates. I would usually inspect the precision-recall curve and select a threshold based on the business cost of false positives and false negatives.”

26. Multiclass classification

Now move from:

5 vs not-5

to:

0
1
2
3
...
9

This explains two broad approaches.

Some models natively support multiclass:

Random Forest
Naive Bayes

Others are inherently binary and require strategies to construct multiclass classification.

27. One-vs-Rest / One-vs-All

For 10 classes:

Classifier 1 → is digit 0?
Classifier 2 → is digit 1?
Classifier 3 → is digit 2?
...
Classifier 10 → is digit 9?

Then choose the highest score.

Architecture:

Input
               │
      ┌────────┼────────┐
      ▼        ▼        ▼
     0-vs-all 1-vs-all ... 9-vs-all
      │        │             │
      └────────┴─────────────┘
               │
          highest score
               │
               ▼
            class

28. One-vs-One

Instead of:

10 classifiers

you train:

[
\frac{N(N-1)}{2}
]

For 10 classes:

10 × 9 / 2 = 45

This demonstrates OneVsOneClassifier(SGDClassifier(...)) and obtains 45 binary estimators for MNIST.

29. OvA vs OvO

One-vs-Rest

N classifiers

Good when:

N = relatively small

One-vs-One

N(N-1)/2 classifiers

Can be useful for algorithms where training on smaller binary datasets is advantageous.

Concern

You need to consider:

Training cost
Inference cost
Memory
Latency
Class imbalance
Algorithm characteristics

Not just which strategy is theoretically elegant.

30. Random Forest multiclass

Random Forest directly supports multiclass classification.

from sklearn.ensemble import RandomForestClassifier

forest_clf = RandomForestClassifier(
    random_state=42
)forest_clf.fit(
    X_train,
    y_train
)prediction = forest_clf.predict(
    [some_digit]
)

And:

forest_clf.predict_proba([some_digit])

returns class probabilities. This illustrates an example where the model assigns 80% probability to digit 5.

31. Scaling can dramatically matter

It shows that scaling the MNIST inputs improves SGD multiclass accuracy from roughly:

84%

to:

91%

in the example.

Preprocessing
      ↓
Model
      ↓
Evaluation

The important lesson:

Model performance isn’t only about model choice; representation and preprocessing matter enormously.

32. Error Analysis

This becomes particularly valuable for a interview.

Suppose:

Accuracy = 91%

Don’t immediately tune hyperparameters.

Ask:

What mistakes is the model making?

Examine the confusion matrix to identify problematic digit pairs such as 8/9 and 3/5.

33. Error analysis workflow

I recommend memorizing:

Model
  ↓
Out-of-fold predictions
  ↓
Confusion Matrix
  ↓
Identify high-error classes
  ↓
Sample individual errors
  ↓
Find root cause
  ↓
Fix data/features/model
  ↓
Re-evaluate

Possible root causes:

Bad labels
Poor features
Insufficient training data
Class imbalance
Data quality
Distribution shift
Model capacity
Preprocessing

34. Example: 3 vs 5

This explains why the linear SGD classifier confuses 3 and 5.

It essentially assigns weights to pixels and sums the weighted pixel intensities. Since 3 and 5 differ in relatively subtle spatial patterns, the linear model struggles.

Possible improvements:

Better preprocessing
Better feature representation
Center images
Reduce rotation
More training examples
More expressive model

This is much better than saying:

“Let’s increase the number of epochs.”

35. Multilabel classification

Now something more interesting.

Normal classification:

Input → ONE class

Multilabel:

Input → MULTIPLE binary labels

Example:

Photo
 ↓
Alice = 1
Bob = 0
Charlie = 1

Output:

[1, 0, 1]

It uses exactly this face-recognition example to explain multilabel classification.

36. Coding multilabel classification

It creates two labels:

y_train_large = (y_train >= 7)

y_train_odd = (y_train % 2 == 1)y_multilabel = np.c_[
    y_train_large,
    y_train_odd
]

Then:

knn_clf = KNeighborsClassifier()

knn_clf.fit(
    X_train,
    y_multilabel
)

The output becomes:

[large?, odd?]

Note that not all classifiers support multilabel classification.

37. Multilabel metrics

You now have:

Label 1 → precision/recall/F1
Label 2 → precision/recall/F1
Label 3 → ...

This demonstrates:

f1_score(
    y_true,
    y_pred,
    average="macro"
)

Macro means:

F1_label1
+
F1_label2
+
...
----------------
number of labels

Discussed weighted averaging when labels have different support.

38. Multioutput classification

This is:

Multiple outputs
+
Each output can have multiple values

This illustrates it by removing noise from MNIST images.

Input:

Noisy 784-pixel image

Output:

Clean 784-pixel image

Each pixel is effectively an output variable.

39. Multioutput architecture

Think:

Input
                    │
                    ▼
               ML Model
                    │
        ┌───────────┼───────────┐
        ▼           ▼           ▼
     output 1    output 2    output 3
        │           │           │
        ▼           ▼           ▼
     class/value  class/value  class/value

This pattern appears extensively in modern systems:

Recommendation
→ multiple recommendations

Computer vision
→ multiple detectionsNLP
→ multiple attributesMultitask learning
→ multiple predictions

The last examples are general ML extensions; the source specifically establishes the multioutput concept using image denoising.

40. Data augmentation

This exercise introduces an important practical technique:

Original image
       │
 ┌─────┼─────┐
 ▼     ▼     ▼
left  right  up/down
       │
       ▼
More training examples

This is called:

Data augmentation / training-set expansion

The exercise specifically proposes shifting MNIST images by one pixel in four directions and adding the resulting examples to the training set.

For computer vision, this becomes extremely important.

41. Production ML design

Now let’s translate into a ML system design.

Suppose the interviewer asks:

Design a fraud classification system.

You should think:

TRANSACTION
                       │
                       ▼
                Feature Pipeline
                       │
                       ▼
                 ML Classifier
                       │
                       ▼
                 Risk Score
                       │
                       ▼
               Decision Policy
                       │
          ┌────────────┼─────────────┐
          ▼            ▼             ▼
       APPROVE       REVIEW        DECLINE

Notice:

Model ≠ decision system.

The model produces:

risk_score = 0.83

The policy decides:

0.00–0.40 → approve
0.40–0.80 → review
0.80–1.00 → decline

This is a very strong architecture.

42. Production monitoring

For classification, I would monitor:

Data

Feature drift
Missing values
Schema changes
Category distribution
Input freshness

Model

Prediction distribution
Probability distribution
Confidence
Precision
Recall
FPR
FNR
AUC

Business

Fraud loss
False decline rate
Customer complaints
Manual-review volume
Conversion rate
Revenue impact

It gives you the classification metrics; It provides the broader production monitoring framework.

43. Threshold monitoring

A subtle but important production problem:

Suppose model quality stays constant:

AUC = 0.95

but the underlying class distribution changes.

For example:

Fraud rate:
0.5% → 2.5%

Your:

Precision
Recall
FPR
Operational volume

can all change.

Therefore, monitoring only:

AUC

is insufficient.

An engineer monitors model + data + business outcome.

44. Coding interview cheat sheet

If the interviewer gives you a classification problem, your basic code skeleton should look like:

from sklearn.model_selection import (
    StratifiedKFold,
    cross_val_score,
    cross_val_predict
)

from sklearn.metrics import (
    confusion_matrix,
    precision_score,
    recall_score,
    f1_score,
    roc_auc_score,
    precision_recall_curve,
    roc_curve
)# Train
model.fit(X_train, y_train)# Cross-validation
scores = cross_val_score(
    model,
    X_train,
    y_train,
    cv=5,
    scoring="f1"
)# Out-of-fold predictions
y_pred = cross_val_predict(
    model,
    X_train,
    y_train,
    cv=5
)# Confusion matrix
cm = confusion_matrix(
    y_train,
    y_pred
)# Precision
precision = precision_score(
    y_train,
    y_pred
)# Recall
recall = recall_score(
    y_train,
    y_pred
)# F1
f1 = f1_score(
    y_train,
    y_pred
)

For score-based models:

scores = cross_val_predict(
    model,
    X_train,
    y_train,
    cv=5,
    method="decision_function"
)

Then:

precision, recall, thresholds = \
    precision_recall_curve(
        y_train,
        scores
    )

45. What NOT to do

❌ Don’t say:

“Accuracy is 95%, so the model is good.”

Instead:

“I need to understand class distribution and the cost of FP/FN before deciding whether accuracy is meaningful.”

❌ Don’t say:

“I’ll use threshold 0.5.”

Instead:

“I’ll select the threshold based on the required precision/recall operating point and business cost.”

❌ Don’t say:

“AUC is 0.98, therefore production-ready.”

Instead:

“AUC measures ranking/separation performance; I still need to select an operating threshold and validate precision, recall, calibration, latency, and business KPIs.”

The source itself focuses on selecting metrics and precision/recall tradeoffs rather than treating one metric as universally sufficient.

46. 12 questions you should expect in an interview

Fundamental

1. Why is accuracy bad for imbalanced data?

Because the majority class can dominate the metric.

2. Explain precision.

TP / (TP + FP)

Of predicted positives, how many are actually positive?

3. Explain recall.

TP / (TP + FN)

Of actual positives, how many did we detect?

4. Precision vs recall?

Precision ↑ → usually Recall ↓
Precision ↓ → usually Recall ↑

Controlled largely through threshold selection.

5. What is F1?

Harmonic mean of precision and recall.

6. ROC vs PR?

ROC → TPR vs FPR

PR  → Precision vs Recall

PR is particularly useful for rare positives.

47. Questions

7. How would you choose a classification threshold?

Answer:

Business cost
     ↓
Required precision/recall
     ↓
Operational capacity
     ↓
PR curve
     ↓
Select threshold
     ↓
Validate on holdout

8. Why can ROC-AUC be misleading?

Especially under severe class imbalance, because the huge number of negatives can make FPR look small even when precision is poor.

9. How would you debug a classifier with 90% accuracy?

Class distribution
        ↓
Confusion matrix
        ↓
Precision / Recall
        ↓
PR curve
        ↓
Error slices
        ↓
Data quality
        ↓
Feature analysis

10. How would you design a fraud classifier?

You should discuss:

Real-time features
      ↓
Model
      ↓
Risk score
      ↓
Threshold/policy
      ↓
Approve/review/decline
      ↓
Feedback labels
      ↓
Retraining

11. How do you handle multiple classes?

Discuss:

Native multiclass models
One-vs-Rest
One-vs-One

and compare computational tradeoffs.

12. What is multilabel classification?

One input can have multiple binary labels:

Input → [1,0,1,1]

versus multiclass:

Input → exactly one class

It explicitly distinguishes these concepts.

48. Mental model

I would memorize this:

RAW DATA
                    │
                    ▼
               CLASSIFIER
                    │
                    ▼
               SCORE / P
                    │
                    ▼
              THRESHOLD
                    │
                    ▼
             FINAL DECISION
                    │
                    ▼
          ┌─────────┴─────────┐
          │                   │
       Positive            Negative
          │                   │
          └─────────┬─────────┘
                    ▼
              CONFUSION MATRIX
                    │
        ┌───────────┼────────────┐
        ▼           ▼            ▼
    Precision     Recall        F1
        │           │            │
        └───────────┼────────────┘
                    ▼
              PR / ROC CURVE
                    │
                    ▼
              ERROR ANALYSIS
                    │
                    ▼
              MODEL / DATA
              IMPROVEMENT

49.

This distinction is worth remembering :

End-to-end MLClassification and Regression example

MNIST classification

RMSE / MAE

Accuracy

Data pipeline

Confusion matrix

Feature engineering

PrecisionPreprocessing

Recall

Model selection F1,Cross-validation,PR curve,

Hyper parameter tuning,ROC/AUC,Deployment, Multi class Monitoring, Multi label

Retraining

Error analysis

So:

E2E Machine Learning teaches you how to build an ML project. Classification teaches you how to reason about classification decisions and classifier failures.

50. The answer I would give in a interview

If the interviewer asks:

“Explain how you would build a classification system.”

I’d answer:

“I would first define the positive and negative classes and understand the business cost of false positives versus false negatives. I would establish a representative validation strategy, especially if the classes are imbalanced. I would start with a simple baseline, use stratified cross-validation, and evaluate the classifier using a confusion matrix rather than relying only on accuracy. I would measure precision, recall and F1, and inspect the precision-recall curve to select an operating threshold based on the business objective. If appropriate, I would also compare ROC-AUC, particularly for ranking quality. Then I would perform slice-based error analysis to understand where the model fails, improve the data or features, and retrain. For production, I would separate the model’s score from the decision policy, monitor prediction distributions, precision/recall, data drift and business KPIs, and establish a feedback and retraining loop.”

Answer:

And one final point: it’s own conclusion is essentially that good classification is not merely about training a classifier — it is about choosing appropriate metrics, selecting the precision/recall operating point, comparing models, and performing error analysis.

I would rate these topics in priority:

🔥🔥🔥🔥🔥  Precision / Recall / F1
🔥🔥🔥🔥🔥  Confusion Matrix
🔥🔥🔥🔥🔥  Threshold selection
🔥🔥🔥🔥🔥  Imbalanced classification
🔥🔥🔥🔥    ROC vs PR
🔥🔥🔥🔥    Error analysis
🔥🔥🔥🔥    Cross-validation
🔥🔥🔥     Multiclass / OvA / OvO
🔥🔥🔥     Multilabel
🔥🔥       Multioutput

These are the concepts I would expect you to be able to code from scratch, explain mathematically, and translate into a production system design.

Continue the Series:

Next → Part 4 → Linear Regression from Scratch