Data Science

Machine Learning

Python

Model Evaluation Metrics

Arth Jain

3 min read

·

11 hours ago

--

Listen

Share

(Part 1)

First, I am starting with model evaluation metrics. Why?

Due to when i am learning about classification metrics i found difficult to understand.

So,i think to start machine learning journey from these concept.

In this story i will talk about :

• Accuracy (Easy)

Accuracy

It tells us how many predictions our model got correct out of all the predictions it made.

Formula to calculate:

Suppose our model predicted as :

Actual Data(y) | Predicted Data(y^)
      1        | 1   (Correct)
      0        | 1     
      1        | 0
      0        | 0   (Correct)
      0        | 0   (Correct)

So seeing above result.How we can calculate the accuracy?

Acc = Total No. of Correct Prediction
      --------------------------------
      All Prediction(Wrong + Correct)

So, Our accuracy score will be:

Acc = 1+1+1/1+1+1+1+1

Acc = 3/5 or 0.6

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Q> Why we not just relay on accuracy score 
   to select(consider) model after comparing with other models?

Limitation of Accuracy:

1. It score can be misleading in imbalance dataset.

example:

Suppose we our working on Diabetes dataset.

Main Goal to predict whether a patient has diabetes or not .

As common sense there was the few data of patients who has diabetes disease(100 rows) out of 1000 rows of data with both (having disease and not having disease).

So,where is the problem? Everything looks fine.

Our dataset have less records of patients with diabetes disease.Our model not able to find more valuable patterns to classify as 1(having diabetes).

But as compare to patient with no diabetes (0) having large records and our model find patterns easily.

And Our model will classify large no. of patients as 0 (not having diabetes). Even the patient has Diabetes disease(1) but our model classify it as 0.

So,If we calculate the accuracy score then it might be as:

acc = 900 (correct predicted data)/1000 (total records)

acc = 0.90

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

How to prevent class imbalance ?

You might think,

Is there was any method to handle the class imbalance situation

Yes in (scikit learn library) there was a method to handle the class imbalance the situation.

Use class_weight = “balanced”

model = LogisticRegression(class_weight = "balanced")

It gave high priorities to minority class and less priority to majority class.

Note -> We also use other metrics also to select the best model as compare to other model.

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

How to use(code) Accuracy score in Python :

from sklearn.metrics import accuracy_score

model = LogisticRegression(class_weight = "balanced")

model.fit(X_train,y_train)


pred_data = model.predict(X_test)

acc = accuracy_score(y_test,pred_data)
print("Accuracy of Model",acc)

2. Accuracy just tell how many prediction was correct but it not tell type of wrong prediction.

What i mean?

• patient have diabetes(1) but model predicted as not having diabetes(0).
• patient have diabetes(1) but model predicted as not having diabetes(0)

— — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Thankyou very much (you read till end)

and also thank yourself too.Why ?

you get the idea of one metric.

— Arth Jain