Imbalanced Data Problems

Some effective ways to handle imbalanced classes for machine learning models

4 min readJun 20, 2019

--

When it comes to classification problems in machine learning, imbalanced classes can make it very difficult to construct a model that performs well with regards to the desired metrics. In the real world, it is very uncommon that you will find the perfect dataset to work with, and many issues deal with heavily imbalanced classification problems such as credit card fraud, medical detections, and spam filtering.

This blog will go over some methods that were used in our Capstone project in order to handle this imbalanced class issue.

Predicting Accident Severity

In my last blog, I discussed how for one of our models we wanted to take a look at traffic accident data in the UK to see if we could make a model that could predict the level of accident severity.

For this classification problem, the target variable was broken down like so:

Fatal and Serious accidents are combined into one class

Right away you can see that we are dealing with an imbalanced classification problem and evaluating our models based on accuracy wouldn’t be very helpful in this situation. This is because a naive classifier that simply always predicts a ‘slight’ label would be 89.43% accurate.

Our first model was a simple LogisticRegression model and it did exactly what a naive classifier would do.

With our first model predicting that all the accidents as slight, our accuracy score is really good, but this would not be a very useful model. Here the number of False Positives (predicting a slight accident when it is actually a fatal/serious accident) is the more costly error to have in this situation. Therefore, F1 is a good metric balancing recall and precision, but we also want to pay close attention to the recall of the minority class (fatal/serious accidents).

Sampling Methods

One technique that is very common when dealing with imbalanced data is to resample the data by oversampling and undersampling. This helps the model train on newly produced synthetic data by either oversampling the minority class or undersampling the majority class.

Get Sabatino Chen’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Note that it is very important that you maintain a test set from the original data in order to accurately judge the performance of the model and avoid any “leakage”. Below are some of the advantages and disadvantages of common sampling methods and they can be written in Python.

Undersampling

Advantages: Can help improve run time and storage problems by reducing the number of training data when dealing with a large amount of data.
Disadvantages: Could be removing potentially important information that could be vital to the performance of a model. Also there is a possibility that the random sample taken is a biased data, causing the inaccurate results.

Oversampling

Advantages: No information loss like in undersampling.
Disadvantages: Increases the chances of overfitting due to replicating the minority class.

SMOTE (Synthetic Minority Oversampling)

Advantages: Helps with the issue of overfitting since the oversampling is with synthetic data instead of duplicating data of the minority class. Also there is no information loss.
Disadvantages: Can result in overlapping of classes and cause additional noise. Also not very effective with high dimensional data.

Results

Here are the results after we incorporated undersampling onto our training data using a Random Forest model. You can see that recall on the minority class (fatal/serious) is much better than the original model which had a recall of 0. Also there are far less False Positives which is what we are trying to minimize.

Conclusion

These are just a few examples of different sampling techniques that can be used when dealing with a heavily imbalanced dataset. Other methods can be used to resample your data such as clustering, bagging, boosting, and many more.

Another way to help deal with imbalance issues is by setting different class weights and probability thresholds for your model. These methods can help by giving more importance to predicting the minority class correctly.

I hope that this blog was helpful and if you would like to take a further look into our Capstone project you can find that here. Thanks for reading and I hope you enjoyed!