Proper way to handle highly imbalanced data - binary classification [closed]

Question

I have a really large dataset with 60 million rows and 11 features. It is highly imbalanced dataset, 20:1 (signal:background). As I saw, there are two ways to tackle this problem:

First: Under-sampling/Oversampling.
I have two problems/questions in this way. If I make under-sampling before train test split, I am losing a lot of data. But more important, If I train a model on a balanced dataset, I am losing information about the frequency of my signal data(let's say the frequency of benign tumor over malignant), and because model is trained on and evaluated, model will perform well. But if sometime in the future I am going to try my model on new data, it will bad perform because real data is imbalanced.

If I made undersampling after train test split, my model will underfit because it will be trained on balanced data but validated/tested on imbalanced.

Second - class weight penalty Can I use class weight penalty for XBG, Random Forest, Logistic Regression?

So, I am looking for an explanation and idea for a way of work on this kind of problem.

Answer 1

The above issue is pretty common when dealing with medical datasets and other types of fault detection where one of the classes (ill-effect) is always under-represented.

The best way to tackle this is to generate folds and apply cross validation. The folds should be generated in a way to balance the classes in each fold. In your case this creates 20 folds, each has the same under-represented class and a different fraction of the over-represented class.

Generating balanced folds and using cross validation also results in a better generalised and robust model. In your case, 20 folds might seem to harsh, so you can possibly create 10 folds each with a 2:1 class ratio.