The Speed-Reader vs. The Method Actor of Feature Selection
This blog is about a choice almost every machine learning task requires and nobody questions: which feature selection technique fits your dataset and problem. There are two competing families — the filter (speed reader) and the wrapper (method actor). How do we decide which is better? This blog gives you the tools to pick the right one, by running both and reporting what actually happened.
Any ML online project will have this line as the template — ‘filter is faster, wrapper is better’. But are they? That’s what we’re finding out here.
Take an example. Say 1000 resumes land on your desk with twenty minutes to shortlist a few. First approach: score each resume individually against a checklist — simple and fast. But a checklist alone misses patterns, so try the second approach — pull out a handful, analyse them as a team, swap one out, analyse again, until you reach a set that works together. Slower, a lot slower. Extend this to a dataset with many features, and it makes sense why we want to understand which one to pick.
This disagreement between filter and wrapper is what we’re exploring. In one line, feature selection means deciding which original features are worth keeping, so we can eliminate noise and redundancy. Scikit-learn offers filter methods (mutual information, chi-squared, ANOVA F), wrapper methods (RFE, sequential selection), and embedded methods. Embedded method offers feature importance as a byproduct of training a model, like a decision tree’s feature importance score.
Now, the dataset: scikit-learn’s digits dataset — 1797 handwritten digit images (8x8 each), flattened to 64 features each, across 10 classes. Enough features that picking 15 of 64 is a real decision, small enough that both methods finish in seconds, so we can run this repeatedly for this comparative study.
NOTE: I have included only important parts of the code, so as not to clutter the flow of this blog. But if at all you need to see the full notebook, I have hosted it here — GitHub link.
# importing the digits dataset with 1797 handwritten digit images,
# each flattened into 64 pixel features, across 10 classes (0-9).
data = load_digits()
X, y = data.data, data.target
print("No of rows:", X.shape[0])
print("No of features:", X.shape[1])
# splitting the dataset
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=0, stratify=y)
# scaling is needed for the SVM inside RFE, as it is scale sensitive
scaler = StandardScaler().fit(X_train)
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)
# number of features each method is allowed to pick out of 64
K = 15
# output
# No of rows: 1797
# No of features: 64Meet the speed reader
Mutual information (MI) evaluates each feature independently, calculating how much knowing this feature will reduce the uncertainty about the target label. if you are more interested in the details, check out the scikit-learn API for this. But for the scope of this blogs, this much is fine. Implementation of mutual information is given below. We used this along with the SelectKBest, which will, based on the score returned by MI will retain only the top k.
# speed reader: filter method using mutual information
# to get an estimate of the run time this method takes, we use the %timeit
filter_timing = %timeit -o SelectKBest(mutual_info_classif, k=K).fit(X_train, y_train)
filter_selector = SelectKBest(mutual_info_classif, k=K).fit(X_train, y_train)
selected_filter = filter_selector.get_support(indices=True)
mi_scores = filter_selector.scores_
print(f"Selected feature indices: {sorted(selected_filter.tolist())}")424 ms ± 2.57 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Selected feature indices: [2, 10, 20, 21, 26, 28, 30, 33, 34, 36, 38, 42, 43, 58, 61]
Here, we can see the score for each feature computed using the MI. The time it took to run this, averaged across several runs, is 424 ms. Lets keep that in mind for comparison with the next method.
Meet the method actor
For the method actor part, we use Recursive feature Elimination (RFE) for comparison. RFE starts by fitting a model on all your features and scoring each one using its built in weights or importance metrics. It then repeatedly drops the lowest ranked features and retrains the model on the remaining set in a continuous pruning loop. This process keeps repeating until it hits your target feature count, leaving behind the best performing subset.
# method actor: wrapper method using Recursive Feature Elimination
wrapper_timing = %timeit -o RFE(SVC(kernel='linear'), n_features_to_select=K).fit(X_train, y_train)
wrapper_selector = RFE(SVC(kernel='linear'), n_features_to_select=K).fit(X_train, y_train)
selected_wrapper = np.where(wrapper_selector.support_)[0]
print(f"Selected feature indices: {sorted(selected_wrapper.tolist())}")897 ms ± 11.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Selected feature indices: [4, 5, 10, 19, 21, 26, 27, 28, 30, 42, 43, 44, 45, 53, 58]
We can see from the image that accuracy stays flat, almost saturated, for a good stretch as features get eliminated, which means even a much smaller subset gets you essentially the same performance as the full set. It’s only past a certain point that accuracy actually starts dropping. So the real decision isn’t “fewer features hurt”, it’s finding that cutoff point before the drop begins.
The Actual Audition
Till now, we have only implemented the two methods and checked their run time speeds. But is that enough to determine the quality of their output, no. So to test the quality of their output, which is a subset of features, we are going to train a same classifier model, and scored on a held out test data, to compare their accuracy.
# accuracy comparison on the selected subset of features from both methods
clf_filter = SVC(kernel='linear').fit(X_train[:, selected_filter], y_train)
filter_acc = clf_filter.score(X_test[:, selected_filter], y_test)
clf_wrapper = SVC(kernel='linear').fit(X_train[:, selected_wrapper], y_train)
wrapper_acc = clf_wrapper.score(X_test[:, selected_wrapper], y_test)
print(f"Filter accuracy: {filter_acc:.1%} (avg selection time {filter_timing.average*1000:.0f} ms)")
print(f"Wrapper accuracy: {wrapper_acc:.1%} (avg selection time {wrapper_timing.average*1000:.0f} ms)")
print(f"Speedup: {wrapper_timing.average / filter_timing.average:.2f}x")Filter accuracy: 91.1% (avg selection time 424 ms)
Wrapper accuracy: 93.1% (avg selection time 897 ms)
Speedup: 2.12x
As of now we can tell that the speed reader is 2.12x faster than the method actor, even in this dataset of only 64 features. We can now understand the time implications of these two, when the features and the dataset size are even higher.
Get Thillaiambalam Arumugam’s stories in your inbox
Join Medium for free to get updates from this writer.
The numbers here agrees with the popular saying — filter is faster, wrapper is better. But are we going to buy this claim, just from accuracy and the run time speed, absolutely no. We are going to put them to test and see which one is more stable across multiple runs, which is essential to identify whether the results are just a one time fluke or a solid one.
The Overlap Check
Accuracy and run time aren’t the only things worth comparing — do these two methods even agree on which features matter, or are they just arriving at similar scores by picking entirely different things?
overlap = len(set(selected_filter) & set(selected_wrapper))
print(f"Feature overlap: {overlap}/{K}") # Feature overlap: 8/15Turns out, not really. The overlap between the two selected subsets is just nearly half (8 of 15). Same data, same target count, and they still can’t agree on nearly half their picks. That is enough reason to check whether either method is even consistent with itself, before trusting either one’s picks.
The Stability Check
# using CV to test the methods across different folds for stability, accuracy
from sklearn.model_selection import StratifiedKFold
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
filt_sets, wrap_sets = [], [] # to hold the selected features per fold
filt_results, wrap_results = [], [] # per fold time and accuracy
for tr_idx, te_idx in skf.split(X, y):
Xf, Xv = scaler.fit_transform(X[tr_idx]), scaler.transform(X[te_idx])
yf, yv = y[tr_idx], y[te_idx]
t0 = time.time()
f_sel = SelectKBest(mutual_info_classif, k=K).fit(Xf, yf)
f_time = time.time() - t0
f_idx = f_sel.get_support(indices=True)
f_acc = SVC(kernel='linear').fit(Xf[:, f_idx], yf).score(Xv[:, f_idx], yv)
filt_sets.append(set(f_idx.tolist()))
filt_results.append((f_time, f_acc))
t0 = time.time()
w_sel = RFE(SVC(kernel='linear'), n_features_to_select=K).fit(Xf, yf)
w_time = time.time() - t0
w_idx = np.where(w_sel.support_)[0]
w_acc = SVC(kernel='linear').fit(Xf[:, w_idx], yf).score(Xv[:, w_idx], yv)
wrap_sets.append(set(w_idx.tolist()))
wrap_results.append((w_time, w_acc))
def avg_jaccard(sets):
vals = [len(sets[i]&sets[j])/len(sets[i]|sets[j]) for i in range(len(sets)) for j in range(i+1, len(sets))]
return sum(vals)/len(vals)
print(f"Filter stability: {avg_jaccard(filt_sets):.2f}") # 0.81
print(f"Wrapper stability: {avg_jaccard(wrap_sets):.2f}") # 0.52What this code is doing: we split the data into 5 folds and re-run both methods fresh on each one, recording two things per fold — the selected feature set (for stability scoring) and the time, accuracy pair (for the scatter plot below). avg_jaccard averages how much any two folds feature sets overlap with, 1.0 means every fold agreed, 0 means none did.
We can see from the jacard score and the heatmap, the speed reader barely moves, while the method actor moves a lot. This is because, the filter method’s final subset never depends on the whole, rather how much each individual feature contributes. That’s why the stability score is good at 0.81, meaning 81% agreement on subset features across folds. The wrapper opinion is relative by nature, it weighs features against each other rather than in isolation and since the wrapper uses all the remaining features together to decide who gets cut each round, its subset depends heavily on exactly which data was passed to it for that particular fold, and that’s why there is only a 52% agreement across folds.
Now we have all three pieces on the table — run time, accuracy, and fold-to-fold stability, the scatter plot above is where they all come together. Notice that even with its lower stability score, the method actor is still consistently landing above the speed reader on accuracy, fold after fold. That’s telling: a lower agreement score doesn’t mean the wrapper is picking bad or random subsets, it means it’s picking a different subset for each fold, tailored to whatever data it was given at that time, and that adaptability is exactly what is pushing its accuracy higher each time, even though the actual features chosen keep shifting underneath.
Also worth noticing: within each method, accuracy itself isn’t a single fixed number and it moves from fold to fold. We can see the vertical spread of accuracy within each method across each fold. And yet, across that entire spread, the method actor still comes out ahead of the speed reader on every fold. That spread within each method, means there’s real room to optimize within each method itself. The subset a method gives you on its first run isn’t necessarily the best it’s capable of.
The Winner
Now that we have all the numbers estimated, can we conclude which one is the winner. Well, its actually a bit more hard to choose one winner, based on this dataset context alone. But from the numbers we can conclude some important things.
- If speed or prototyping faster is our priority — go for the filter method, and this will give a huge speed gain compared to wrapper at a slight overall performance cost
- If accuracy is what we aim for — the wrapper wins here, and with a much bigger dataset, this wrapper method of trying to arrive at a subset, is much more rewarding in performance metrics, yet run time takes a huge hit
- Whichever method you pick, don’t settle for its first subset — accuracy varies fold to fold, so rerun a few times to get the best performing one
Conclusion
As seen, there is no universal winner here, and treating this like a competition kind of misses the whole point of data science, which is exploring the options we have and letting the actual results decide, not choosing a favourite before even looking at the numbers. The real takeaway is not about picking a favourite, its about figuring out what you’re lacking before you decide.
GitHub link to the full notebook:
https://github.com/thillaiambalam15/feature_selection-speed-reader-vs-method-actor
Acknowledgement
This blog came out of the feature selection part taught at Imarticus. After learning it, I wanted to explore further and see how far I could push what was taught in class, to actually test the ideas instead of just accepting them. Thanks to Imarticus for the strong foundation and support that made it possible for me to go explore this on my own. Grateful for the environment Imarticus builds around learning, one that pushes you to go beyond the classroom and actually try things out for myself.
References
- Feature selection overview by scikit-learn
- Mutual information by scikit-learn
- SelectKBest by scikit-learn
- Recursive feature elimination by scikit-learn
My LinkedIn: https://linkedin.com/in/thillaiambalam
