Does fancy hyperparameter search beat a humble grid? A bake-off on 8 datasets
Part two of a series putting common ML habits to the test on 8 imbalanced datasets. The habit here: reaching for Optuna “because Bayesian search must beat a grid.” I put grid, random, TPE and CMA-ES — and the multi-fidelity crowd (Hyperband, BOHB) plus a Gaussian- Process Bayesian sampler — head to head, and measured not just accuracy but time.
The starting point (the same across the whole series)
The shared rig, briefly:
- 8 imbalanced binary datasets, metric Average Precision (PR-AUC).
- A fixed, shared test set touched once. Base learner: LightGBM with early stopping; per-fold preprocessing, no leakage.
- Search runs in the E2 framework — the one Part 1 endorsed: cross-validation over the whole development pool, and each configuration is ranked by its out-of-fold (OOF) AP (no separate validation set; the OOF picks both the number of trees and the winning config), and the winner is scored on the held-out test set. On 8 representative datasets at 30% and 100% of dev, 2 seeds.
- The rule that governs this article: judge at equal compute, not equal config count — because, as we’ll see, “same number of configs” is emphatically not “same time.”
- Every p-value is a paired t-test on the same splits. In plain terms: I compare two strategies case by case on the exact same train/test partitions, then ask whether one systematically beats the other. Pairing on identical data cancels out the “some datasets are just easier” noise, so a small p-value means the winner won on merit — not because it drew a luckier split. (Rule of thumb: p = 0.01 means a gap this consistent would happen by chance about 1 time in 100.)
The contenders (and the one thing that separates them)
The key difference between search methods is whether the method learns from its own results as it goes:
- Grid search — the baseline. Hand-pick a list of configurations up front and try them all. Never learns; every config is decided before you see a single result. Systematic, dead simple, embarrassingly parallel.
- Random search — same “no learning”, but samples configurations at random from the ranges you give. Sounds naïve, yet it’s a famously strong baseline: it spreads its budget across the space instead of over a rigid lattice.
- TPE (Optuna’s Bayesian search) — the first that learns: it models which regions tend to give good vs bad scores and keeps sampling where things look promising.
- CMA-ES (an evolution strategy) — also learns, differently: it keeps a Gaussian “cloud” of configs and shifts and shrinks it toward the best performers, generation by generation.
Then the multi-fidelity crowd, popular in Kaggle and AutoML circles, which learns and saves time by killing bad configs early on a cheap “fidelity” (here, few boosting rounds):
- Hyperband — races many configs on tiny budgets, keeps the best half, doubles their budget, repeats. Random sampling + aggressive early-stopping.
- BOHB — Hyperband’s early-stopping schedule, but the configs are proposed by Bayesian (TPE) sampling instead of at random. “Best of both worlds” on paper.
- GP Bayesian — Bayesian optimization with a Gaussian-Process surrogate, the method the tree-boosting literature often reports as the strongest full-budget searcher.
Finding 1 — nobody reliably beats the grid (and the blind ones lose)
Across budgets the seven methods sit within ~0.01 AP of each other, but there’s a subtle pattern worth seeing. At the smaller budgets (20–80 configs) BOHB posts the best AP — smart sampling plus early pruning finds good configs with fewer tries, so it shines when you can only afford a handful. But its edge is tiny (≤0.006) and never significant. Give everyone the full budget (160 configs) and the plain grid pulls level and nudges ahead (0.707, the top score) — the fancy searchers have just caught up to where the grid already was.
Get Alejandro Gómez Ruiz’s stories in your inbox
Join Medium for free to get updates from this writer.
And at that full budget the only significant gaps are the methods that trail the grid: plain random search −0.0104 (p = 0.02) and GP-Bayesian −0.0080 (p = 0.04); TPE is borderline (−0.0048, p = 0.07); CMA-ES, Hyperband and BOHB tie it. So the honest verdict: no search method reliably beats a sensible grid on accuracy — the clever ones merely reach the same place, and the blind ones fall behind. And more budget barely helps — 20→160 configs moves AP by a few thousandths at most.
Finding 2 — “same number of configs” is not “same compute”
Time doesn’t rescue the search methods either. On the same 160 configs, the full-budget methods land in a modest band (TPE 182s, GP 196s, grid 199s, CMA-ES 218s), with random search the slowest (260s) — fitting, since it’s also the worst on AP: the true worst of both worlds. The grid fixes a low learning_rate (many boosting rounds), while the learning samplers drift toward faster-converging configs — but the spread is small, and none of it buys better AP.
Finding 3 — the multi-fidelity “speedup” is a mirage under honest CV
This is the twist, and it ties the whole series together. Hyperband and BOHB are supposed to be the time win: train each config on a few trees, kill the losers early, only grow the survivors. And with a single held-out validation set (E1) they absolutely are — I ran the same search that way too, and pruning on that one cheap signal makes them about 20× faster than the grid at the same AP. That’s exactly why multi-fidelity is famous.
But Part 1 concluded you should select by cross-validation (OOF), not a separate val — and that’s what we do here. Under honest CV the cheap trick inverts: to prune on the out-of-fold signal you have to train all K folds at each rung, so instead of ~20× faster, Hyperband and BOHB come out ~1.5× slower than the grid — for no gain in AP. (A note on reading times: I don’t compare E1 seconds against E2 seconds — those runs happened at different times under different machine load — but within each framework, where the grid and the multi-fidelity methods run back-to-back on the same cell, so each method’s speed is measured against its own framework’s grid.) The famous speedup was an artifact of the very habit the series argues against; hold yourself to consistent CV selection and it evaporates.
The moral
Under honest cross-validated selection, a small sensible LightGBM grid is the best deal on both axes at once — the highest AP and among the fastest to run. Bayesian, evolutionary and multi-fidelity search don’t beat it on accuracy (and random and GP are significantly worse), while multi-fidelity’s celebrated time savings vanish the moment you select by CV instead of a held-out set.
- Grid: best AP, cheap. Hard to beat, easy to run.
- Random / GP: significantly worse AP. Blind or Bayesian, they lose here.
- Hyperband / BOHB: no AP gain and ~1.5× slower than the grid under CV — the famous ~20× speedup only exists with a held-out val, not with honest out-of-fold pruning.
So don’t reach for fancy hyperparameter search to squeeze more out of a well-chosen grid — on this problem it doesn’t help and can hurt. Save it for spaces a grid genuinely can’t cover.
Does this contradict the textbooks?
Not really — and it’s worth being clear about why, so nobody reads this as “Bayesian optimization is useless.” The literature is actually consistent on when fancy search pays off: when each evaluation is expensive or the space is large (deep nets, huge models, dozens of knobs). For a cheap model tuned over a small, sensible grid — exactly the regime here — the gains shrink into the noise. Even the classic “random beats grid” result ([Bergstra & Bengio, 2012](https://jmlr.org/papers/v13/bergstra12a.html)) is about dense grids that squander budget on knobs that don’t matter; a small grid hand-focused on the few that do sidesteps it. And multi-fidelity (Hyperband/BOHB) was built to prune expensive low-fidelity runs cheaply — a bargain that quietly disappears once “one evaluation” means a full cross-validation. So read this narrowly: on cheap tabular gradient boosting with a modest budget, a sensible grid is hard to beat. Scale up to expensive models or big search spaces and the clever methods are back in their element.
Next in the series: the ensemble myth — is “a thousand LightGBM configs” really faster than building an ensemble? Code and all the numbers are in the study repo; every p-value is a paired t-test on the same splits.
