Nested vs. Non-Nested Cross-Validation

Select features on pure noise and watch non-nested cross-validation manufacture a high AUC out of nothing.

cross-validation
leakage
feature selection
simulation

Non-nested mean AUC

0.780

optimistic

Nested mean AUC

0.506

honest

Optimism

0.274

Nested SD

0.079

Sampling distribution of the cross-validated AUC

Non-nested (selection outside CV)Nested (selection inside each fold)

Protocol comparison

Both protocols use the same data, the same model and the same folds. Only the position of the feature-selection step differs.
Protocol	Mean AUC	SD	Pr(AUC > 0.7)
Non-nested	0.780	0.034	1.000
Nested	0.506	0.079	0.000

The only difference is one line of code

Non-nested selection ranks features using all labels before the folds are cut. Nested selection re-ranks inside every training fold. With the signal slider at zero the correct answer is 0.50 — any excess in the orange distribution was manufactured by the pipeline.

Preparing the simulation…