Data Leakage Laboratory

Six common pipeline shortcuts, one honest baseline, and the optimism each shortcut buys you.

leakage
partitioning
study design
pipeline

Honest AUC (strict)

0.889

Selection leakage AUC

0.886

Optimism

-0.003

Optimism by leak type

ROC: strict protocol vs. the selected leak

Strict protocolSelection leakage

Leak catalogue

Leak	Reported AUC	Optimism	Mechanism
Strict protocol (no leakage)	0.889	0.000	Every data-dependent step is refit inside the training split; subjects appear in one split only.
Subject-level leakage	0.775	-0.114	Multiple slices from one patient are split across train and test, so the model can memorize the patient.
Preprocessing leakage	0.889	0.000	Scaling and imputation statistics are computed on the pooled train + test data.
Selection leakage	0.886	-0.003	Feature ranking uses all labels before the split is made.
Temporal leakage	0.977	0.088	A post-outcome variable is included among the pre-operative predictors.
Duplicate leakage	0.775	-0.114	Near-identical repeat scans are treated as independent observations.
Target leakage	1.000	0.111	A predictor is a deterministic proxy for the label by construction.

Detection is a design question

None of these leaks announce themselves in the model output — every one of them produces a well-behaved ROC curve. They are visible only in the protocol, which is why the methods section, not the results section, is where a reader should look for them.

Preparing the simulation…