Data Leakage Laboratory
Six common pipeline shortcuts, one honest baseline, and the optimism each shortcut buys you.
- leakage
- partitioning
- study design
- pipeline
Honest AUC (strict)
0.889
Selection leakage AUC
0.886
Optimism
-0.003
Optimism by leak type
ROC: strict protocol vs. the selected leak
Strict protocolSelection leakage
Leak catalogue
| Leak | Reported AUC | Optimism | Mechanism |
|---|---|---|---|
| Strict protocol (no leakage) | 0.889 | 0.000 | Every data-dependent step is refit inside the training split; subjects appear in one split only. |
| Subject-level leakage | 0.775 | -0.114 | Multiple slices from one patient are split across train and test, so the model can memorize the patient. |
| Preprocessing leakage | 0.889 | 0.000 | Scaling and imputation statistics are computed on the pooled train + test data. |
| Selection leakage | 0.886 | -0.003 | Feature ranking uses all labels before the split is made. |
| Temporal leakage | 0.977 | 0.088 | A post-outcome variable is included among the pre-operative predictors. |
| Duplicate leakage | 0.775 | -0.114 | Near-identical repeat scans are treated as independent observations. |
| Target leakage | 1.000 | 0.111 | A predictor is a deterministic proxy for the label by construction. |
Detection is a design question
None of these leaks announce themselves in the model output — every one of them produces a well-behaved ROC curve. They are visible only in the protocol, which is why the methods section, not the results section, is where a reader should look for them.