Reproducibility Auditor

Audit five hypothetical excerpts from medical imaging AI papers against a reproducibility checklist and mark each 'Rigorous' or a 'Red Flag'.

reproducibility
data leakage
checklist
peer review

Excerpt 1

To increase our sample size from 50 to 5000, we extracted 100 patches per patient. We split the patches randomly into 80% train and 20% test sets.

Your audit

Excerpt 2

We monitored validation loss and halted training when it stopped improving for 10 epochs. All subsequent metrics (AUC, Dice) were reported on the locked test set.

Your audit

Excerpt 3

To handle class imbalance (90% normal), we applied random oversampling to the test set so it contained 50% abnormal cases. We report AUC on this balanced test set.

Your audit

Excerpt 4

We applied random horizontal flips to all training images. The model is used to detect left vs. right kidney tumors.

Your audit

Excerpt 5

Our preprocessing pipeline resampled all scans to 1x1x1mm, clipped intensities to [-100, 200] HU, and normalized to zero mean. We provide this code on GitHub.

Your audit

Weights are not a model

A trained weight file without the full pipeline — preprocessing, augmentation, architecture, configuration, and data splits — is not reproducible. The reproducibility crisis in biomedical AI is driven less by fraud than by undocumented pipelines: missing seeds, unspecified preprocessing, and silent test-set leakage.