Medical AI Research · OOD Evaluation · 10-Model Benchmark
BrainMRI-OOD-10Models
A leakage-aware brain MRI classification case study that compares 10 model families on an external, variable-resolution OOD test set instead of relying on an easy same-distribution split.
Overview
BrainMRI-OOD-10Models is a research-focused binary brain MRI classification benchmark for Tumor versus No Tumor images. Its value is not simply that it trains several deep-learning models; it asks a harder and more realistic question: which models still behave well when the test images come from a different visual distribution than the training pool?
The training side uses a fixed-resolution pool, while the external OOD test set contains MRI images with broader source and resolution variation. That design turns the page from a standard accuracy showcase into a robustness study about dataset shift, augmentation behavior, recall, precision, and the gap between model confidence and real generalization.
This project is presented for education and research. It is not a clinical diagnosis product, and the results should be read as a model-generalization study rather than medical advice.
Fixed-resolution MRI images at 256px and 512px
Variable-resolution MRI images from 190px to 800px
CNN, hybrid, transformer-fusion, and custom MSAF variants
Augmentation and no-augmentation comparisons
Why OOD Matters
Medical-image models can look strong until the data distribution moves.
The README frames the project around external testing because medical imaging metrics can be inflated when train and test samples are too visually or subject-wise similar.
Dataset Shift
A model may learn scanner style, preprocessing artifacts, or dataset-specific texture instead of tumor-relevant structure. External OOD testing asks whether the learned signal survives outside the original pool.
Resolution Mismatch
Training images are fixed around 256px and 512px, while the external test images span 190px to 800px. This deliberately stresses scale handling and preprocessing robustness.
Subject-Leakage Risk
Similar images from the same subject can make a split appear easier than it is. A separate external test set reduces that risk and makes the comparison more defensible.
Key Features
OOD-first evaluation design. The benchmark compares models on 3,500 external MRI images with variable resolution instead of presenting only same-source split metrics.
10-model research benchmark. CNN baselines, hybrid combinations, transformer-fusion variants, and a custom MSAF-EffB0 architecture are measured under one result table.
Metric interpretation beyond accuracy. Accuracy, AUC, F1, recall, precision, and Cohen's kappa are shown together so high AUC can be compared against practical sensitivity on tumor cases.
Augmentation-aware comparison. Augmented and non-augmented runs expose how data perturbation affects generalization under resolution and source shift.
Reproducible research artifacts. The repository includes notebooks, scripts, CSV/TXT outputs, reports, curves, and inference utilities instead of reducing the work to a single screenshot.
Results Summary (OOD test)
The full OOD result table, redesigned as the main research artifact.
The best model is not just the highest AUC model; it balances accuracy, F1, recall, precision, and kappa on the external distribution.
| Rank | Model | Family | Acc | AUC | F1 | Recall | Precision | Kappa | Signal |
|---|---|---|---|---|---|---|---|---|---|
| 1 | My Model / aug | MSAF | 0.908 | 0.988 | 0.901 | 0.822 | 0.998 | 0.817 | Best Overall |
| 2 | Hybrid DN121+EffB0 / aug | Hybrid CNN | 0.861 | 0.967 | 0.841 | 0.726 | 1.000 | 0.723 | Runner-up |
| 3 | Hybrid DN121+EffB0 / no-aug | Hybrid CNN | 0.839 | 0.939 | 0.812 | 0.684 | 1.000 | 0.680 | Strong Baseline |
| 4 | My Model / no-aug | MSAF | 0.805 | 0.936 | 0.764 | 0.618 | 0.999 | 0.613 | No Aug |
| 5 | Hybrid SwinEff / aug | Transformer Hybrid | 0.795 | 0.975 | 0.748 | 0.599 | 0.997 | 0.593 | High AUC |
| 6 | ResNet34 / no-aug | CNN | 0.794 | 0.954 | 0.747 | 0.596 | 0.999 | 0.591 | CNN Baseline |
| 7 | DenseNet121 | CNN | 0.785 | 0.984 | 0.732 | 0.578 | 1.000 | 0.573 | High AUC |
| 8 | ConvNeXt-Tiny | CNN | 0.775 | 0.960 | 0.716 | 0.557 | 1.000 | 0.553 | CNN Baseline |
| 9 | Hybrid SwinEff / no-aug | Transformer Hybrid | 0.745 | 0.956 | 0.665 | 0.498 | 1.000 | 0.494 | No Aug |
| 10 | ResNet50 / no-aug | CNN | 0.719 | 0.962 | 0.619 | 0.448 | 1.000 | 0.444 | High Precision |
| 11 | InceptionV3 / no-aug | CNN | 0.710 | 0.901 | 0.602 | 0.430 | 1.000 | 0.426 | No Aug |
| 12 | EfficientNet-B0 | CNN | 0.693 | 0.903 | 0.568 | 0.397 | 0.997 | 0.392 | Backbone |
| 13 | MobileNetV2 / no-aug | Mobile CNN | 0.639 | 0.889 | 0.450 | 0.290 | 1.000 | 0.286 | Lightweight |
My Model / aug leads on accuracy, F1, recall, and kappa, which makes it the strongest OOD candidate in the README result summary.
DenseNet121 and Hybrid SwinEff / aug show very high AUC, but their lower recall means the ranking changes when missed tumor cases matter.
The augmented MSAF and hybrid runs outperform their no-augmentation variants, suggesting augmentation helped under external resolution shift.
Experiment Design
The benchmark separates the problem into a controlled training pool and a harder external test. This makes the comparison more useful for portfolio review because every model is judged by the same OOD target: can it retain tumor/no-tumor discrimination when image scale, source, and visual conditions no longer mirror training?
Fixed-resolution MRI images at 256px and 512px provide a consistent training regime.
Variable-resolution OOD samples from 190px to 800px create the real stress test.
Accuracy is interpreted alongside AUC, F1, recall, precision, and kappa.
Model Family
ConvNeXt-Tiny, DenseNet121, EfficientNet-B0, InceptionV3, MobileNetV2, ResNet34, and ResNet50 create a grounded comparison set.
DenseNet121 + EfficientNetB0 and Swin-T + EfficientNetB0 test whether feature diversity improves robustness under OOD shift.
MSAF-EffB0 combines EfficientNetB0 features with multi-scale attention, SE recalibration, and softmax-weighted scale selection.
CSV metrics, ROC/AUC curves, training notebooks, error-analysis outputs, a report, and universal single-image inference support reproducibility.
MSAF-EffB0
A custom multi-scale attention model built for the OOD setting.
The custom model uses EfficientNetB0 as the feature backbone, then adds multi-scale attention logic so the network can weigh different feature scales instead of trusting one representation. That is especially useful when external MRI images arrive at different resolutions and visual preprocessing histories.
Combines feature views so local texture and broader structural cues can both contribute.
Squeeze-and-Excitation blocks emphasize informative channels and suppress weaker responses.
Softmax scale weighting helps the model select which representation should dominate.
The augmented run shows the strongest OOD result, suggesting better robustness to shift.
Inference Path
The repository also includes a universal single-image inference path so trained weights can be exercised outside the notebooks. That matters because a benchmark is more useful when model artifacts can be loaded, inspected, and run in a consistent script.
One image can be passed through trained model weights without rebuilding the notebook flow.
The inference utility can run on CPU or CUDA depending on the available environment.
Threshold and input-size parameters can be adjusted for controlled inference checks.
Model metadata helps the script restore the correct architecture and preprocessing setup.
Engineering Decisions
External testing over easy split metrics. The evaluation avoids leaning only on same-source train/test splits, which can make medical imaging performance look stronger than it is.
Benchmark-first model comparison. The custom architecture is presented against many baselines, making its advantage easier to judge and harder to overstate.
Recall and kappa are treated as first-class signals. In a medical context, a high AUC model with weak recall can be less compelling than a model with stronger tumor sensitivity and agreement.
Reportable research artifacts. Results are stored as structured outputs, plots, scripts, notebooks, and a technical report so the work reads like a research study rather than a demo page.
My Contribution
I designed the OOD evaluation protocol, trained and compared the model family, developed the custom MSAF-EffB0 architecture, organized the result artifacts, prepared the technical report, and built a reusable single-image inference path. The project strengthened my research workflow: dataset reasoning, baseline comparison, metric interpretation, error analysis, and transparent reporting.
Technology Stack