I trained a model that looked great on every metric — so I built the evaluation that proved it wasn't.

The moment I stopped trusting my numbers

Virtual contrast-enhanced CT is a genuinely useful idea: roughly 15% of patients can’t receive iodine contrast (kidney disease, allergy), so a model that synthesizes a contrast scan from a plain one could spare them a second, contrast-loaded acquisition. The clinical team handed me paired plain and contrast scans and asked me to train a synthesis model.

I did. The loss curves were beautiful. SSIM climbed to 0.88. By every standard metric, the model was learning.

Then I opened a few predictions next to the ground truth, and the anatomy didn’t match. The model wasn’t synthesizing contrast into the right slice. It was painting contrast onto the wrong part of the body and still scoring well, because the metric was comparing the wrong things.

What I found: the slices were shifted

The paired scans weren’t aligned. A plain CT and a contrast CT of the same patient are two separate acquisitions, and they start at different positions along the body. The original pairing was done by array index, which assumed they didn’t — and across the dataset that assumption was off by an average of 37 slices, roughly 185 mm. So “slice 47 of the plain scan” and “slice 47 of the contrast scan” were nowhere near the same anatomy, and SSIM rewarded the model for mapping one to the other anyway, because SSIM measures structural similarity, not whether you’re looking at the same organ.

The alignment algorithm

1. Resample both volumes to a common voxel grid, so coordinates mean the same thing in both.
2. Slide one volume along the z-axis and compute normalized cross-correlation at each offset — the offset with the highest correlation is the true alignment.
3. Gate on correlation: pairs below NCC 0.9 are dropped.
4. Confirm by eye: render the aligned slices side by side and check the anatomy actually matches.

Step 4 is a manual step, and it stays manual on purpose. Numbers lie here — that’s the whole lesson of this project — and I don’t have an automated check I trust to catch “mathematically aligned, anatomically wrong.”

There used to be a second automated gate, on how much the two scans overlapped. I turned it off. It had been running on an undocumented default threshold nobody had chosen deliberately, and when I went looking for a defensible value in the literature, there wasn’t one. An arbitrary threshold that silently drops patients is worse than no threshold.

Building an evaluation that could say no

The first dataset was small: 17 patients and 55,639 slices as delivered, but only a handful of pairs survived alignment. Too small to conclude anything — every result came back “n is too small to tell.” In August the hospital delivered a much larger cohort: 52 patients, 49 with an arterial phase. That was finally enough to answer the question properly.

So I built the evaluation I’d been unable to run: a patient-level three-way split, five-fold. Train, validate, and test are disjoint by patient, not by slice — a patient’s slices never straddle the boundary. The validation set picks the checkpoint, which means only the test set is genuinely untouched, and it’s evaluated exactly once.

The result

The model does not work. Not “we can’t tell yet” — the confidence intervals are ±0.02 wide and nowhere near the bar. This is the project’s first generalization number precise enough to support a conclusion, and the conclusion is negative.

The diagnosis is specific, which is what makes it actionable: the model learned to raise brightness roughly uniformly across the body rather than to concentrate contrast where vessels are. Measured as the ratio between enhancement in high-uptake regions and enhancement overall, the ground truth is 4.5× in the chest and 4.6× in the abdomen; the synthesis is 2.0× and 1.0× — meaning that in the abdomen, the model enhances the aorta and the liver by the same amount.

Where things stand

• The current work is a systematic ablation over the loss function, chasing that under-enhancement — an air-region penalty and a residual-connection architecture have both already been tried and judged failures.
• I also ran a membership-inference audit to check whether the model memorized training patients. On the pixel-level metrics it came back AUC 0.70–0.80, leaning unfavourable; on the feature-level ones the interval still covers chance, so those say nothing either way. With only five clusters the intervals aren’t trustworthy anyway — which is exactly why the project has a standing rule against writing this up as “no evidence of memorization.”
• Deployment is not on the table. The data is under hospital NDA, so there’s no public repo.

The honest headline of this project isn’t a model. It’s that I built the evaluation strict enough to rule out my own work, and reported the result it gave me.