I trained a classifier to read analog clocks. On the benchmark, it scores 99.82%. On photographs of real clocks it scores 2.2%, against a chance rate of 0.7%.
This post is the diagnosis. It is organized around four hypotheses for the gap — rotation, calibration, augmentation realism, and training data — three of which I falsified. Along the way, the benchmark number itself turned out to be softer than it looked: a label-ordering trap that silently costs 33 points, a resize-filter dependency that manufactures a fake failure mode and reverses which checkpoint wins, and two rendering defects in the dataset.
Every number below was measured. Where a result is statistically weak, I say so with the test statistic, including for the one hypothesis that worked.
TL;DR of the findings:
A 12-hour dial quantized to 5-minute increments gives 144 classes. Class i is 5·i minutes past 12, so the label space is a discrete circle: class 143 (11:55) and class 0 (12:00) are adjacent, but a flat softmax has no idea. That mismatch shows up repeatedly later.
The dataset is gpiosenka/time-image-datasetclassification (CC0): 14,400 synthetic 224×224 RGB renders, exactly 100 per class, split 80/10/10 into 11,520 / 1,440 / 1,440. It bundles a pretrained EfficientNet-B3 checkpoint, time-99.68.h5.
Identical for every run in this post:
EfficientNetB3 (ImageNet init, fully unfrozen)
→ GlobalMaxPooling2D
→ BatchNormalization
→ Dense(256, relu)
→ Dropout(0.3)
→ Dense(144, softmax)
11,220,159 parameters. Optimizer Adamax @ 1e-3, loss categorical cross-entropy, batch size 32. ReduceLROnPlateau(factor=0.5, patience=2) on val_loss; EarlyStopping(monitor="val_accuracy", patience=12, restore_best_weights=True) against a 120-epoch budget. Runs early-stop between epochs 32 and 42.
Two non-obvious implementation details, both of which cost me time:
Input is 150×150, not 224×224. The images are stored at 22⁴² and every one gets downsampled at load. Section 2b is entirely about what that innocuous step does.
Preprocessing lives inside the model. Rescaling(1/255) and Normalization are layers, so the model takes raw floats [0, 255]. Dividing by 255 yourself out of habit feeds it , [0, 0.004] and accuracy collapses silently.
Augmentation is keras.Sequential applied to the raw-valued training stream:
augment = keras.Sequential([
keras.layers.RandomRotation(rotation_factor, fill_mode="nearest"),
keras.layers.RandomZoom(0.05, fill_mode="nearest"),
keras.layers.RandomTranslation(0.05, 0.05, fill_mode="nearest"),
], name="augment")
rotation_factor is a fraction of 2π, so 0.15 = ±54°. It is the one knob that varies across the checkpoints in section 4.
Hardware: one A100-SXM4–40GB or H100 per run (Slurm, dgxa100 partition, 10 CPUs, 64 GB). A full training run is ~15 minutes; the four-model rotation sweep was 1 h 11 m wall-clock, the real-data experiment 2 h 16 m.
Top-1 / top-5 are the usual thing. But a 144-way argmax hides the structure of the errors, because being one class off and being six hours off are both “wrong.” So the primary error metric is circular minutes-off: map each class to minutes on a 720-minute dial and take the shorter arc.
def label_to_minutes(label): # "10-35" -> 635
h, m = label.split("-")
return (int(h) % 12) * 60 + int(m)
def circular_error(a, b, period=720):
d = abs(a - b) % period
return min(d, period - d) # shorter way round the dial
For calibration, I use ECE over 15 equal-width bins of the max-softmax confidence,
ECE = Σ_b (n_b / N) · |acc(b) − conf(b)|,
with MCE the max of that gap, plus NLL and Brier. Temperature scaling fits a single scalar T by minimizing NLL on the validation split (softmax(log p / T), re-normalized), and selective prediction reports accuracy versus coverage as a confidence threshold sweeps upward.
For attention, I use Grad-CAM at top_activation and summarise each heatmap by its focus: the fraction of total CAM mass in the hottest 10% of pixels. A concentrated heatmap means the model committed to a region; a diffuse one means it never found the hands.
Before diagnosing a domain gap it is worth establishing that the source-domain number means what you think. Three things said otherwise.
A Keras classifier’s output unit i means whatever class sat at index i during training. This dataset offers two plausible ways to build that list, and they disagree:
# (A) alphabetical over the UNDERSCORE labels in clocks.csv — how it was trained
order_A = [l.replace("_", "-") for l in sorted(clocks["labels"].unique())]
# ['10-00', '10-05', ..., '1-00', ..., '9-55']
# (B) alphabetical over the HYPHENATED directory names - the intuitive choice
order_B = sorted(d.name for d in (DATA_DIR / "train").iterdir() if d.is_dir())
# ['1-00', '1-05', ..., '10-00', ..., '9-55']
The cause is ASCII: - is 45 and _ is 95, so the hyphen sorts before digits and the underscore after. Whole hours shift.
Nothing crashes. image_dataset_from_directory accepts B without complaint, training converges, the loss curve looks healthy. You lose a third of your accuracy to an hour shift, and the only tell is the suspiciously clean "every error is an exact multiple of 60 minutes" pattern — which is precisely why circular-error binning, not top-1, is the metric that catches it.
The CSV’s class index column looks like an authoritative third option. It is assigned in exactly the directory order, so it is the same trap wearing a different hat.
Fix: pass class_names= explicitly, derived from the labels column.
This is the one I did not see coming.
224 → 150 downsampling happens on every image, and nothing in the pipeline pins the interpolation method. I swapped bilinear for nearest-neighbor — same weights, same files, same labels — and re-ran the full 1,440-image test split.
Because both filters are evaluated on the same images, the correct test is McNemar’s exact test on the discordant pairs, not a two-sample proportion test:
Read the released model’s row carefully: 0 / 7. Nearest-neighbor is strictly better — it gets every image bilinear gets, plus seven more. Six of those seven are the 11-10 → 7:55 cluster at p ≈ 0.9 that the dataset's own documentation describes as the model's signature weakness. Under nearest, that failure mode does not exist. It was never a property of the weights.
The cause is mundane. The released checkpoint was trained with Keras 2.8’s ImageDataGenerator, whose flow_from_dataframe defaults to interpolation="nearest". I was evaluating with image_dataset_from_directory, which defaults to bilinear. I had been scoring it on a distribution it never trained on and attributing the difference to the model. Clock hands are one-pixel-wide, high-frequency, near-aliasing structure — exactly what a resampling kernel mangles.
And it is symmetric: my own model, trained through the bilinear path, degrades by 1.7 points under nearest. Each model wins under the filter it was trained with, and the ranking between them flips on a choice neither of us wrote down.
What this rules out, and what it doesn’t. It does not overturn the checkpoint comparison within my own bilinear pipeline — that comparison is internally consistent. It does mean any cross-pipeline comparison is partly a measurement of the pipelines, and that the 0.9938-vs-0.9979 gap I had been quoting is inflated: under its own preprocessing the released model is at 0.9986.
Grad-CAM is usually where you confirm the model looks at the right thing. I expected diffuse attention on the errors and got the opposite: tight, confident heatmaps on the hand tips, on images the model was supposedly failing.
So I recomputed the signed offset rather than the circular distance. On my own checkpoint, all 13 non-blank dataset-wide errors landed on exactly ±195 minutes — 3 h 15 m, no fuzz, no near-misses. 11:10 → 7:55 is precisely 11:10 − 3:15.
An exact, repeated, confident offset is a signal to go and look, not a conclusion — and looking turned up two distinct causes:
Both cluster on specific file indices (0, 36, 38, 51, 72) across unrelated classes. Only 1–4 of the 144 files at a given index are affected, but of those, 100% are the exact ±195-minute shift and 0% are any other error type. A model confusion would scatter across magnitudes; a renderer bug does this.
Net of both defects, true accuracy on all 14,400 images is about 99.97%, not 99.82%. Reading failures individually moved the number up.
Every number so far is on synthetic renders: upright, centred, evenly lit, frontal, and set to an exact 5-minute mark. The target distribution violates all five simultaneously.
Protocol. 92 CC0 photographs from kongaskristjan/real-clocks, labelled in the filename to the minute. Because real times are not on the 5-minute grid, ground truth is scored two ways: the true time is rounded to the nearest class for exact-match accuracy, and the unrounded time is used for circular error, so rounding never flatters the model.
def nearest_5min_label(hour, minute):
r = int(round(minute / 5.0)) * 5 # carry rounding into the hour
hour = (hour + r // 60) % 12
return f"{hour if hour else 12}-{r % 60:02d}"
Result.
Chance on 144 classes is 0.7%. A 95% Wilson interval on 2/92 is [0.6%, 7.6%] — so the honest statement is “somewhere between chance and slightly above it.” This is not degradation; it is a different regime.
Ruling out a single systematic bug. The appealing hypothesis is one clean offset — a hand swap, or the ±195-minute defect recurring. Binning circular error says no:
Errors spread across every magnitude. The ~180° “opposite side of the dial” band is present but only ~15% of cases — nowhere near dominant. That is out-of-distribution confusion, not a fixable bug.
Qualitatively, the six best predictions are clean frontal well-lit faces — the photos closest to the synthetic manifold. The six worst are ornate dials, odd framings and reflective glass.
The immediate story: real photos are taken off-axis, synthetic renders are perfectly upright, so the model is rotation-brittle. Widen the augmentation and close the gap.
Setup. Four checkpoints, identical except --rotation-factor ∈ {0.03, 0.06, 0.10, 0.15} (±10.8° … ±54°), all seed 0. Each is swept over test-time rotation from −90° to +90° in 15° steps on 144 test images (one per class), rotating with fill_mode="nearest" to match training rather than introducing black corners that would confound the measurement.
Result — the source-domain part works beautifully.
The robust plateau’s edge tracks the literal training RandomRotation range. And widening it is free: ±54° is simultaneously the most rotation-robust and the most accurate upright checkpoint. No trade-off, so it became the default.
The cliff has sharp structure. Outside the plateau the released model (no rotation aug) scores 0.993 at exactly −90°, 0°, +90° and 0.000 at every angle between — it learned the dial’s 90° symmetry and nothing else. Confidence in that dead zone stays at 0.44–0.88, and Grad-CAM stays sharply locked on the rotated hands. The model has not lost the hands; it finds them and misreads them with conviction. Localisation is intact, decoding is not.
Result — the target-domain part is a flat null.
Fisher exact on 5/92 vs 2/92: p = 0.44. A five-fold widening of rotation robustness bought nothing measurable, and the point estimate moved the wrong way.
What this rules out. Rotation as the dominant axis of the gap. It does not rule out rotation as a contributing factor — with n = 92 the minimum detectable improvement at 80% power is about +10 points, so anything smaller is invisible here. The correct claim is “no large effect,” not “no effect.”
If you cannot make the model accurate, make it know when it is wrong: a selective-prediction system that answers above a confidence threshold and defers the rest to a human. That plan needs calibration to hold under shift.
Setup. Reliability diagrams and ECE (15 bins), temperature scaling fit on synthetic valid, and an accuracy-vs-coverage sweep, run on synthetic test and on the real photos.
In-distribution the calibration is textbook: ECE 0.002, fitted temperature 1.07, essentially nothing to correct.
Out of distribution it inverts. The synthetic-only model’s p ≥ 0.8 predictions on real photos are 0% accurate. Not merely miscalibrated — anti-correlated in the region you would actually gate on. Temperature scaling fit on synthetic valid does not help, because this is domain shift, not a scaling error: one scalar cannot repair a ranking that has inverted.
Interestingly, fine-tuning on real data (section 7) repairs the ranking without repairing the scale: that model is wildly overconfident in absolute terms (ECE 0.48) yet its top ~12% by confidence are 86% correct. For selective prediction, ranking is what you need — but note that 12% of 57 photos is ~7 images, so treat that cell as directional.
What this rules out. Deploying a human-fallback system gated on this model’s confidence. The signal you would gate on points the wrong way exactly where you would trust it most.
Setup. Stack a realism pipeline on top of the rotation augmentation — RandomTranslation(0.12), RandomZoom(-0.2..+0.3), RandomShear(0.15), RandomPerspective(0.35), RandomBrightness(0.3), RandomContrast(0.3), Gaussian blur. This is the appealing move: no data collection, scales to all 144 classes uniformly.
Result. Real top-1 7.0% → 8.8% (4/57 → 5/57; Fisher p = 1.00), at a cost of ~0.8 points synthetic (0.9972 → 0.9889).
One image changed. Photorealism is not a composition of affine and photometric jitter — augmented renders are still line drawings, and the gap is generative (dial art, materials, 3-D perspective, motion blur, specular glare), not affine.
Setup. A second labelled source brings the real pool to 195 photos, split 138 train / 57 test, stratified by hour bucket with seed 0 — per-class stratification is impossible at ~1.4 photos per class. The 57 test photos are never trained on; the manifest builder and the mixing code both filter on split == "train".
The real photos are concatenated into the synthetic stream and oversampled 25×, then fine-tuned from the strong synthetic checkpoint:
train_stream = cm.make_dataset("train", batch_size, shuffle=True).unbatch()
if args.real_mix:
real_stream = (cm.make_real_dataset("train", batch_size, shuffle=True)
.unbatch().repeat(args.real_oversample)) # 25x
train_stream = train_stream.concatenate(real_stream)
train_ds = (train_stream.shuffle(4096, reshuffle_each_iteration=True)
.batch(batch_size).prefetch(2)) Result.
Real top-1 nearly triples, median circular error halves, synthetic accuracy holds within 0.35 points, and initialising from the synthetic checkpoint beats training from ImageNet. Grad-CAM agrees qualitatively: heatmap focus rises from 0.262 ± 0.10 to 0.357 ± 0.14, tighter on 72% of photos — from wandering over the bezel and background to locking on the hub.
And now the part that most write-ups would leave out. These are paired predictions on the same 57 images, so McNemar’s exact test applies:
baseline 4/57 = 0.070 real-mix 11/57 = 0.193
discordant: baseline-only-right 2, real-mix-only-right 9
McNemar exact p = 0.065
p = 0.065. The headline “3× improvement” does not clear α = 0.05. The direction is consistent across four independent signals — top-1, median error, attention focus, confidence ranking — and 9-versus-2 discordant pairs is suggestive. But on a 57-image test set it is not established, and I am not going to write it as though it were.
Note the contrast with section 2b: the resize-filter finding, which I stumbled into and was least excited about, is the statistically solid one (p = 0.016 and 4 × 10⁻⁷). The finding I most wanted to be true is the weakest. That ordering is not a coincidence — it is what happens when effect size is large but n is 57.
Having used confident model-label disagreement to find defects in someone else’s dataset (2c), I ran the same procedure against my own.
The 92 kongaskristjan photos, labelled in their filenames, check out — flagged images there are genuine model errors on genuinely hard photographs.
The 103 photos from the second source do not. Its labels come from a label.csv indexed by row, and spot-checking flagged images turns up plain errors: one labelled 12:03 shows ~6:05, one labelled 2:20 shows ~4:00. The errors are not a consistent offset, so there is no row-misalignment to undo — just noise.
My 57-photo test split is ~58% drawn from that source. So the 19.3% is contaminated in a direction I cannot determine without re-labelling: label noise in the test set usually depresses measured accuracy, but if the fine-tune partially learned the same noise from the training half, it could equally be flattered.
Combined with p = 0.065, the honest summary of section 7 is: directionally supported, quantitatively unestablished. The clean version is a re-run on the 92 trustworthy photos only, and it is the next thing on the list.
Small evaluation sets were the binding constraint on everything in sections 4–7, so it is worth stating the resolution limit explicitly. At 80% power and α = 0.05:
Wilson 95% intervals on the key results:
Those intervals are 7–20 points wide. Any intervention worth less than ~10 points is invisible at this sample size — which means my two “falsified” hypotheses are precisely falsified as large effects, and the supported one sits right at the edge of detectability. The source-domain results (n = 1,440 and n = 14,400) do not have this problem, which is exactly why the resize-filter finding is airtight and the sim2real findings are not.
I am data-bound in both directions: not enough real photos to train on, and not enough to measure with.
99.8% was never a lie; it was a narrow claim. The benchmark says “this model reads these renders.” I heard “this model reads clocks.” Every subsequent surprise came from that substitution.
Pin your preprocessing before you rank checkpoints. An undocumented resize filter was worth 0.5 points on one model and 1.7 on another, in opposite directions, and it fabricated a signature failure mode that no weight in the network was responsible for. Match evaluation preprocessing to training preprocessing, and state it in the README.
Structure in the error distribution beats aggregate accuracy. “Every error is an exact multiple of 60 minutes” identified a label-ordering bug; “every error is exactly ±195 minutes” identified a renderer bug. Neither is visible in a top-1 number, and the second is not visible in a confusion matrix either — it needed the signed circular offset.
Grad-CAM discriminates failure modes, not just correctness. Tight attention
Calibration is domain-shifted too. ECE 0.002 in-distribution and 0% accuracy in the p ≥ 0.8 bucket out-of-distribution are the same model. Temperature scaling cannot fix an inverted ranking, and a selective-prediction system built on the in-distribution calibration would fail hardest on exactly the inputs it was meant to catch.
Report the test statistic on your favourite result, not just your least favourite. It is easy to demand rigour of a hypothesis you are falsifying and grant a pass to the one that worked. My best result is p = 0.065 on a partly mislabelled 57-image split. Saying so costs nothing and is the difference between a finding and a claim.
When you are data-bound, say which direction. 195 photographs across 99 of 144 classes, one source noisily labeled, and an evaluation set that cannot resolve anything under 15 points. No architecture search fixes that. The next lever is more photographs, cleanly labeled — or a renderer good enough that the output stops looking like a diagram.
The clock on my wall still wins. But the gap is now decomposed, three of the four obvious explanations are eliminated with numbers attached, and the remaining one has a price tag: labeled photographs, in the hundreds, not the dozens.
Every checkpoint, the failed runs included, plus a full dated changelog of each experiment above:
python scripts/evaluate.py --split test # 0.9972 (default ckpt)
python scripts/rotation_range_sweep.py # section 4
python scripts/calibration.py --split real-test # section 5
python scripts/train.py --realism-aug --real-mix \
--init-weights models/clock_model_rot54_s0.keras # section 7
python scripts/flag_suspects.py --split real-all # section 8
Code, checkpoints, changelog, and a browser demo running a 627 KB int8 model client-side: the project page.
<hr><p>From 99.8% to 2.2%: Diagnosing the Sim2Real Gap in a 144-Class Clock Classifier was originally published in Artificial Intelligence in Plain English on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>