Case study
Traffic Sign Recognition Using GTSRB
Cropped CNN on German GTSRB. Official Test.csv accuracy 99.01% and macro-F1 98.4% on 12,630 images. Speed-limit classes and held-out corruptions are the honest failure table.
At a glance
Plain summary for recruiters and visitors. Technical detail follows below.
- What it is
- I built a German traffic sign classifier on cropped images and reported where it still fails. It reaches 99.01% on the official test set with weak classes listed below.
- What I owned
- Solo end to end. Training pipeline, nested model selection, ONNX export for the private console, evaluation tables, and portfolio gallery.
- Why it matters
- Strong clean-crop accuracy helps ADAS and robotics prototypes. The failure table shows speed limit confusion and weather corruption limits before any production claim.
- Try it on this page vs full project
- The gallery shows curated label pairs with Meta reference images. Live ONNX inference runs in the private research console. The full training repo stays on request.
Problem
Clean-crop GTSRB models look strong on easy validation. Blur, lighting, noise, and compression still hurt. Full dashcam frames need detection first. This project classifies crops only.
Method
PyTorch CNN with ROI crop and weather domain randomization. Inverse-frequency weights and a weighted sampler were ablated. Nested multi-seed selection used corrupted-validation macro-F1. Optuna ran only after promote. ONNX export keeps Torch and ORT parity for the private research console.
Result
Official Test.csv accuracy 99.01% and macro-F1 98.4% on 12,630 images. Speed-limit slice accuracy is 98.6% but macro-F1 is 63%. Held-out rain, pixelate, zoom, glass, and elastic drop mean macro-F1 to 34%. Validation 99.7% is not the published claim.
Architecture
How the system is shaped. Full implementation stays private.
Step 1
Data
GTSRB German traffic-sign crops, 43 classes, severe imbalance. Track-level train, select, and calibrate split. Official Test.csv sealed until evaluate.
Step 2
Prep
48x48 RGB with ROI crop. Weather domain randomization on train (blur, JPEG, lighting, noise, lens dirt). Inverse-frequency sampler and class-weighted loss.
Step 3
Model
PyTorch cnn_deep, selected by corrupted-validation macro-F1 on blur, dark, noise, and JPEG. ONNX export with Torch and ORT parity for the private research console. The public gallery shows curated Meta reference pairs only.
Step 4
Eval
One pass on full Test.csv. Primary robustness table is held-out rain, pixelate, zoom, glass, and elastic. Speed-limit siblings and pedestrians vs road work remain the clean-test failure modes.
Official Test.csv (frozen nested evaluate)
Full official Test.csv, 12,630 images, after nested selection that never opened test for hyperparameter search. Speed-limit sibling confusions keep that slice macro-F1 at 63% even though slice accuracy is 98.6%. Held-out rain, pixelate, zoom, glass, and elastic mean macro-F1 is 34%. In-family blur and JPEG are selection probes only. Validation 99.7% is not the published claim.
| Signal | Score | Note |
|---|---|---|
| Official Test accuracy | 99.01% | Quote |
| Official Test macro-F1 | 98.40% | |
| Speed-limit slice macro-F1 | 63% | |
| Held-out shift mean macro-F1 | 34% |
Per-class Test.csv performance
Frozen sklearn report on all 12,630 official test images after nested selection. Overall accuracy 99.01% hides weak slices. Speed-limit siblings and triangle warnings drive most errors.
Speed-limit family slice
Classes 0–8 and 32 on Test.csv (n=4320). Accuracy 98.61%. Macro-F1 63.20%. Digit lookalikes dominate the errors.
| Class | Precision | Recall | F1 | Support |
|---|---|---|---|---|
| 27 Pedestrians | 96.08% | 81.67% | 88.29% | 60 |
| 6 End of speed limit 80 | 99.22% | 84.67% | 91.37% | 150 |
| 40 Roundabout mandatory | 86.41% | 98.89% | 92.23% | 90 |
| 41 End of no passing | 90.91% | 100.00% | 95.24% | 60 |
| 21 Double curve | 93.75% | 100.00% | 96.77% | 90 |
| 42 End of no passing by vehicles over 3.5t | 93.75% | 100.00% | 96.77% | 90 |
| 5 Speed limit 80 | 95.42% | 99.21% | 97.28% | 630 |
| 20 Dangerous curve right | 94.74% | 100.00% | 97.30% | 90 |
| 22 Bumpy road | 99.15% | 96.67% | 97.89% | 120 |
| 3 Speed limit 60 | 99.77% | 96.44% | 98.08% | 450 |
| 30 Beware of ice/snow | 100.00% | 96.67% | 98.31% | 150 |
| 39 Keep left | 100.00% | 96.67% | 98.31% | 90 |
Showing 12 weakest classes by F1 on Test.csv.
Top test confusions
Most frequent true → predicted pairs on the sealed Test.csv pass.
- Speed limit 60 → Speed limit 80(15 errors)
- Keep right → Roundabout mandatory(14 errors)
- End of speed limit 80 → Speed limit 80(11 errors)
- End of speed limit 80 → End of no passing(6 errors)
- Pedestrians → Road work(6 errors)
Algorithm
Nested robust-val CNN classification
Ablate on val, promote a recipe, tune, confirm over three seeds, then evaluate Test.csv once. Held-out corruptions are reported separately from in-family probes.
ablate recipes on val_robust (blur, dark, noise, jpeg)
promote winner
tune lr, weight decay, dropout on val only
confirm three seeds, still no Test.csv
evaluate full Test.csv once
report held-out rain, pixelate, zoom, glass, elasticKey logic
Compact illustrative snippet (Selection uses in-family probes only (illustrative)). Not the full codebase.
SELECTION = ("blur", "dark", "noise", "jpeg") # train overlap, not the paper table
HELD_OUT = ("rain", "pixelate", "zoom", "glass", "elastic")
score = mean_macro_f1(model, val, corruptions=SELECTION)
# early-stop and Optuna maximize score
# Test.csv and HELD_OUT stay sealed until evaluate / robustnessStack
Try a sign gallery
Free Cloudflare demo with curated German traffic-sign classes. Thumbnails use official GTSRB Meta reference images. Headline numbers are full official Test.csv (12,630 images), not this widget.
Source code
Full source is available to hiring managers on request. The public page shows architecture, algorithms, and compact proofs only.