Object Detection Lab

About

An object detector answers two questions at once: what is in the image, and where. Its output is a list of boxes, each with a class label and a confidence score.

The detector here is real. It is COCO-SSD from the TensorFlow.js model zoo: an SSDLite detection head on a MobileNetV2 backbone, trained on the COCO dataset's 80 everyday object classes. The weights (about 18 MB) are downloaded once, and every image or camera frame is processed by your own GPU through WebGL. Nothing you show it leaves your device.

It is built for speed rather than accuracy. Google's model zoo lists this configuration at 22 mAP on COCO, a mid-range score even in 2018, but it runs comfortably in real time inside a browser tab. That trade-off makes its mistakes easy to find, which is the point: lower the threshold and watch what a detector actually believes.

On each frame the network scores about 1,900 candidate boxes against 90 COCO category ids. This lab keeps every candidate scoring at least 10%, runs its own non-max suppression, then applies your threshold, so you can change both thresholds without re-running the network.

The 80 classes it knows

Things to try

1. Slide the thresholdOn the Crossing photo, drag confidence from 50% down to 15%. Small, distant people appear, and so do the first false alarms.
2. Break NMSTurn on raw boxes, then drag the IoU threshold up to 0.9. Duplicate boxes for the same person survive. At 0.1, people standing close together start to disappear.
3. Ask about the unknownPoint the camera at something outside the 80 classes, such as a lamp or a plant pot. The model either stays silent or picks the closest class it knows.
4. OcclusionOn camera, cover half your face with your hand, then move partly out of frame. Watch how the person score falls.
5. BreakfastCups and wine glasses are small and cluttered. Compare what is found at 50% with 25%.

Sample photos

All four are CC0 (public domain) images from Wikimedia Commons, resized for the web: People crossing street, Cat on laptop yawning, Breakfast in Île d'Orléans and a street photo of cyclists in Amsterdam by Fons Heijnsbroek.

How it works

1. Backbone. The image is resized to 300 × 300 pixels and passed through MobileNetV2, a convolutional network designed for phones. Its layers turn pixels into feature maps: grids of numbers that describe edges, textures and eventually object parts. The convolutions are the same weighted sums you can step through in the Image Filter Playground, with learned weights.

2. Default boxes. SSD (Single Shot MultiBox Detector) places a fixed set of default boxes, also called anchors, at every cell of several feature maps: many small boxes on the fine early maps, a few large boxes on the coarse late ones. Different aspect ratios cover tall people and wide cars.

3. Predictions per box. For every default box a small convolutional head predicts a score for each class and four numbers that nudge the box's centre, width and height. SSDLite makes these heads cheap by using depthwise separable convolutions. All of it happens in one forward pass, hence “single shot”.

4. Clean-up. Most boxes score near zero for everything. The survivors overlap heavily, because neighbouring anchors all see the same person. A confidence threshold removes weak boxes and non-max suppression removes duplicates. What is left is the list you see.

Training in one paragraph

During training each ground-truth box in a COCO image is matched to the default boxes that overlap it well. Those anchors are trained to predict its class and the offsets to its exact position; all others are trained to predict background. Because background anchors vastly outnumber object anchors, SSD keeps only the hardest negatives (roughly three for every positive) so the loss is not swamped by easy empty sky.

For the full story from R-CNN to YOLO, read the Object Detection lesson.

Thresholds and NMS

Confidence threshold. Each box carries a score between 0 and 1. The threshold decides which ones to believe. Raise it and you get fewer false alarms but miss more real objects (higher precision, lower recall). Lower it and the reverse. There is no single right value: a photo-tagging app can afford false alarms, a safety system cannot afford misses. Precision-recall curves, and the mAP score built from them, summarise a detector across every threshold at once.

Intersection over union. To decide whether two boxes describe the same object, compare their overlap with their combined area:

1 means identical boxes, 0 means no overlap. COCO counts a detection as correct when its IoU with the true box is at least 0.5 (and averages over stricter thresholds up to 0.95).

Non-max suppression. Sort boxes by score. Keep the best one, delete every box of the same class whose IoU with it is above the NMS threshold, and repeat with the next survivor. This lab runs that greedy loop per class in plain JavaScript on every frame.

What to look for

1. Raw boxesWith raw boxes on, the dashed outlines are all candidates above the threshold. Often a single person has five to ten.
2. IoU too highAt 0.9 almost nothing counts as a duplicate, so several boxes stack on one object.
3. IoU too lowAt 0.1 two people standing side by side overlap enough that the weaker one is deleted. Crowds are the classic NMS failure.
4. Threshold versus NMSThe two work together: the stats bar shows how many candidates pass the threshold and how many survive suppression.

Limits

A closed vocabulary. The model can only output the 80 COCO classes. Anything else is either ignored or forced into the nearest class: a cushion might become a teddy bear, a tablet a laptop or a tv. Its confidence says nothing about whether the right answer was even available.

Small and crowded objects. Downscaling to 300 × 300 and a limited set of default boxes make small objects and dense crowds hard, and NMS can merge neighbours.

Dataset bias. COCO images come mostly from Flickr photos of everyday scenes. Unusual viewpoints, lighting, cultures and objects are under-represented, and a detector inherits those gaps. Studies of pedestrian detectors have found accuracy differences across skin tones, which is why evaluation on the population a system will actually serve matters.

Scores are not probabilities. A 90% score does not mean the box is right nine times out of ten. Detectors are often overconfident; the Calibration Lab shows how to check and fix that.

Using it responsibly

A 2018 mobile detector is a good teaching tool and a poor safety system. Anything that acts on detections in the physical world, such as a car, a robot or a security camera, needs a much stronger model, evaluation on its own data, and a plan for what happens when the detector is wrong.

Beyond SSD

SSD appeared in 2016 alongside Faster R-CNN (a two-stage detector that first proposes regions, then classifies them) and YOLO (another single-shot design). The YOLO family kept evolving and is still the default choice for fast detection. In 2020 DETR recast detection as a set-prediction problem with a transformer, removing hand-designed anchors and NMS altogether.

The biggest change since is the move to open-vocabulary detection. Models such as OWL-ViT and Grounding DINO are trained with paired images and text, so you can ask for “red umbrella” or “forklift” without retraining. Multimodal language models can now point at and describe objects in the same way. The Multimodal AI lesson covers how that works.

Sources and further reading

1. 01SSD: Single Shot MultiBox Detector. Liu et al., ECCV 2016
2. 02MobileNetV2: inverted residuals and linear bottlenecks. Sandler et al., CVPR 2018 (introduces SSDLite)
3. 03Microsoft COCO: common objects in context. Lin et al., ECCV 2014
4. 04COCO-SSD for TensorFlow.js. TensorFlow.js models repository
5. 05TensorFlow 1 detection model zoo. TensorFlow Object Detection API (speed and COCO mAP per model)
6. 06Predictive inequity in object detection. Wilson, Hoffman and Morgenstern, 2019
7. 07Faster R-CNN: towards real-time object detection with region proposal networks. Ren et al., NeurIPS 2015
8. 08You only look once: unified, real-time object detection. Redmon et al., CVPR 2016
9. 09End-to-end object detection with transformers (DETR). Carion et al., ECCV 2020
10. 10Simple open-vocabulary object detection with vision transformers (OWL-ViT). Minderer et al., ECCV 2022
11. 11Grounding DINO: marrying DINO with grounded pre-training for open-set object detection. Liu et al., 2023