Object detection is a computer-vision task that identifies what objects are present in an image or video frame and predicts where each object is located. Most detectors return a class label, a bounding box, and a confidence score for every reported object.
Unlike image classification, which often assigns one or more labels to an entire image, object detection can find several objects and locate each one separately. That difference makes detection useful whenever a system needs to answer both “what is here?” and “where is it?”
What an object detector produces
Imagine a photograph of a park path containing a person, a dog, a bicycle, and a bench. A detector might report four predictions:
- person — bounding box around the walker — confidence 0.96
- dog — bounding box around the animal — confidence 0.94
- bicycle — bounding box around the bike — confidence 0.91
- bench — bounding box around the seat — confidence 0.88
Each prediction combines three ideas: classification names the object, localization describes its position, and the confidence score indicates how strongly the model supports that candidate prediction. Post-processing may remove or combine overlapping candidates, depending on the model and deployment pipeline.
Object detection vs classification vs segmentation
| Computer-vision task | Question it answers | Main output | Example |
|---|---|---|---|
| Classification | What is in the image? | Image-level category label or labels | “This image contains a dog.” |
| Object detection | What objects are present, and where? | Object labels, locations, and confidence scores | “Dog at this box; bicycle at that box.” |
| Segmentation | Which pixels belong to each object or region? | Pixel- or region-level masks | “These exact pixels form the dog.” |
Classification is the simplest of the three outputs. Detection adds object-level location. Segmentation adds more detailed boundaries. A system can combine these tasks, but they are not interchangeable. For more background, read Image Classification Explained and Computer Vision Explained.
How object detection works
There is no single universal detector pipeline. Modern systems usually learn visual features from training images and then predict both class information and localization information. During training, the model compares its predictions with annotated examples and adjusts its parameters to reduce classification and localization errors.
- Process the image. A neural network converts image pixels into useful visual representations.
- Generate object candidates. The detector predicts possible objects through dense image locations, region proposals, learned object queries, or a hybrid approach.
- Predict labels and locations. Each candidate receives class information and a localization prediction such as a bounding box.
- Filter the results. Confidence thresholds and, for many architectures, overlap-based post-processing determine which predictions remain.
Architecture families continue to evolve. The durable idea is the task itself: a detector must recognize individual objects and localize them, not merely label the image as a whole.
Bounding boxes
A bounding box is a rectangle that approximates an object’s location. Systems may store a box as two corner coordinates or as a center point plus width and height. Either representation describes the same basic region.
Localization quality matters. A detector could predict the correct class but draw a box that barely covers the object. Under many evaluation rules, that prediction would not count as a correct match.
Intersection over Union (IoU)
Intersection over Union measures the overlap between a predicted region and a reference region. For bounding boxes, the calculation is:
IoU = area of overlap ÷ area of union
For a simple example, suppose the overlapping portion of two boxes has an area of 40 square units and their combined union has an area of 100 square units. The IoU is 40 ÷ 100, or 0.40. Perfectly aligned boxes have an IoU of 1.00; boxes with no overlap have an IoU of 0.
An evaluation protocol can use an IoU threshold to decide whether a predicted box matches a ground-truth object. Different benchmarks may use different thresholds or average results across several thresholds, so the threshold must always be stated or understood in context.
Precision and recall in object detection
Detection evaluation accounts for both recognition and localization:
- Precision asks what share of reported detections are correct matches.
- Recall asks what share of ground-truth objects were successfully detected.
Changing the confidence threshold can change both metrics. Read Accuracy vs Precision vs Recall for a broader introduction to these measures.
What is mAP?
Mean Average Precision, or mAP, summarizes object-detection performance under a defined evaluation protocol. Average precision captures the precision–recall relationship for a class under specified matching rules. mAP averages results across classes and, in some benchmarks, across multiple IoU thresholds.
An mAP value is not meaningful in isolation. Dataset classes, IoU thresholds, object-size ranges, maximum detections, interpolation rules, and implementation details can all affect the reported number. Compare mAP values only when the benchmark and protocol are compatible.
Confidence scores and thresholds
Detectors often assign confidence scores to candidate predictions. A deployment threshold decides which candidates are shown or acted upon.
- A higher threshold can remove weak predictions and reduce false positives, but it may miss more real objects.
- A lower threshold can retain more real objects and improve recall, but it may also report more false positives.
The right setting depends on the cost of each error. A manufacturing system that must catch rare defects may tolerate more false alarms. A low-risk consumer feature may prioritize fewer distracting false detections. Thresholds should be tuned and validated on data that reflects the intended environment.
Modern object-detection architectures
Modern detection includes several broad architecture families:
- One-stage detectors predict object information directly across image locations and are often designed for efficient inference.
- Two-stage detectors first generate candidate regions and then classify and refine them.
- Transformer-based detectors can formulate detection as set prediction using learned object queries.
- Hybrid systems combine ideas from multiple families.
No family is best for every use. Model choice depends on accuracy requirements, latency, hardware, object size and density, available training data, and the deployment environment.
Real-world applications
- Manufacturing: finding components or visible defects on a production line.
- Retail and inventory: locating products, shelves, or empty spaces.
- Robotics: helping a robot perceive objects it may navigate around or manipulate.
- Traffic analysis: counting and locating vehicles, cyclists, and pedestrians.
- Agriculture: identifying crops, fruit, weeds, or visible signs of stress.
- Medical-imaging research and support: locating candidate regions for expert review.
- Video analytics: detecting objects frame by frame before tracking or event analysis.
High-stakes uses require appropriate validation, human oversight, privacy controls, and domain-specific evaluation. A model that performs well on a general benchmark is not automatically safe or effective in a specialized setting.
Common challenges
Small or crowded objects
Small objects contain fewer visual details, and crowded scenes can make neighboring instances difficult to separate.
Occlusion
Objects partly hidden behind other objects may be harder to recognize and localize reliably.
Distribution shift
Changes in cameras, lighting, weather, geography, backgrounds, or object appearance can reduce performance after deployment.
Class imbalance
Rare categories may have poor recall even when aggregate metrics look strong. Per-class results can reveal failures hidden by an average.
Annotation quality
Inconsistent boxes, missing objects, and ambiguous class labels limit what a model can learn and complicate evaluation.
Frequently asked questions
Is object detection the same as image classification?
No. Image classification labels an entire image. Object detection identifies individual objects and predicts a location for each one.
What is a bounding box?
A bounding box is a rectangle that approximates an object’s location in an image. It is commonly stored using corner coordinates or a center point with width and height.
What is IoU?
Intersection over Union measures the overlap between a predicted region and a reference region relative to their combined union area.
What is mAP?
mAP summarizes detection performance across classes under a defined precision–recall and localization protocol. It must be interpreted with its benchmark and evaluation settings.
Is a higher confidence threshold always better?
No. Higher thresholds can reduce false positives but also miss real objects. The useful threshold depends on the application and the consequences of each error type.
Where to learn next
Technical references
- Ren et al., Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.
- Carion et al., End-to-End Object Detection with Transformers.
- Lin et al., Microsoft COCO: Common Objects in Context.