Detectron2 for Production Object Detection: Architecture, Training, and Inference at Scale

When Facebook AI Research released Detectron2, I was in the middle of a painful project: building a custom object detection pipeline for a forestry application that needed to identify individual tree crowns in high-resolution aerial imagery. The existing approach—YOLOv3 with a custom training loop, manual anchor tuning, and brittle data augmentation—was consuming more engineering time than the actual modeling work. Detectron2 changed that equation completely, providing a modular, well-engineered framework that let me focus on the problem rather than the plumbing.

After nine years of building computer vision systems—from satellite imagery analysis to medical image diagnostics to industrial quality inspection—Detectron2 has become my default choice for object detection when accuracy matters more than raw inference speed. This is a practitioner’s guide to using it effectively in production.

Why Detectron2

Detectron2 is Facebook AI Research’s next-generation library for object detection and segmentation. Built on PyTorch, it provides a clean, modular architecture that supports a wide range of detection models: Faster R-CNN, Mask R-CNN, RetinaNet, FCOS, and more.

The reasons I reach for Detectron2 over alternatives:

1. Model zoo with pretrained weights: Detectron2’s model zoo provides COCO-pretrained weights for dozens of architectures. Transfer learning from COCO-pretrained models is the single most effective technique for custom detection tasks.

2. Modular configuration: The config system (based on YAML files and a chain-of-responsibility pattern) makes it trivial to swap architectures, backbones, and training parameters without touching code.

3. First-class support for instance segmentation: If you need pixel-level masks (not just bounding boxes), Detectron2’s Mask R-CNN implementation is the gold standard.

4. Clean PyTorch integration: Unlike some frameworks that abstract away PyTorch to the point of opacity, Detectron2 exposes PyTorch’s training loop, optimizers, and gradient management in a way that makes debugging and customization straightforward.

Architecture Selection: Choosing the Right Model

Detectron2 supports multiple detection architectures. Choosing the right one for your use case is the most important decision you will make.

Faster R-CNN

The workhorse of object detection. Faster R-CNN uses a Region Proposal Network (RPN) to generate candidate bounding boxes, then classifies and refines each proposal.

When to use it:

• General-purpose object detection
• When you need high accuracy across a wide range of object sizes
• When inference speed is not the primary constraint (30-100ms per image on GPU)

Key configuration:

MODEL:
  META_ARCHITECTURE: "GeneralizedRCNN"
  RESNETS:
    DEPTH: 101
    STRIDE_IN_1X1: True
    NUM_GROUPS: 32
    WIDTH_PER_GROUP: 8
    RES5_DILATION: 1
  FPN:
    IN_FEATURES: ["res2", "res3", "res4", "res5"]
    OUT_CHANNELS: 256
  ANCHOR_GENERATOR:
    SIZES: [[32], [64], [128], [256], [512]]
    ASPECT_RATIOS: [[0.5, 1.0, 2.0]]
  RPN:
    IN_FEATURES: ["p2", "p3", "p4", "p5", "p6"]
    PRE_NMS_TOPK_TRAIN: 2000
    PRE_NMS_TOPK_TEST: 1000
    POST_NMS_TOPK_TRAIN: 1000
    POST_NMS_TOPK_TEST: 1000
  ROI_HEADS:
    IN_FEATURES: ["p2", "p3", "p4", "p5"]
    NUM_CLASSES: 10
  ROI_BOX_HEAD:
    NAME: "FastRCNNConvFCHead"
    NUM_FC: 2
    POOLER_RESOLUTION: 7

RetinaNet

A single-stage detector that uses a Feature Pyramid Network (FPN) with focal loss to handle class imbalance. Faster than Faster R-CNN but slightly less accurate on small objects.

When to use it:

• Real-time detection where speed matters
• Datasets with severe class imbalance
• When you need a simpler architecture (no region proposals)

FCOS (Fully Convolutional One-Stage)

An anchor-free detector that predicts bounding boxes directly from feature maps. Simpler than anchor-based methods and often competitive in accuracy.

When to use it:

• When anchor tuning is a pain point
• Objects with unusual aspect ratios
• Cleaner architecture for research and experimentation

Mask R-CNN

Faster R-CNN extended with a mask prediction branch. Use this when you need instance segmentation—pixel-level masks for each detected object.

When to use it:

• Instance segmentation tasks
• When precise object boundaries matter (medical imaging, autonomous driving)
• Panoptic segmentation (with extensions)

Custom Dataset Preparation

Detectron2 uses a specific dataset format. Getting your data into this format correctly is critical.

The COCO Format

import json
from detectron2.structures import BoxMode

def create_coco_annotation(images, annotations, categories):
    """Create COCO-format annotation dictionary."""
    coco_dict = {
        "images": images,
        "annotations": annotations,
        "categories": categories,
    }
    return coco_dict

# Image entry
image_entry = {
    "id": 1,
    "file_name": "image_001.jpg",
    "width": 1920,
    "height": 1080,
}

# Annotation entry
annotation_entry = {
    "id": 1,
    "image_id": 1,
    "category_id": 0,
    "bbox": [100, 200, 50, 80],  # [x, y, width, height]
    "bbox_mode": BoxMode.XYWH_ABS,
    "area": 4000,
    "iscrowd": 0,
    "segmentation": [[100, 200, 150, 200, 150, 280, 100, 280]],  # Polygon
}

# Category entry
category_entry = {
    "id": 0,
    "name": "tree_crown",
    "supercategory": "vegetation",
}

Dataset Registration

from detectron2.data import DatasetCatalog, MetadataCatalog
import json
import os

def get_tree_crown_dicts(split):
    """Load tree crown dataset for given split."""
    json_file = f"data/tree_crown/{split}.json"
    with open(json_file) as f:
        imgs_anns = json.load(f)

    dataset_dicts = []
    for idx, (filename, v) in enumerate(imgs_anns.items()):
        record = {
            "file_name": os.path.join("data/tree_crown/images", filename),
            "image_id": idx,
            "height": v["height"],
            "width": v["width"],
            "annotations": [
                {
                    "bbox": obj["bbox"],
                    "bbox_mode": BoxMode.XYWH_ABS,
                    "category_id": obj["category_id"],
                    "iscrowd": obj.get("iscrowd", 0),
                }
                for obj in v["annotations"]
            ],
        }
        dataset_dicts.append(record)
    return dataset_dicts

# Register datasets
for split in ["train", "val", "test"]:
    DatasetCatalog.register(
        f"tree_crown_{split}",
        lambda split=split: get_tree_crown_dicts(split),
    )
    MetadataCatalog.get(f"tree_crown_{split}").set(
        thing_classes=["tree_crown", "dead_tree", "clearing"],
        thing_colors=[(0, 255, 0), (255, 0, 0), (255, 255, 0)],
    )

Training Pipeline

Configuration

from detectron2.config import get_cfg
from detectron2 import model_zoo

cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file(
    "COCO-Detection/faster_rcnn_R_101_FPN_3x.yaml"
))

# Dataset
cfg.DATASETS.TRAIN = ("tree_crown_train",)
cfg.DATASETS.TEST = ("tree_crown_val",)

# Model
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url(
    "COCO-Detection/faster_rcnn_R_101_FPN_3x.yaml"
)
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 3
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5

# Training
cfg.SOLVER.IMS_PER_BATCH = 4
cfg.SOLVER.BASE_LR = 0.001
cfg.SOLVER.MAX_ITER = 30000
cfg.SOLVER.STEPS = (20000, 25000)
cfg.SOLVER.GAMMA = 0.1
cfg.SOLVER.WARMUP_ITERS = 1000
cfg.SOLVER.CHECKPOINT_PERIOD = 5000

# Input
cfg.INPUT.MIN_SIZE_TRAIN = (800, 1000, 1200)
cfg.INPUT.MAX_SIZE_TRAIN = 1333
cfg.INPUT.MIN_SIZE_TEST = 1000
cfg.INPUT.MAX_SIZE_TEST = 1333

# Output
cfg.OUTPUT_DIR = "./output/tree_crown_detection"

# Dataloader
cfg.DATALOADER.NUM_WORKERS = 8
cfg.DATALOADER.FILTER_EMPTY_ANNOTATIONS = True

Training Loop

from detectron2.engine import DefaultTrainer, hooks
from detectron2.evaluation import COCOEvaluator
import os

class TreeCrownTrainer(DefaultTrainer):
    @classmethod
    def build_evaluator(cls, cfg, dataset_name, output_folder=None):
        if output_folder is None:
            output_folder = os.path.join(cfg.OUTPUT_DIR, "inference")
        return COCOEvaluator(dataset_name, cfg, False, output_folder)

    @classmethod
    def build_train_loader(cls, cfg):
        # Custom augmentations for aerial imagery
        from detectron2.data import build_detection_train_loader
        from detectron2.data import transforms as T

        augs = [
            T.RandomBrightness(0.8, 1.2),
            T.RandomContrast(0.8, 1.2),
            T.RandomSaturation(0.8, 1.2),
            T.RandomFlip(prob=0.5),
            T.RandomRotation(angle=[-15, 15]),
        ]
        return build_detection_train_loader(cfg, mapper=None)

trainer = TreeCrownTrainer(cfg)
trainer.resume_or_load(resume=False)

# Custom hooks
trainer.register_hooks([
    hooks.EvalHook(
        period=2000,
        eval_func=lambda: trainer.test(cfg, trainer.model),
    ),
    hooks.PeriodicCheckpointer(
        trainer.checkpointer,
        period=5000,
    ),
])

trainer.train()

Handling Class Imbalance

Real-world detection datasets are almost always imbalanced. In my forestry dataset, tree crowns outnumbered dead trees 50:1. This is where Detectron2’s flexibility shines.

Weighted Loss

from detectron2.modeling.roi_heads import StandardROIHeads
import torch.nn as nn

class WeightedROIHeads(StandardROIHeads):
    def _sample_proposals(self, proposals, targets):
        """Oversample rare classes during proposal sampling."""
        # Custom sampling logic that ensures rare classes
        # appear in each training batch
        pass

# Configure weighted focal loss
cfg.MODEL.ROI_HEADS.LOSS_WEIGHT = [1.0, 5.0, 2.0]  # Per-class weights

Data Augmentation for Rare Classes

from detectron2.data import detection_utils as utils
import copy

def custom_mapper(dataset_dict):
    """Custom data mapper with oversampling for rare classes."""
    dataset_dict = copy.deepcopy(dataset_dict)
    image = utils.read_image(dataset_dict["file_name"], format="BGR")

    # Check if this image contains rare classes
    has_rare_class = any(
        ann["category_id"] in [1, 2]  # dead_tree, clearing
        for ann in dataset_dict["annotations"]
    )

    # Apply more aggressive augmentation for rare classes
    if has_rare_class:
        aug_input = T.AugInput(image)
        transforms = T.AugmentationList([
            T.RandomBrightness(0.7, 1.3),
            T.RandomContrast(0.7, 1.3),
            T.RandomFlip(prob=0.5),
            T.RandomRotation(angle=[-30, 30]),
            T.RandomCrop("relative_range", (0.8, 0.8)),
        ])
        image, transforms = T.apply_transform_gens(transforms, aug_input)
    else:
        aug_input = T.AugInput(image)
        transforms = T.AugmentationList([
            T.RandomBrightness(0.9, 1.1),
            T.RandomFlip(prob=0.5),
        ])
        image, transforms = T.apply_transform_gens(transforms, aug_input)

    annos = [
        utils.transform_instance_annotations(
            ann, transforms, image.shape[:2]
        )
        for ann in dataset_dict["annotations"]
    ]

    return {
        "image": torch.as_tensor(image.transpose(2, 0, 1).astype("float32")),
        "instances": utils.annotations_to_instances(annos, image.shape[:2]),
    }

Focal Loss

Detectron2 supports focal loss natively through RetinaNet:

cfg.MODEL.RETINANET.FOCAL_LOSS_GAMMA = 2.0
cfg.MODEL.RETINANET.FOCAL_LOSS_ALPHA = 0.25

For Faster R-CNN, you can implement custom loss weighting in the classification head.

Inference Optimization

Production inference requires optimizing for both latency and throughput.

TorchScript Export

from detectron2.modeling import build_model
from detectron2.checkpoint import DetectionCheckpointer
import torch

# Build and load model
cfg.MODEL.WEIGHTS = "./output/model_final.pth"
cfg.freeze()
model = build_model(cfg)
DetectionCheckpointer(model).load(cfg.MODEL.WEIGHTS)
model.eval()

# Script for production
with torch.no_grad():
    # Create dummy input
    dummy_input = torch.randn(1, 3, 800, 1333)

    # Export with TorchScript
    scripted_model = torch.jit.trace(model, dummy_input)
    scripted_model.save("model_scripted.pt")

ONNX Export

from detectron2.modeling import build_model
import torch.onnx

model = build_model(cfg)
model.eval()

# Export to ONNX
torch.onnx.export(
    model,
    dummy_input,
    "model.onnx",
    opset_version=11,
    input_names=["input"],
    output_names=["boxes", "scores", "classes"],
    dynamic_axes={
        "input": {0: "batch_size"},
        "boxes": {0: "num_detections"},
        "scores": {0: "num_detections"},
        "classes": {0: "num_detections"},
    },
)

TensorRT Optimization

For maximum inference speed:

import tensorrt as trt

# Build TensorRT engine
logger = trt.Logger(trt.Logger.WARNING)
builder = trt.Builder(logger)
network = builder.create_network(
    1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH)
)
parser = trt.OnnxParser(network, logger)

with open("model.onnx", "rb") as f:
    parser.parse(f.read())

config = builder.create_builder_config()
config.max_workspace_size = 1 << 30  # 1GB
config.set_flag(trt.BuilderFlag.FP16)

engine = builder.build_engine(network, config)

In my experience, TensorRT optimization typically provides 3-5x speedup over native PyTorch inference, bringing Faster R-CNN inference down to 15-25ms per image on an NVIDIA T4.

Batch Inference

For processing large image collections (e.g., satellite imagery), batch inference is essential:

from detectron2.engine import DefaultPredictor
import cv2
import numpy as np

class BatchPredictor:
    def __init__(self, cfg):
        self.cfg = cfg.clone()
        self.model = build_model(self.cfg)
        self.model.eval()
        DetectionCheckpointer(self.model).load(cfg.MODEL.WEIGHTS)

    def __call__(self, images: list[np.ndarray]):
        """Process a batch of images."""
        from detectron2.structures import ImageList

        # Preprocess
        inputs = [
            {"image": torch.as_tensor(img.transpose(2, 0, 1).astype("float32"))}
            for img in images
        ]

        # Batch inference
        with torch.no_grad():
            outputs = self.model(inputs)

        return outputs

Production Deployment Architecture

For serving Detectron2 models in production, I use the following architecture:

1. Model server: TorchServe or Triton Inference Server with the exported model
2. Load balancer: AWS ALB or GCP Load Balancer for traffic distribution
3. Async processing: For batch workloads, use SQS/PubSub with worker pools
4. Monitoring: Prometheus metrics for latency, throughput, and error rates

# TorchServe handler
import json
import torch
from ts.torch_handler.base_handler import BaseHandler

class DetectionHandler(BaseHandler):
    def preprocess(self, data):
        images = []
        for row in data:
            image = row.get("data") or row.get("body")
            if isinstance(image, (bytes, bytearray)):
                image = cv2.imdecode(
                    np.frombuffer(image, np.uint8), cv2.IMREAD_COLOR
                )
            images.append(image)
        return images

    def inference(self, data):
        with torch.no_grad():
            return self.model(data)

    def postprocess(self, outputs):
        results = []
        for output in outputs:
            instances = output["instances"]
            results.append({
                "boxes": instances.pred_boxes.tensor.tolist(),
                "scores": instances.scores.tolist(),
                "classes": instances.pred_classes.tolist(),
            })
        return [json.dumps(r) for r in results]

Lessons from Production

1. Pretrained weights are everything: Always start from COCO-pretrained weights. The improvement from transfer learning is 2-5x larger than any architectural change you can make.

2. Data quality trumps quantity: 500 well-annotated images with diverse conditions beat 5,000 poorly annotated images. Invest in annotation quality.

3. Anchor tuning matters less than you think: Modern FPN-based architectures handle multi-scale objects well. Spend your time on data and augmentation, not anchor engineering.

4. Test on production distribution: Your model will see images in production that look nothing like your training data—different lighting, weather, camera angles, occlusion levels. Test on data that reflects these conditions.

5. Profile before optimizing: Do not assume inference is the bottleneck. Profile your entire pipeline—data loading, preprocessing, inference, postprocessing—before investing in optimization.

6. Log predictions systematically: Store predictions alongside inputs for debugging and model improvement. You will need this data for error analysis and active learning.

Conclusion

Detectron2 is not the fastest detection framework—YOLO and its variants will outperform it on raw inference speed. But for tasks where accuracy, flexibility, and maintainability matter more than millisecond-level latency, it is the best tool available. Its clean architecture, comprehensive model zoo, and PyTorch integration make it the framework I reach for when building detection systems that need to work reliably in production.

The key to success with Detectron2 is not the framework itself—it is the engineering discipline around it. Clean data pipelines, rigorous evaluation, proper monitoring, and iterative improvement based on production feedback. The framework handles the modeling; you handle the engineering.