Roboflow Training And Evaluation

For agents — source-of-truth: This skill is authored in `roboflow/computer-vision-skills` and shipped with the Roboflow plugin. If your client has loaded the plugin (you'll see roboflow:<name> skills in your available skills list), use those local skills — they're read fresh from disk every session. The same content served as MCP resources at roboflow://skills/<name>/... is a fallback for clients without the plugin and may lag this repo. Don't call `ReadMcpResourceTool` for `roboflow://skills/...` URIs when a local `roboflow:<name>` skill is available.

Training & Evaluation on Roboflow

Training Flow

Upload/Annotate Images
  → Generate Dataset Version (preprocessing + augmentation + train/val/test split)
    → Pick Model Architecture + Size
      → Pick Checkpoint (COCO, Universe model, or previous version)
        → Train
          → Evaluate (auto-runs for paid users)

Version = frozen snapshot. Changes to the project after version creation do not affect it. Configure preprocessing (resize, contrast, etc.) and augmentation (flip, rotate, mosaic, etc.) during version generation.

Available Model Architectures

Object Detection

Instance Segmentation

Semantic Segmentation

Classification

Keypoint / Pose

Multimodal / VLM

Model Selection Decision Tree

Follow this flowchart to pick the right model. Start at Step 1.

1. Task type? OD / Instance Seg / Keypoint → Step 2. Classification / Semantic Seg / VLM → use specialized block directly. 2. Target classes in COCO 80? Yes → Step 3. No → Step 6. 3. Real-time? No (images/recorded video) → Step 4. Yes (live video) → Step 5. 4. Non-real-time, COCO — Pick model family by task, default Medium size (Small for constrained HW, XL for accuracy-first): OD → RF-DETR, Inst Seg → RF-DETR Seg, Keypoint → YOLO26 pose. Done. 5. Real-time, COCO — Same families, pick Nano–Small, prioritize latency. Done. 6. Non-COCO, which sub-task? OD → Step 7. Inst Seg → Step 8. Keypoint → Step 9. 7. OD, non-COCO — Check Rapid exclusions (see below). If excluded → Step 13. Otherwise → recommend Roboflow Rapid (default) or SAM3 zero-shot as secondary option → Step 10. 8. Inst Seg, non-COCO — SAM3 zero-shot (sam3/sam3_final, set class_names). Rapid does not support segmentation → Step 10. 9. Keypoint, non-COCO — Not real-time → YOLO26 pose Medium–XL. Real-time → YOLO26 pose Nano–Small. Done. 10. Real-time? No → try model (Step 11). Yes → warn about latency, try model (Step 12). 11. Non-real-time trial — User confirms works → Done. Poor results → Step 13. 12. Real-time trial — User confirms works → Done. Poor results → Step 13. 13. Universe Model Search — search community models on Roboflow Universe. Good match → Done. No match → Step 14. 14. Custom Training — Fine-tune RF-DETR on user data. Size by HW constraints. Done.

Model ID Reference

Use these exact model_id values. Do not guess — wrong IDs cause training failures.

Object Detection

Instance Segmentation

Keypoint / Pose

Classification

Semantic Segmentation

Multimodal / VLM

Other

COCO 80 Classes (RF-DETR coverage)

person, bicycle, car, motorcycle, airplane, bus, train, truck, boat, traffic light, fire hydrant, stop sign, parking meter, bench, bird, cat, dog, horse, sheep, cow, elephant, bear, zebra, giraffe, backpack, umbrella, handbag, tie, suitcase, frisbee, skis, snowboard, sports ball, kite, baseball bat, baseball glove, skateboard, surfboard, tennis racket, bottle, wine glass, cup, fork, knife, spoon, bowl, banana, apple, sandwich, orange, broccoli, carrot, hot dog, pizza, donut, cake, chair, couch, potted plant, bed, dining table, toilet, tv, laptop, mouse, remote, keyboard, cell phone, microwave, oven, toaster, sink, refrigerator, book, clock, vase, scissors, teddy bear, hair drier, toothbrush.

Model Selection Quick Guide

RF-DETR NAS (Neural Architecture Search)

Instead of picking a single RF-DETR size manually, NAS trains many variants and reports the speed/accuracy frontier so you can pick the one that fits your hardware budget.

• What: A NAS run explores the RF-DETR architecture search space, then trains the surviving candidates and reports each one's mAP and measured latency on a target hardware (e.g., Jetson, T4 GPU). The output is a set of models on a Pareto frontier, plus an auto-selected "recommended" model chosen using Roboflow's current ranking heuristic to balance validation accuracy and measured latency on the target hardware.
• Tasks: Object Detection (rfdetr-nas) and Instance Segmentation (rfdetr-nas-seg).
• When to use: When you want the best speed/accuracy tradeoff for a specific deployment target and don't want to A/B-test sizes manually. Especially valuable for edge hardware where latency budgets are tight.
• Phases:

1. Mining — explores architectures and builds a Pareto frontier (latency vs mAP). Live updates while running. 2. Training — trains each frontier candidate end-to-end. Each becomes a regular model you can deploy.

• Plan gating: Requires the canTrainNas workspace feature flag. Self-serve plans (basic/starter/sandbox/research/trial) need to upgrade; enterprise/legacy plans need to contact sales.
• Start a run: Train page with ?engine=nas (UI: pick Neural Architecture Search as the training engine). Results land at /{workspace}/{project}/nas-runs/{versionId}.
• Deploy: Each NAS-produced model deploys like any other — pick one (typically the recommended) and use it as a normal Roboflow model. Inference type is rfdetr-nas / rfdetr-nas-seg, but it's served through the standard inference paths.
• References: RF-DETR paper (arxiv), ICLR 2026, What is NAS? (blog).

Roboflow Instant / Rapid

Roboflow Instant

• What: Few-shot model, trains in minutes, free
• Task: Object Detection only
• When to use: PoC, <1000 images, quick iteration
• Auto-trains when you approve a batch and no Instant model exists yet
• No preprocessing/augmentation -- uses images as-is
• Deploy: Available in Workflows like any trained model
• Manual trigger: Project > Models > Train Model > Roboflow Instant Model

Roboflow Rapid

• What: Interactive annotation-and-training workflow — SAM3 pre-annotates a small image set, user reviews/corrects, a fast custom OD model trains automatically. Model keeps improving as it captures more production data.
• Task: Object Detection only, non-COCO classes
• When to use: Default path for non-COCO object detection when exclusions don't apply

Do NOT use Rapid when:

When Rapid is excluded → recommend custom training with RF-DETR fine-tuning.

Checkpoint Training

Training Controls

• Cancel Training: Stops job, no weights saved. Refund if early in training.
• Early Stopping: Stops job, saves weights. Use when graphs show convergence with many epochs remaining. Charges for used credits.
• NAS Training: Shows paired charts (mining progress + Pareto curve, then per-model training curves). May auto-stop on convergence. See RF-DETR NAS section below.

Post-Training Metrics

Metrics vary by project type:

Model Evaluation (Paid Plans)

Auto-runs after training. Access: Models > click model version > View Evaluation.

Viewing & Comparing Models

• Models page: Project sidebar > Models. Shows all Instant + fine-tuned models with metrics, architecture, license, dataset version used.
• Universe tab: Starred Universe models available for transfer learning.
• Visualize: Quick test on test-set images, uploaded images, or webcam. Works for OD, segmentation, classification, keypoint. Not supported for multimodal.

MCP Tools Reference

Related Pages

• roboflow://skills/roboflow-model-improvement/SKILL — diagnostic decision tree, confusion matrix guide, per-class metrics, architecture switching, iterative improvement checklist
• roboflow://skills/training-and-evaluation/active-learning — production feedback loop: Dataset Upload workflow block, confidence-based sampling, review and retrain cycle

Active Learning on Roboflow

Source-of-truth note: This page ships with the Roboflow plugin. If your client has the plugin loaded, prefer the local skill (roboflow:training-and-evaluation) over fetching roboflow://skills/training-and-evaluation/active-learning via ReadMcpResourceTool — the MCP resources are a fallback for non-plugin clients and may lag the source repo.

Active learning closes the gap between your training dataset and the real world. Instead of hunting for images to upload, your deployed workflow automatically saves production images back to your Roboflow project. You review, annotate, retrain, and repeat.

This is the right answer when a user asks how to:

• improve their model using end-user or production images
• add images from their running application to their dataset
• set up a feedback loop between deployment and retraining
• implement active learning, data flywheel, or continuous learning

The Core Pattern

Add a Dataset Upload block to your existing production inference workflow. The block conditionally saves images — with predictions as pre-annotations — to a Roboflow project.

Image Input
  → Model Block (object detection / classification / segmentation)
    → [Optional: ContinueIf / confidence filter]
      → Dataset Upload Block → saved to Roboflow project
    → [Other output blocks — visualization, Slack, etc.]

If the user is starting a new integration without an existing workflow, this is a natural reason to set one up — the Dataset Upload block drops in as a zero-friction addition when a workflow is already in place.

Use workflow_blocks_get_schema (with the manifest key from workflow_blocks_list) to get the current block schema; block properties can change so look them up rather than relying on hardcoded names. Follow the Mode A or Mode B authoring flow in roboflow://skills/inference/SKILL to create and save the workflow.

Filtering What Gets Uploaded

Uploading every frame is rarely useful. Three common approaches:

Low-confidence sampling — gate the Dataset Upload block with a ContinueIf block set to pass only predictions below a confidence threshold. These are the images most likely to help the model.

Random sampling — use ContinueIf or Expression to sample a configurable percentage of frames. Useful for capturing distribution shifts even on high-confidence predictions.

Class-based filtering — route images containing specific classes or failing specific conditions. Useful when certain classes are underperforming (see improvement playbook).

Reviewing and Using Uploaded Images

1. Review in Roboflow — Images land in the project's unassigned pool. Saved predictions appear as pre-annotations, so annotation is correction rather than drawing from scratch. 2. Annotate — Accept, correct, or discard pre-annotations. Use AI-assisted labeling for blank images. 3. Generate a new version and retrain — Use the previous model as the checkpoint to preserve what it already knows.

Connecting to the Improvement Playbook

Active learning is most effective when it's targeted, not random. Use the model improvement diagnostics to decide what to collect:

See roboflow://skills/roboflow-model-improvement/SKILL for the full diagnostic decision tree.

Common Mistakes

Model Improvement Playbook

Source-of-truth note: This page ships with the Roboflow plugin. If your client has the plugin loaded, prefer the local skill (roboflow:training-and-evaluation) over fetching roboflow://skills/training-and-evaluation/improvement-playbook via ReadMcpResourceTool — the MCP resources are a fallback for non-plugin clients and may lag the source repo.

Diagnostic Decision Tree

Model not good enough?
├─ mAP/accuracy very low (<30%)?
│  ├─ Too few images → Add more data (target 500+ per class)
│  ├─ Labeling errors → Audit annotations, use AI labeling for consistency
│  └─ Wrong model type → Verify project type matches task (OD vs seg vs cls)
│
├─ High false positives (model sees objects that aren't there)?
│  ├─ Check confusion matrix → Which classes are confused?
│  │  ├─ Two classes confused → Visually similar? Merge or add distinguishing examples
│  │  └─ Background false positives → Add null/negative examples (images with no objects)
│  └─ Raise confidence threshold → Use Production Metrics Explorer optimal threshold
│
├─ High false negatives (model misses real objects)?
│  ├─ Check per-class metrics → Which classes underperform?
│  │  ├─ Specific class weak → Add more examples of that class
│  │  └─ Small objects missed → Increase training resolution, add small-object examples
│  └─ Lower confidence threshold → Trade precision for recall
│
├─ Some classes good, others bad?
│  ├─ Class imbalance → Check class distribution, add underrepresented classes
│  └─ Inconsistent labeling on weak classes → Re-label with tighter guidelines
│
└─ Plateaued after several versions?
   ├─ Try different architecture → Switch YOLO to RF-DETR or vice versa
   ├─ Try larger model size → Nano→Small, Small→Medium
   ├─ Use Universe checkpoint → Transfer learn from domain-similar model
   └─ Review augmentations → Over-augmentation can hurt; simplify

Reading the Confusion Matrix

Tip: Click any cell to see the actual images. Toggle between Ground Truth and Model Predictions to understand the failure mode.

Reading Per-Class Metrics

Common Issues & Roboflow-Specific Fixes

Insufficient Data

Class Imbalance

Wrong Augmentation

Overfitting

Signs: Training loss drops but validation loss increases or plateaus.

Architecture Switching Guide

Roboflow Instant vs Full Training

mAP Field Reference

The mAP@50 metric appears under different field names depending on the endpoint:

Both are mAP@50. Cast to number before comparing.

Iterative Improvement Checklist

1. Check evaluation -- Open Model Evaluation, review confusion matrix and per-class metrics 2. Identify weakest classes -- Sort by lowest recall/precision 3. Diagnose root cause -- Use decision tree above 4. Take action -- Add data, fix labels, adjust augmentation, or switch architecture 5. Generate new version -- New preprocessing/augmentation settings if needed 6. Train new model -- Use previous version as checkpoint if prior model was decent 7. Compare -- Check if mAP/precision/recall improved vs previous version 8. Repeat -- Target: mAP >70% for production use (domain-dependent)

Related Pages

• roboflow://skills/training-and-evaluation/active-learning — set up a production feedback loop with the Dataset Upload workflow block