How to Choose the Right Dataset Format for Your ML Project
Choose your dataset format by working backwards from the model and framework you'll train with: match the format to the task (detection, segmentation, classification, or robot learning), then to the tooling that reads it natively, then to your scale. COCO JSON is the safe default for detection and instance segmentation, YOLO for real-time detectors, PNG masks for semantic segmentation, and LeRobot/RLDS for robot-learning-and keep a versioned source of truth you can convert from, so the format never locks you in.
A dataset format sounds like a boring, late-stage detail-until a mismatched one costs you a week of conversion scripts, a silent label-drop bug, or a training run that quietly ignores half your annotations. The format is the contract between your labeled data and your training code: it decides what information is preserved (boxes, masks, keypoints, attributes, tracks), how splits are expressed, and which frameworks can read it without custom tooling. This guide walks through the formats that actually matter in computer vision and ML, how to choose by task and framework, the trade-offs of each, and how to avoid lock-in by keeping a convertible source of truth. Baaz has delivered datasets across detection, segmentation, keypoints, and robot-learning formats through our managed annotation service, Payana (payana.ai). This is the decision framework we use with clients before a single image is labeled.
Why the Dataset Format Matters More Than Teams Expect
A format is not just a file layout-it is a decision about what gets preserved. A bounding-box format silently discards the pixel-accurate masks you paid for. A classification format has nowhere to put object locations. A format without an attribute field drops the truncated/occluded flags a tracking model needs. Choose wrong and you either re-annotate or lose signal.
The second cost is tooling friction. Every hour spent writing and debugging a converter is an hour not spent training. Frameworks read some formats natively and choke on others, and a subtly wrong converter (off-by-one class indices, unnormalized coordinates, RGB-vs-index mask confusion) fails silently-your model just trains on garbage and you find out at validation.
The third is reproducibility. If your split lives in a spreadsheet or a one-off script rather than in the dataset itself, next quarter's retrain is not comparable to this quarter's. A good format carries its train/valid/test split with it.
What Are the Main Dataset Formats in Computer Vision?
Most vision projects land on one of a handful of formats. Here is what each is for and where it fits.
| Format | Best for | Notes |
|---|---|---|
| COCO JSON | Object detection, instance segmentation, keypoints | One JSON with images, annotations, categories - the de-facto interchange format |
| YOLO (txt + data.yaml) | Real-time detection (YOLO family) | One .txt per image, normalized xywh, class index - minimal and fast to load |
| Pascal VOC (XML) | Detection, legacy pipelines | One XML per image - verbose but human-readable and widely supported |
| PNG masks | Semantic segmentation | Index or color masks aligned to the image - simple, framework-agnostic |
| Classification folders / CSV | Image classification | Class-per-folder or an image,label CSV - the simplest format there is |
| TFRecord | Large-scale TensorFlow pipelines | Sharded binary records for fast, streaming input - TF-specific |
| LeRobot / RLDS | Robot learning, embodied AI | Episode-structured trajectories (observations + actions) - for policy training |
How Do You Choose a Format for Your Task?
Start from the task, because the task dictates what information the format must carry. Getting this right eliminates most candidates immediately:
- Object detection - COCO JSON or YOLO. Use YOLO if you're training a YOLO-family model - COCO if you want the broadest tooling compatibility.
- Instance segmentation - COCO JSON (polygon or RLE masks). It carries per-object masks and categories in one file.
- Semantic segmentation - PNG index masks alongside the images. Every framework can read them, and there's no ambiguity about pixel classes.
- Image classification - class-per-folder directories or a simple CSV. Anything heavier is over-engineering.
- Pose / keypoints - COCO keypoints schema, which encodes skeletons per object per image.
- Robot learning - LeRobot or RLDS / Open X-Embodiment, which structure data as episodes of observations and actions rather than independent frames.
Match the Format to Your Framework and Scale
Once the task narrows the field, let your framework break the tie. Detectron2 and MMDetection read COCO natively. The Ultralytics YOLO stack expects YOLO txt + data.yaml. TensorFlow object-detection pipelines favor TFRecord. PyTorch is format-agnostic but happiest with COCO via standard dataset classes. Pick the format your training code already understands and you delete an entire class of conversion bugs.
Scale is the other axis. A few thousand images? Plain JSON, XML, or PNG masks on disk are fine, and their human-readability is a feature. Millions of images with GPU-bound throughput? A sharded binary format like TFRecord (or WebDataset / Parquet) keeps the input pipeline from starving the GPU. Don't reach for sharded binaries before you need them-you trade away debuggability for I/O you may not be bottlenecked on.
How Do You Avoid Format Lock-In?
The formats you train on will change-new model, new framework, a collaborator on a different stack. The way to stay flexible is to treat one representation as your source of truth and generate the rest from it:
- Keep a rich, lossless source of truth. Store the most information-dense representation you have (masks, attributes, tracks, class hierarchy)-usually COCO-style-so no signal is thrown away.
- Generate training formats on export. Produce YOLO, Pascal VOC, PNG masks, or TFRecord as derived artifacts from the source, never by hand-editing the training files.
- Version every export. Each snapshot should be reproducible and carry its own train/valid/test split, so a model trained six months ago can be rebuilt byte-for-byte.
- Validate after conversion. Visualize a sample of the converted labels over the images-class indices, coordinate normalization, and mask encoding are the usual silent failure points.
Common Format Mistakes That Silently Corrupt Training
Most format failures don't crash-they degrade the model quietly. Watch for these:
- Coordinate convention mismatches: absolute pixels vs. normalized, xywh vs. xyxy, top-left vs. center. A converter that assumes the wrong one shifts every box.
- Off-by-one or remapped class indices: a 0-indexed vs. 1-indexed class list, or categories reordered between export and training config.
- Mask encoding confusion: RGB color masks read as index masks (or vice versa), collapsing multiple classes into one.
- Split leakage: frames from the same video or the same field split across train and test, inflating validation accuracy.
- Dropped attributes: occlusion, truncation, or track IDs lost in conversion, quietly disabling the model behavior they were meant to teach.
Related guides, services & programmes
Frequently Asked Questions
For most teams, COCO JSON is the safest default-it's read natively by Detectron2, MMDetection, and most PyTorch pipelines, and it carries boxes, masks, and keypoints. If you're specifically training a YOLO-family model, use the YOLO txt + data.yaml format, which those tools expect directly.
COCO stores all annotations in a single JSON file with absolute pixel coordinates and supports detection, instance segmentation, and keypoints. YOLO uses one .txt file per image with class index and normalized center-x, center-y, width, height, and is optimized for the YOLO detector family. COCO is richer and more portable. YOLO is leaner and faster to load for its target models.
PNG index masks aligned to each image are the most portable choice-every framework can read them and there's no ambiguity about which pixel belongs to which class. Keep the class-to-index mapping documented alongside the masks so no one confuses index masks with RGB color masks.
Keep a single, information-rich source of truth (usually COCO-style, preserving masks, attributes, and tracks) and generate every training format-YOLO, VOC, PNG masks, TFRecord-as versioned exports from it. Never hand-edit the training files, and validate a sample after each conversion.
Reach for a sharded binary format (TFRecord, WebDataset, or Parquet) when your dataset is large enough that disk I/O starves the GPU during training-typically hundreds of thousands to millions of samples. Below that, human-readable formats like JSON and PNG masks are easier to debug and inspect, and the I/O savings won't matter.
Want this distilled for your roadmap? Brief the Baaz squad or scan ship evidence in case studies.