What are the biggest hidden failure modes in popular computer vision datasets that don’t show up in benchmark metrics?

[shaipai]

I’ve been working with standard computer vision datasets (object detection, segmentation, and OCR), and something I keep noticing is that models can score very well on benchmarks but still fail badly in real-world deployments.

I’m curious about issues that aren’t obvious from accuracy or mAP, such as:

• Dataset artifacts or shortcuts models exploit
• Annotation inconsistencies that only appear at scale
• Domain leakage between train/test splits
• Bias introduced by data...

Read more

Continue reading...