Case context: You are building a classifier. You gather a sample of 100 dev set examples that your system misclassified. During manual error analysis, you count that 60% of these misclassified examples are dog images, whereas only 5% of them are cat images.

Question: Based on the manual review of these 100 examples, which type of error should you prioritize fixing, and how does the error analysis justify this decision?

Sample answer: You should prioritize fixing dog image misclassifications. The manual error analysis shows that dog images account for 60% of the error sample, while cat images account for only 5%. Prioritizing the dog category offers a significantly larger potential reduction in the overall error rate.

Key points:

• Identify dog images as the largest category of error (60% vs 5% for cats)
• Choose to prioritize dog images for fixing
• Ground the decision in the manual error analysis counts

Rubric: The answer must identify dog images as the priority category and justify this choice by pointing out that they account for a much higher fraction (60%) of the errors in the manual sample of 100 misclassified examples compared to cats (5%).