Case context: You are building a cat classifier. Your dev set has 1,000 images, and your algorithm misclassified 50 of them. To improve the quality of your dev set labels, your colleague suggests reviewing only those 50 misclassified images to find and fix any incorrect labels.

Question: What is the flaw in your colleague's proposed label review strategy, and what specific type of error might remain undetected if you follow their advice?

Sample answer: The flaw in the strategy is that it ignores the 950 correctly classified images, which might also contain labeling errors. If you only review the misclassified images, you fail to detect cases where an image was incorrectly labeled and the algorithm also incorrectly predicted it as that wrong label. In this scenario, the algorithm appears to be correct because its prediction matches the label, but both are actually wrong.

Key points:

• The strategy fails to review correctly classified examples.
• It misses cases where both the label and the prediction are wrong.
• An incorrect prediction matching an incorrect label appears as a correctly classified example.

Rubric: The response must identify that the strategy ignores correctly classified examples. It must also explain the specific undetected error: when an incorrect original label matches an incorrect prediction made by the learning algorithm.