Google

Although one may first formulate categories such as Dog, Great cat, and Blurry and then categorize examples by hand, looking through examples will probably inspire new error categories. For example, after seeing that many mistakes occur with Instagram-filtered pictures, one can add a new Instagram column to the spreadsheet. Manually looking at misclassified examples and asking how or whether a human could have labeled the picture correctly will often inspire new categories of errors and solutions.

Discovering New Error Categories While Reviewing Examples

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

What commonly happens when you manually examine misclassified examples during error analysis?

Error categories must be fully finalized before you begin examining any misclassified examples.

After noticing many misclassified examples involve Instagram-filtered pictures, you should add a new _____ to the error analysis spreadsheet.

Match each error analysis action to its primary purpose in discovering new error categories.

Order the steps of Ng's iterative error category discovery process during error analysis.

Which question should you ask when reviewing a misclassified image to inspire new error categories and solutions?

The Instagram error category illustrates how manual review can reveal categories absent from the original error analysis framework.

Manually looking at examples that the algorithm _____ and asking whether a human could label them correctly often inspires new error categories.

Match each term from Ng's error analysis framework to its correct description.

Order the reasoning steps a practitioner follows when a new error pattern is spotted during manual example review.

Question:
Explain why an error analysis framework shouldn't remain static after the initial categories are defined. How does the manual review process drive the evolution of these categories, and what impact might this have on developing solutions?

Sample answer:
An error analysis framework should remain dynamic because initial categories are often based on prior assumptions rather than the model's actual failure modes. By manually reviewing misclassified examples, a practitioner can discover unforeseen patterns, such as the Instagram-filter issue, which inspire new error categories. Asking whether a human could label the misclassified examples correctly helps identify specific areas for improvement, directly leading to new, targeted solutions rather than just quantifying errors.

Key points:
- Initial categories may not capture all error modes.
- Manual review of examples reveals new patterns.
- Adding new categories (e.g., Instagram filters) to the spreadsheet.
- Evaluating human performance on misclassifications inspires new solutions.

Rubric:
The response must explain that initial categories are often incomplete. It should mention that manual review uncovers unanticipated error patterns. It needs to describe how asking if a human can correctly label the example leads to new categories and inspires actionable solutions.

Dynamic Error Analysis Frameworks

Case context:
You have defined three initial error categories for a bird classification algorithm: 'Lighting issues', 'Small subject', and 'Blurry'. After categorizing 50 misclassified images, you notice that 15 of them involve birds partially hidden behind leaves or branches, which doesn't perfectly fit the initial categories.

Question:
Based on Ng's recommendations for reviewing examples, what should your next immediate step be to improve your error analysis process, and what question should you ask yourself about these 15 images?

Sample answer:
I should add a new category column to my spreadsheet, such as 'Occluded' or 'Hidden by foliage', to capture this specific pattern. While reviewing these 15 images, I should also ask myself how or whether a human could have correctly labeled the bird despite the obstruction, in order to inspire a potential solution.

Key points:
- Add a new error category column (e.g., occlusion) to the spreadsheet.
- Formulate the new category based on the observed pattern during manual review.
- Ask how or whether a human could correctly label the misclassified image.

Rubric:
The response should explicitly state adding a new error category to the spreadsheet. It should also state asking if or how a human could label the images correctly.

Uncovering Hidden Error Patterns

Question:
When manually looking at examples misclassified by an algorithm, what specific question should you ask yourself to help inspire new error categories and solutions?

Sample answer:
You should ask how or whether a human could have labeled the picture correctly.

Key points:
- Ask if a human could have labeled the picture correctly.
- Ask how a human could have labeled the picture correctly.

Rubric:
The answer must identify the question of whether or how a human could correctly label the misclassified example.

Learn Before

Related