Google

With about 100 Eyeball dev set mistakes, manual review gives a very good sense of the major error sources. Reviewing even more errors can be fine when enough data is available.

One Hundred Eyeball Dev Set Errors Give Very Good Error-Source Sense

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

Sufficient Mistakes for Error Sense

Harm in Analyzing More Errors

Baseline Number for _____ Mistakes

Matching Error Quantities to Concepts

Decision Process for Eyeball Dev Set Review

Question:
Discuss the recommended baseline number of errors to manually review in an Eyeball dev set and explain the condition under which reviewing significantly more errors is considered acceptable.

Sample answer:
The recommended baseline is to manually review approximately 100 mistakes in an Eyeball dev set. This amount provides a very good sense of the major sources of errors the algorithm is making. However, some practitioners choose to analyze even more errors, sometimes up to 500. Doing so is acceptable and causes no harm as long as there is enough data available to support the expanded review.

Key points:
- Baseline of ~100 mistakes
- Provides a very good sense of major error sources
- Can review more errors (e.g., 500)
- Must have enough data to review more

Rubric:
A strong answer will explicitly state the baseline of 100 mistakes, explain its purpose (finding major error sources), and correctly identify the condition for reviewing more errors (having enough data).

Analyzing Eyeball Dev Set Error Volumes

Case context:
A data science team is debugging their image classifier. They have a massive dataset and the model is generating thousands of errors. The lead engineer wants to manually analyze 500 errors to get an extremely detailed breakdown of the failure modes, but a junior engineer argues they only need to look at 100.

Question:
Based on best practices for Eyeball dev sets, evaluate the lead engineer's proposal. Is it acceptable, and why?

Sample answer:
The lead engineer's proposal is acceptable. While reviewing about 100 mistakes is sufficient to get a very good sense of the major sources of errors, manually analyzing more errors (such as 500) is perfectly fine. The key condition for doing this without harm is having enough data, which the team currently possesses with their massive dataset.

Key points:
- 100 mistakes is enough for major error sources
- Analyzing 500 errors is acceptable
- The condition of having enough data is met

Rubric:
The answer should validate that looking at 100 mistakes is the baseline, but clearly state that looking at 500 is acceptable provided there is enough data.

Scaling Up Manual Review

Question:
What is the primary benefit of manually reviewing about 100 mistakes in your Eyeball dev set?

Sample answer:
The primary benefit is that it gives you a very good sense of the major sources of errors that your algorithm is making.

Key points:
- Gives a very good sense
- Identifies major sources of errors

Rubric:
Credit is given for mentioning that it reveals the major sources of errors.

Learn Before

Related