Google

During error analysis, one may notice that some examples in the dev set are mislabeled. Here, "mislabeled" means that the pictures were already mislabeled by a human labeler before the algorithm encountered them; that is, in an example $(x, y)$, the class label $y$ has an incorrect value. For example, some pictures that are not cats may be mislabeled as containing a cat, and vice versa.

Mislabeled Examples in the Dev Set

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

If one suspects that the fraction of mislabeled images is significant, one should add a Mislabeled category to keep track of the fraction of examples that are mislabeled. The shown columns are Image, Dog, Great cat, Blurry, Mislabeled, and Comments.

Adding a Mislabeled Category to the Error Analysis Spreadsheet

Mislabeled dev set labels are worth fixing when the fraction of the dev set that is mislabeled impedes the ability to quickly evaluate algorithms and decide whether Algorithm A or B is better.

When to Fix Mislabeled Dev Set Labels

Whatever process is used to fix dev set labels should also be applied to test set labels so that the dev and test sets continue to be drawn from the same distribution. Fixing them together helps avoid optimizing dev-set performance only to be judged later on a different test-set criterion.

Fix Dev and Test Set Labels Together

What does the term "mislabeled" mean in the context of error analysis on a dev set?

True or False: Mislabeled examples in the dev set refer to errors made by the algorithm.

In dev set error analysis, a "mislabeled" example has an incorrect class label _____ before the algorithm runs.

Match each component of a mislabeled example with its correct definition.

Order the sequence of events leading to the discovery of a mislabeled dev set example.

Question:
Based on Andrew Ng's Machine Learning Yearning course, define what "mislabeled" means when inspecting a dev set, specify which component of the data point (x, y) is affected, and provide a concrete example of how this occurs.

Sample answer:
In the context of dev set error analysis, "mislabeled" refers to examples that were assigned incorrect labels by a human labeler before the machine learning algorithm processed them. In a labeled example (x, y), this means the class label y has an incorrect value. For example, a picture that does not contain a cat might be incorrectly labeled as containing a cat, or a picture that does contain a cat might be labeled as not containing one.

Key points:
- Mislabeled examples are incorrect due to human labeler error before the algorithm encounters them.
- In a data point (x, y), the class label y has an incorrect value.
- Pictures not containing a cat may be labeled as a cat, or vice versa.

Rubric:
The answer should define "mislabeled" as an error introduced by human labelers before the algorithm encounters the data. It must specify that the class label y in (x, y) has the incorrect value. It must also provide an example of mislabeling (e.g., cat vs. non-cat pictures).

Explain the concept of mislabeled examples in the dev set during error analysis.

Case context:
You are developing a cat classifier. During error analysis of the dev set, you notice a picture of a dog that the algorithm correctly predicted as "not cat", but it was marked as an error because the target label was set to "cat".

Question:
Diagnose the nature of this error. What is this type of issue called, who caused it, and what specific part of the data point (x, y) is incorrect?

Sample answer:
This issue is a "mislabeled" example. The error was caused by a human labeler who assigned the label before the algorithm encountered the picture. Specifically, in the data point (x, y), the class label y was set to "cat" (incorrect value) instead of "not cat", while the input x is the picture of the dog.

Key points:
- Identify the issue as a "mislabeled" example.
- The error was introduced by a human labeler before the algorithm encountered the data.
- The class label y in (x, y) has an incorrect value.

Rubric:
The candidate must identify the issue as a "mislabeled" example. They must state that the error was made by a human labeler before the algorithm encountered it. They must specify that the class label y is the part of the data point (x, y) that has the incorrect value.

Diagnose the data issue in a cat classifier's dev set.

Question:
According to the source text, who introduces the error in a "mislabeled" dev set example and when does this happen relative to the algorithm's execution?

Sample answer:
A human labeler introduces the error. This happens before the algorithm encounters the example.

Key points:
- The error is introduced by a human labeler.
- It occurs before the machine learning algorithm encounters the example.

Rubric:
The answer must state that the human labeler introduced the error and that this occurred before the algorithm processed the data.

Learn Before

Related