Learn Before
Fix Dev and Test Set Labels Together
Whatever process is used to fix dev set labels should also be applied to test set labels so that the dev and test sets continue to be drawn from the same distribution. Fixing them together helps avoid optimizing dev-set performance only to be judged later on a different test-set criterion.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Adding a Mislabeled Category to the Error Analysis Spreadsheet
When to Fix Mislabeled Dev Set Labels
Fix Dev and Test Set Labels Together
What does the term "mislabeled" mean in the context of error analysis on a dev set?
True or False: Mislabeled examples in the dev set refer to errors made by the algorithm.
In dev set error analysis, a "mislabeled" example has an incorrect class label _____ before the algorithm runs.
Match each component of a mislabeled example with its correct definition.
Order the sequence of events leading to the discovery of a mislabeled dev set example.
Explain the concept of mislabeled examples in the dev set during error analysis.
Diagnose the data issue in a cat classifier's dev set.
What is the origin of a mislabeled example in a dev set?
Which of the following is an example of a mislabeled data point (x, y) in a cat classification dev set?
True or False: In a mislabeled example (x, y), the class label y has an incorrect value.
Learn After
Why must the same label-fixing process applied to the dev set also be applied to the test set?
Fixing only dev set labels without applying the same process to the test set can cause the two sets to be drawn from different distributions.
Whatever process you apply to fixing dev set labels, you must also apply it to the _____ labels.
Match each label-fixing scenario to its consequence for dev/test set evaluation.
Order the steps for correctly fixing mislabeled examples while keeping dev and test sets from the same distribution.
What is the primary risk when a team optimizes against a dev set whose labels were fixed differently from the test set?
Applying different label-fixing methods to dev and test sets is acceptable as long as both sets achieve high overall label accuracy.
Fixing dev and test set labels together prevents the team from optimizing for dev set performance only to be judged on a _____ criterion.
Match each key concept from the label-fixing principle to its correct definition.
Order the events that lead to misaligned evaluation when only dev set labels are fixed and not test set labels.
Analyze the consequences of using different label-fixing processes for the development and test sets in a machine learning project.
Diagnose the evaluation issue when a team corrects dog breed labels in a development set but leaves the test set unchanged.
State the primary reason for maintaining consistency in the label-fixing process across both dev and test sets.