Learn Before
Explain how specific error comparisons point to data mismatch.
Question: Explain how comparing the 1.5% error on unseen training-distribution data to both the 1% training error and the 10% dev-set error isolates data mismatch as the primary problem.
Sample answer: The small gap between the 1% training error and 1.5% unseen training-distribution error indicates that the model generalizes well to the training distribution and does not suffer from high variance (overfitting). However, the large gap between the 1.5% unseen training-distribution error and the 10% dev-set error reveals that the model struggles to generalize to the dev-set distribution. Since variance is low, this massive gap is clearly caused by a mismatch between the training and dev data distributions.
Key points:
- Comparing 1% training error to 1.5% unseen same-distribution error shows low variance.
- Comparing 1.5% unseen same-distribution error to 10% dev error isolates the distribution shift.
- The conclusion is that the algorithm fails on the dev set specifically because its data distribution differs from the training data.
Rubric: The response should explicitly define the purpose of both comparisons (train vs. unseen same-distribution, unseen same-distribution vs. dev) and clearly conclude that the model generalizes to the training distribution but fails on the dev distribution.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Which error comparison most directly reveals the data mismatch problem in the example with 1% train, 1.5% same-dist, and 10% dev error?
The error pattern (1% train, 1.5% same-dist unseen, 10% dev) primarily indicates a high variance (overfitting) problem.
In the data mismatch example, the algorithm achieves _____ error on the dev set.
Match each error measurement in the data mismatch example to its correct value.
Order the analytical steps used to diagnose the data mismatch problem in the Machine Learning Yearning example.
What does the 0.5% gap between training error (1%) and same-distribution unseen error (1.5%) indicate in the data mismatch example?
In the data mismatch example, the 1% training error vs. near-perfect human performance represents the most significant problem to address.
In the data mismatch example, error on unseen data drawn from the same distribution as the training set is _____.
Match each error comparison in the data mismatch example to the type of problem it diagnoses.
Order the observations that build the case that data mismatch — not overfitting or underfitting — is the dominant problem in the Machine Learning Yearning example.
Explain how specific error comparisons point to data mismatch.
Diagnose a 10% dev-set error in a cat recognition app.
State the significance of the unseen training-distribution error.