Example: Diagnosing High Variance from a Training-to-Training-Dev Gap
Suppose the best possible error is near 0%, while a defect-detection model has 1% training error, 5% training-dev error, and 5% dev error. The four-percentage-point increase from training to training-dev error indicates high variance because the model performs substantially worse on new examples from the training distribution. The 1% training error suggests little avoidable bias, and the equal training-dev and dev errors suggest little data mismatch.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Use the Strongest Practical Human Benchmark as a Proxy
Estimating Human Error on Mobile Cat Images
Why is the optimal error rate for detecting a bicycle in a clear photo close to 0%?
If 8% of satellite images are impossible for experts to label correctly because thick cloud cover hides the target, the best achievable classification error is approximately 8%.
Human-level performance is often used as a proxy for the _____ error rate on a task.
Match each situation with the approximate best possible error rate suggested by human performance.
Order the steps for using human performance as a stand-in for the best achievable error rate and deciding whether to reduce bias.
A defect detector has 12% error, while expert annotators have 4% error. What is the avoidable bias and what should you do next?
Assuming a Zero Error Floor
In a task where people almost never miss the correct answer, the ideal error rate is nearly _____.
Match each term to its meaning when human performance is used as a practical benchmark for model error.
Order the steps for deciding whether a task has a near-zero or a much higher best-possible error rate.
How model and expert error comparison reveals bias
Selecting a Desired Error Rate Proxy for a Noisy Dictation System
What information should a person use when checking one stage of a pipeline against human performance?
Example: Diagnosing High Variance from a Training-to-Training-Dev Gap
Troubleshooting a deep learning model
Data Mismatch Problem in Deep Learning
Avoidable bias
Interpreting 10% Training, 11% Training-Dev, and 12% Development Error
Example: Diagnosing High Variance from a Training-to-Training-Dev Gap
Learn After
What main issue is suggested by 2% training error and 6% training-dev error when the best possible error is near 0%?
True or False: If training error is 2%, training-dev error is 9%, and the best achievable error is near 0%, the main issue is high bias.
A large gap between training error and training-dev error, with near-zero optimal error, indicates high _____.
Match the Error Metrics in a Variance Check
Order the steps for identifying the main error source when given training, training-dev, and dev error values.
Which gap points most directly to high variance when training error is 2%, training-dev error is 6%, and dev error is 6%?
True or False: In a diagnostic example of high variance, the model has 2% training error, 6% training-dev error, and 6% dev error.
The avoidable _____ is about 1% here because the training error (2%) is close to the best achievable error (~1%).
Match each diagnostic gap to the model issue it reveals in a retail demand forecast analysis.
Order the reasoning steps that support the conclusion that this model has high variance.
Interpreting Training and Training-Dev Errors for Variance
Diagnosing Variance in a Warehouse Package Sorter
How to Spot High Variance from a Gap in Errors