A Same-Distribution Dev/Test Gap Suggests Dev-Set Overfitting
Repeated model selection or tuning on a development set can make decisions overly tailored to that set. If performance is then much worse on an independent test set drawn from the same distribution and evaluated in the same way, distribution mismatch is less plausible, so overfitting to the development set is the likely diagnosis. A practical remedy is to collect more development-set examples.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
Make the Dev Set Match the Main Improvement Goal
Why Different Dev and Test Distributions Complicate Diagnosis
Mismatched Validation and Test Splits Can Make Chance Matter More
If a model is tuned on a development set and then performs worse on a separate test set, even though both sets come from the same source distribution, what is the most likely explanation?
True or False: If validation and test data are drawn from different populations, a score gap between them always has one clear cause.
If a model has started fitting the validation set too closely and the training and validation data come from the same distribution, the usual remedy is to get more _____ data.
Why should the development set match the main goal of the project?
When validation and test data come from the same distribution, a strong validation score followed by a much weaker test score suggests overfitting to the validation set.
If the development set is being overused and the training and development data come from the same distribution, the practical fix is to collect more _____ data.
Match each development-and-test-set situation with its consequence for debugging.
Order the troubleshooting steps when validation performance is strong but holdout performance is weak.
Why a Test Set Can Score Lower Than a Dev Set
A test score from a different data distribution tells you exactly why the model failed.
After the development and test sets are set, the team will spend its effort improving _____ set performance.
Match each concept about dev and test set distributions to the correct description.
How to Choose Dev and Test Sets for Reliable Iteration
Explain what poor final performance means when the tuning set and final set do or do not match.
Why a model can look strong in development but weak in final evaluation
What is the diagnosis and remedy when test results lag far behind dev results under the same data distribution?
A Same-Distribution Dev/Test Gap Suggests Dev-Set Overfitting
Overfitting-Underfit Example:
A Same-Distribution Dev/Test Gap Suggests Dev-Set Overfitting
Learn After
A model does well on the validation set but performs poorly on the final test set, and both sets come from the same process. What is the most likely diagnosis?
True or False: If validation and test data come from the same source, a large gap between validation and test scores always proves the model has too many layers.
The practical fix for overfitting the validation set is to gather more _____ examples.
Match each performance pattern to its best diagnosis
Order the steps for diagnosing a validation/holdout performance gap from the same data source
Why a Shared-Distribution Validation/Test Gap Signals Validation Overfitting
Diagnose a dev/test gap when both evaluation sets come from the same source
Why Dev and Test Sets Should Match
What is the practical fix when a model has fit the development set too closely?
A gap between validation and test performance always means you should add more test examples.