Bird-Identification App Dev/Test Distribution Mismatch
A bird-identification app is evaluated on a dev/test set containing mostly clear daytime photos of robins and finches, while users mainly upload dim phone photos of backyard birds at dusk. Because the evaluation and real-user inputs come from different distributions, dev/test performance may give a misleading picture of deployment performance.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
If a dev/test set does not reflect the distribution the model will face after deployment, what should the team do?
True or False: A validation set drawn from a very different population than deployment data can still reliably show which model changes will help in the real world.
When the validation set does not match deployment conditions, the recommended action is to _____ the validation set.
Identify the distribution concepts used in model evaluation.
Arrange the actions a team should take when a validation set stops predicting real-world performance.
Why a Non-Representative Dev/Test Set Can Mislead Model Improvement
True or False: Evaluation data should match the conditions the final system will face in practice.
Core idea about dev and test data
Match each situation to the best action for dev/test data.
Order the reasoning steps showing why a development set that misses the real distribution should be revised.
Why mismatched evaluation data can mislead a project
Diagnose a field-to-test mismatch in apple inspection photos
What should a team do if its evaluation data no longer reflects real-world use?
Bird-Identification App Dev/Test Distribution Mismatch
Learn After
Recognizing a Dev/Test Distribution Shift
A video app can keep using a test set made only of desktop screen recordings after most users switch to short phone clips.
A Dev/Test Set Should Match _____ Data
Distribution Shift in a Wildlife Classifier
Finding a Dataset Distribution Mismatch
Why a Test Set Can Fail to Reflect Real Usage
The Shift in Field Photos
Validation Data and Production Shift
What the dev/test set says about deployment data
When the Evaluation Set No Longer Matches User Behavior