Accuracy Resolution on a 100-Example Development Set
For ordinary unweighted accuracy on 100 examples, each additional correct prediction changes the measured accuracy by 1 percentage point. The set therefore cannot directly register a 0.1-percentage-point difference such as 88.0% versus 88.1%, and sampling variation makes such a tiny underlying gain unreliable to infer from only 100 cases.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
Typical Development Set Sizes for Tiny Accuracy Gains
High-Impact Business Systems May Justify a Larger Development Set
Formal significance tests for validation-set changes
What dev set size is most suitable for spotting a 0.1 percentage-point gain in accuracy?
A development set should always be expanded to the maximum possible size, even after it is already large enough to reveal meaningful performance changes.
Validation set size for noticing a tiny accuracy change
Match each evaluation target with the dev set size it suggests.
Order the steps for deciding whether a dev set is large enough to detect a useful accuracy gain.
Match dev-set size to the smallest gain you care about.
Choose a dev set size that can detect a tiny but important gain.
Why is a 150-example dev set not enough to tell 83.0% from 83.4% accuracy?
When is a validation set much larger than 10,000 examples most justified?
If a validation set is already large enough to tell whether one model is meaningfully better than another, it does not need to be made much larger.
Sizing a Training Development Set
Accuracy Resolution on a 100-Example Development Set
Learn After
What can a 100-example validation set tell you about 84.0% versus 84.1% accuracy?
Can a dev set with 100 examples reliably detect a 0.1 percentage-point accuracy gap?
Small Validation Sets Miss Tiny Accuracy Changes
Match each element in the comparison with its meaning.
Order the steps for judging whether a validation set can detect a small accuracy gap.
Limits of a 64-Example Development Set
Can a tiny validation set resolve a 0.1-point accuracy gap?
Why a 100-item validation set can miss a tiny gain
What is the clearest sign that the validation set is too small for this comparison?
A development set with 100 examples is usually small.