Typical Development Set Sizes for Tiny Accuracy Gains
Development sets with about 1,000 to 10,000 examples are common. A set near 10,000 examples gives a reasonable chance of detecting a 0.1% improvement in accuracy.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
Typical Development Set Sizes for Tiny Accuracy Gains
High-Impact Business Systems May Justify a Larger Development Set
Formal significance tests for validation-set changes
What dev set size is most suitable for spotting a 0.1 percentage-point gain in accuracy?
A development set should always be expanded to the maximum possible size, even after it is already large enough to reveal meaningful performance changes.
Validation set size for noticing a tiny accuracy change
Match each evaluation target with the dev set size it suggests.
Order the steps for deciding whether a dev set is large enough to detect a useful accuracy gain.
Match dev-set size to the smallest gain you care about.
Choose a dev set size that can detect a tiny but important gain.
Why is a 150-example dev set not enough to tell 83.0% from 83.4% accuracy?
When is a validation set much larger than 10,000 examples most justified?
If a validation set is already large enough to tell whether one model is meaningfully better than another, it does not need to be made much larger.
Sizing a Training Development Set
Accuracy Resolution on a 100-Example Development Set
Learn After
What dev set size is commonly suggested for detecting a 0.1% improvement in accuracy?
True or False: A validation set with 1,000 examples is enough to reliably detect a 0.1% improvement in accuracy.
Development sets with _____ to 10,000 examples are often considered common.
Which dev set size range is commonly used in practical machine learning projects?
True or False: A dev set with 10,000 examples can often reveal an accuracy gain of about 0.1 percentage point.
Validation Set Size for Tiny Accuracy Gains
Match each development-set idea to its best description.
Put the validation-set sizing logic in order.
What size of accuracy change is a 10,000-example dev set generally able to detect?
True or False: A development set with 1,000 examples is smaller than the commonly suggested range for spotting very small improvements.
Common development-set sizes begin at _____ examples.
Match each development-set situation to the most accurate classification.
Arrange the steps for deciding when a dev set must be large enough to notice a tiny metric gain.
Why Dev Set Size Matters for Detecting Small Gains
Selecting a Dev Set Size for a Small Accuracy Gain
Choosing a Development Set for Tiny Metric Gains