Learn Before
Nominal Level, Branch-Conditional Error and Whole-Procedure Error
When an analysis includes a data-dependent choice, three error rates are easy to confuse. The error probabilities below are evaluated under the same specified model in which the null hypothesis is true and the stated assumptions hold.
- The nominal level (alpha), for example .05, is the target, or upper bound, for false rejection that a test is designed to meet when it is chosen in advance. A valid test keeps its false-rejection probability at or below alpha; it attains alpha exactly only when it is exactly calibrated.
- A branch-conditional rate is the probability of false rejection given that the data led to one branch of a procedure, for example the datasets for which a preliminary check selected a particular test. It is defined only for a branch that has a positive probability of occurring.
- The whole-procedure rate is the probability that the complete procedure, including every data-dependent choice, ends in rejection. When the branches are mutually exclusive and together cover every possible dataset, it is the sum of the branch rates, each weighted by the probability of its branch: , where the sum runs over the branches that have positive probability. Branches with probability zero are omitted, because their conditional rates are undefined.
Data-dependent choices can move branch rates above or below the nominal level, and the whole-procedure rate can end up above, below or close to it, so it is the complete procedure that has to be evaluated. A whole-procedure rate below the target, such as .04 against .05, is conservative, not too liberal.
(Advanced, optional: when the null hypothesis covers many possible models, a guarantee at level alpha has to hold for every one of them, not just for one convenient case.)
Background on marginal and conditional probability: Goodfellow, Bengio and Courville, Deep Learning (2016), sections 3.4–3.5.
0
1
Contributors are:
Tags
KPU
Research Methods in Psychology - 4th American Edition @ KPU
1Cademy
Related
File Drawer Problem
p-hacking
Replication of Studies in Psychology
Example of Type I and Type II Errors
In null hypothesis testing, which of the following best defines a Type I error?
A researcher evaluates a new cognitive training program that, in reality, has no effect on memory. Due to an unusual sample, the statistical analysis produces a significant result, causing the researcher to incorrectly conclude that the program works. This situation describes a Type I error.
To understand a Type I error, one must distinguish between the true state of the population and the decision made by the researcher. Match each component of a Type I error to the description that best explains its role.
A Type I error is the result of a specific logical failure during the hypothesis-testing process. Arrange the following events in the correct order to illustrate the progression of a Type I error, starting from the actual state of the population to the researcher's final conclusion.
You are designing a computer simulation to help students visualize the logic of statistical decision-making in psychology. To successfully create a scenario where the software can generate a Type I error, which combination of population characteristics and decision rules must you program into the model?
In psychological research, a Type I error is also known as a 'false positive.'
A researcher must decide between two significance levels for a study on a new behavioral therapy. They evaluate the trade-offs and conclude that it is more damaging to give patients 'false hope' with a treatment that does not work than to miss a potentially helpful therapy. To align with this evaluation, the researcher selects a lower level to minimize the probability of a _____.
Which of the following statements best explains why a researcher might commit a Type I error, even if their study has no design flaws or bias?
Dr. Carter conducts an experiment to see if listening to classical music while studying improves test scores. In reality, classical music has no effect on test scores (the null hypothesis is true in the population). However, due to random sampling error, Dr. Carter's sample happens to perform extremely well, resulting in a statistically significant difference with . Dr. Carter rejects the null hypothesis and concludes that classical music improves test scores. In this scenario, Dr. Ca
A research team is planning several methodological approaches to a study. Analyze how each design choice or statistical scenario affects the probability of committing a Type I error, and match the scenario to its corresponding impact on the Type I error rate.
Nominal Level, Branch-Conditional Error and Whole-Procedure Error