Limiting False Positives in Research

Example of a Non-Significant Result: Nutritional Supplement

Criticisms of Null Hypothesis Testing

In psychological research, a finding is typically considered 'statistically significant' if the probability that the observed difference occurred by random chance is:

According to the standard criteria in psychology research, a result with a 10% probability of occurring by random chance would be classified as statistically significant.

A team of researchers has completed four separate psychological studies. Match each study's calculated p-value to the correct interpretation of its statistical significance based on the standard 5% threshold.

A researcher is evaluating the strength of evidence provided by four separate studies. Arrange these findings in order from the highest level of confidence to the lowest level of confidence that the observed result represents a meaningful effect rather than random chance, based on the standard threshold for statistical significance in psychology.

Match each term related to the standard of statistical significance in psychology with its correct description according to the course definition.

In psychological research, what does it mean to conclude that a finding is 'statistically significant' at the standard $5\%$ level?

A researcher finds that a treatment group performed better than a control group, with a probability of $0.03$ that the result was due to random chance. Since this meets the standard threshold for significance, the researcher concludes that the findings are _____ rather than a fluke.

A neuropsychologist compares reaction times between two groups and finds a difference. Statistical analysis shows there is a $4\%$ probability that this difference occurred by random chance. According to standard criteria in psychological research, this finding should be classified as statistically significant.

A psychologist conducting an experiment on learning techniques must analyze their data to determine if the difference in test scores between two groups is statistically significant. Arrange the steps of this analytical process in the correct logical and chronological order, starting with the baseline assumption and ending with the final research conclusion.

An educational psychologist peer-reviews a manuscript investigating a new study tool. The manuscript's authors found that students using the tool scored higher than those who did not, with a calculated probability of $0.058$ (or $5.8\%$) that the difference was due to random chance. The authors conclude that their tool's benefits are statistically significant. When evaluating this claim against the standard threshold for statistical significance in psychological research, the peer reviewer should judge the authors' conclusion to be _______ (justified/unjustified).