Learn Before
p-hacking
p-hacking is a problematic research practice where researchers make various arbitrary analytical decisions to artificially inflate their chances of obtaining a statistically significant result. By selectively reporting dependent variables, unjustified removal of outliers, or selectively presenting only significant findings, researchers continuously manipulate their data until it yields a desirable -value. This practice severely compromises the reliability of published results by creating an unacceptably high rate of Type I errors in the psychological literature.
0
1
Tags
KPU
Research Methods in Psychology - 4th American Edition @ KPU
Related
Don't make such black and white interpretations
Factors Determining the p-value
Test Statistic
Misinterpretations of the p-value
Alpha (Level of Significance)
p-hacking
The 2015 Ban on Null Hypothesis Testing
Alpha Criterion
What does a p value represent in the context of psychological research?
If a psychology researcher calculates a p-value of 0.08, they will typically declare the experimental result to be statistically significant.
A social psychologist is analyzing data from four different studies. Based on the standard thresholds for statistical significance provided in the table, match each study's obtained p value with its correct interpretation.
Misinterpretation of the p Value
Determinants of the Value
A researcher obtains a value of and labels the result 'highly significant' (). According to standard significance thresholds, where and , this labeling is _____.
A cognitive psychologist conducts an experiment on learning styles and calculates a value of . Applying the standard threshold of significance () described in the text, the researcher should declare this result to be statistically significant.
Define what a value represents in the context of an experiment and state the standard threshold for declaring an experimental result to be statistically significant.
Reporting Effect Sizes in Null Hypothesis Significance Testing
Using the convention stated in this question for marking results with asterisks (a cut-off value itself counts as meeting it), match each label with the range of values it marks.
By a common convention in psychology, a result is declared statistically significant when . For a result exactly at this threshold (), results at least as extreme would occur, under the test's model (the null hypothesis plus the test's assumptions), time in how many?
By a common convention in psychology, a result is declared statistically significant when . For a result exactly at this threshold (), if the null hypothesis and the test's assumptions were true, results at least as extreme would occur 1 time in _____.
Arrange the following three values in ascending numerical order, from the smallest (1) to the largest (3).
A psychologist has values from four different studies. Arrange the values in ascending numerical order, from the smallest to the largest. (This orders the numbers only: values from different studies, each calculated under its own model, do not by themselves rank how strong the evidence is.)
A psychology student runs an analysis on their study's data and obtains a value of . Using the common convention of a significance level of (a result is significant when ), what conclusion should the student draw about their experimental result, and why?
Based on the definition of a value and a significance level of set in advance, explain in your own words what it means if, assuming the null hypothesis and the test's assumptions were true, a result at least as extreme as the researchers' would occur less than 1 time in 20, and what this does not establish.
File Drawer Problem
p-hacking
Replication of Studies in Psychology
Example of Type I and Type II Errors
In null hypothesis testing, which of the following best defines a Type I error?
A researcher evaluates a new cognitive training program that, in reality, has no effect on memory. Due to an unusual sample, the statistical analysis produces a significant result, causing the researcher to incorrectly conclude that the program works. This situation describes a Type I error.
To understand a Type I error, one must distinguish between the true state of the population and the decision made by the researcher. Match each component of a Type I error to the description that best explains its role.
A Type I error is the result of a specific logical failure during the hypothesis-testing process. Arrange the following events in the correct order to illustrate the progression of a Type I error, starting from the actual state of the population to the researcher's final conclusion.
You are designing a computer simulation to help students visualize the logic of statistical decision-making in psychology. To successfully create a scenario where the software can generate a Type I error, which combination of population characteristics and decision rules must you program into the model?
In psychological research, a Type I error is also known as a 'false positive.'
A researcher must decide between two significance levels for a study on a new behavioral therapy. They evaluate the trade-offs and conclude that it is more damaging to give patients 'false hope' with a treatment that does not work than to miss a potentially helpful therapy. To align with this evaluation, the researcher selects a lower level to minimize the probability of a _____.
Which of the following statements best explains why a researcher might commit a Type I error, even if their study has no design flaws or bias?
Dr. Carter conducts an experiment to see if listening to classical music while studying improves test scores. In reality, classical music has no effect on test scores (the null hypothesis is true in the population). However, due to random sampling error, Dr. Carter's sample happens to perform extremely well, resulting in a statistically significant difference with . Dr. Carter rejects the null hypothesis and concludes that classical music improves test scores. In this scenario, Dr. Ca
A research team is planning several methodological approaches to a study. Analyze how each design choice or statistical scenario affects the probability of committing a Type I error, and match the scenario to its corresponding impact on the Type I error rate.
Nominal Level, Branch-Conditional Error and Whole-Procedure Error
Learn After
Which of the following best describes the practice of p-hacking in psychological research?
If a researcher conducts multiple analyses on a dataset but selectively publishes only the analysis that yielded a statistically significant result, they are engaging in p-hacking.
A research ethics board is evaluating several study practices for potential bias. Match each p-hacking strategy to the research scenario that demonstrates it.
A researcher is analyzing data from a psychological study and fails to find a statistically significant result. Arrange the following actions in the order they would occur to demonstrate the logical progression of p-hacking to artificially manufacture a significant finding.
You are designing a standardized research protocol for your psychology lab to structurally prevent any opportunity for p-hacking. Based on the mechanisms that enable this practice, which of the following analytical plans must you construct and enforce?
A researcher justifies the selective removal of certain data points only after seeing that their inclusion prevented the study from achieving statistical significance. A reviewer evaluating the scientific integrity of this study would label this practice as _____, as it artificially manipulates the results to obtain a desirable outcome.
The problematic research practice where researchers make various arbitrary analytical decisions to artificially inflate their chances of obtaining a statistically significant result is known as _____.
Imagine you are developing a standardized research protocol for an upcoming psychology experiment. Based on the concept of p-hacking, describe three specific, actionable rules you would implement in your data analysis plan to prevent researchers in your lab from arbitrarily manipulating their data to artificially inflate their chances of obtaining a statistically significant result.
Analyze Dr. Smith's analytical decisions in the provided case study. Identify the specific practices she engaged in that constitute p-hacking, and explain the direct consequence these actions have on the published psychological literature.
A fellow psychology student argues, 'As long as a researcher eventually achieves a desirable -value, the analytical decisions they made along the way—like removing a few outliers or choosing which dependent variables to report—are justified because the significant result proves the phenomenon is real.' Evaluate this argument based on the concept of p-hacking and its impact on statistical error rates.
p-hacking involves making arbitrary analytical decisions to artificially inflate the chances of obtaining a statistically significant result. Match each specific p-hacking practice to its corresponding description.
A researcher establishes a strict mathematical rule for identifying outliers before beginning data collection. After collecting the data, they apply this pre-established rule and remove three participants whose scores fall outside the acceptable range, which subsequently results in a significant -value. This analytical decision is an example of p-hacking.
A researcher conducts an experiment to test a new therapy but engages in p-hacking by selectively reporting dependent variables. Arrange the following events in the correct chronological order to analyze how this specific mechanism of p-hacking unfolds to artificially inflate the chance of a significant finding.