P-value Trap: Why the Most Essential Statistics is Also the Most Misunderstood
Most common misconceptions around p-value
💡 Introduction: The Desire for Certainty
If you have ever worked with data, you know the overwhelming compulsion to draw a conclusion: Did this thing work?
Statistics is the language we use to move from a vast mountain of raw data into a singular, defensible conclusion. And lurking at the heart of that decision process is perhaps the most powerful, yet also the most misunderstood concept in all of modern science: the P-value.
In this post, we are not going to learn if the P-value is important. We are going to learn how it truly works, and more importantly, how to prevent the common errors.
♟️ Setting the Stage: The Game of Hypothesis Testing
Before we talk about P-value, we must understand the scene.
When we study a population (e.g., all university students), we rarely have access to their entire group. We must use a smaller, manageable sample (e.g., 100 students).
The P-value is the bridge that allows us to use our sample’s limited evidence to make a confident claim about the whole population.
This journey requires us to start with a baseline assumption, which we call the Null Hypothesis.
The Null Hypothesis is simply the status quo. It says: “Nothing interesting happened; any result we saw was probably due to random chance.”
- Example: You test a new study method. Null hypothesis is: “The new study method has no effect on test scores.”
Here, we should also remind ourselves of another assumption: our sample is a representative of the population. So we start with two main assumptions
- Our data sample is a representative of the population: Random sampling is often employed to maximize the chances for a sample to capture essence of the population.
- Null hypothesis is true: We make the assumption of the opposite of what we want to find evidence for is being True.
Once we accept these quiet assumptions, the P-value enters the room.
🧱 What Exactly Is a P-Value? (The Precise Definition)
Forget trying to define it as “proof.” The P-value is a conditional probability, it depends entirely on the condition we impose (i.e., our assumptions).
Get Pankaj Chejara, PhD’s stories in your inbox
Join Medium for free to get updates from this writer.
P-value gives us a measure of how likely it is to see our data sample given that our assumptions are held valid. It is a measure of how well our data confirms to our assumptions including null hypothesis.
p-value is probability of observing our sample in a world where all our assumptions are held valid.
Some examples:
- Small P-value (e.g., P < 0.05): This means seeing our data do not aligns well with our assumptions. But we still observed our data and that raises doubts on our assumptions, i.e., either one or both assumptions must be invalid.
- Large P-value (e.g., P>0.05): This means seeing our data is entirely plausible under our assumptions.
We then introduce a threshold: the Significance Level (α), typically set at 0.05. This is our line in the sand: if the observed rarity (P-value) falls below this line, we collect enough evidence to formally challenge and reject the status quo (null hypothesis).
🚨 The Common Pitfalls: Why We All Go Wrong
❌ Misconception #1: P-value = Probability that null hypothesis is True.
The Correction: This is the most fatal error. The P-value works one way: it measures data compatibility with null hypothesis (or more accurately with all our assumptions). You are testing the hypothesis; you are not trying to probabilistically prove its truth or falsehood from scratch.
❌ Misconception #2: P-value ≤ 0.05 means I have proven something.
The Correction: It does not prove it. A p-value ≤ 0.05 tells us that it is highly unlikely to observe our data if all our assumptions are true. So there is discrepancy between what we assumed and what we observed. And that discrepancy could be due to any of those assumptions, e.g., it was not a random sample.
Our Significance Level (α) tells us that our data earns the right to challenge the null hypothesis and we report by saying that our data offers sufficient evidence against null hypothesis, thus, it could be rejected. But we still have a possibility that our test results’ significance could be due to issues with randomness (i.e., sample was not random) (Type-I error).
❌ Misconception #3: P-value > .05 means that null hypothesis is True or should be accepted.
The Correction: A large P-value (e.g., p-value > 0.05) simply means that our data are in alignment with our assumptions including null hypothesis. And there is no evidence to doubt and challenge our assumptions. We report this by stating that our data does not provide enough evidence against null hypothesis, thus, we fail to reject it.
❌ Misconception #4: P-value > 0.05 means there is no effect.
The Correction: A large P-value simply means your data does not provide enough evidence to cast doubts on the status quo. It does not mean the effect is non-existent. It could be that a real, powerful effect exists in the population, but our sample size was too small, or our collection method flawed, rendering us unable to detect it. We must only say that we “failed to reject null hypothesis.”
Conclusion: Embrace the Humility of Inference
The P-value is not a definitive “effect” or “no effect.” It is the quantification of evidence against initial assumptions.
If you take one thing away from this article, let it be this:
Do not confuse the evidence against “no effect” with a declaration of effect.
References
- Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology, 31(4), 337–350. https://doi.org/10.1007/s10654-016-0149-3
