Statistics for Data Science & Analytics

Understanding P-value is important, as P-values are one of the most widely used and misunderstood concepts in the subject of statistics. Whether you are a novice, a data analyst, or an experienced data scientist, understanding p-values is crucial for hypothesis testing, A/B testing, and scientific research. In this post, we will cover:

What is a p-value? Understanding P-value

A p-value (probability value) measures the strength of evidence against a null hypothesis in a statistical test. The formal definition is

The probability of observing a test statistic as extreme as, or more extreme than, the one calculated, assuming the null hypothesis is true.

Key Interpretation: A low p-value (typically ≤ 0.05) suggests the observed data is unlikely under the null hypothesis, leading to its rejection. For example, suppose you run an A/B test:

Null Hypothesis ($H_o$): No difference between versions A and B.

Observed p-value = 0.03 → There is a 3% chance of seeing this result if $H_o$ were true.

Conclusion: Reject $H_o$ at the 5% significance level.

The P-value of a test statistic is the probability of drawing a random sample whose standardized test statistic is at least as contrary to the claim of the Null Hypothesis as that observed in the sample group.

How to Interpret P-Values Correctly?

To interpret P-values correctly, we need thresholds and Significance. For example,

• $p\le 0.05$: Often considered “statistically significant” (but context matters!).
• $p>0.05$: Insufficient evidence to reject $H_o$ (but not proof that $H_o$ is true).

The following are some common Misinterpretations:

• A p-value is the probability that the null hypothesis is true. → No! It is the probability of the data given $H_o$, not the other way around.
• A smaller p-value means a stronger effect. → No! It only indicates stronger evidence against $H_o$, not the effect size.
• $p>0.05$ means ‘no effect.’ → No! It means no statistically significant evidence, not proof of absence.

Limitations and Criticisms of P-Values

The following are some limitations and criticisms of P-values:

• P-hacking: Cherry-picking data to get $p\le 0.05$ inflates false positives.
• Dependence on Sample Size: Large samples can produce tiny p-values for trivial effects.
• Alternatives: Consider confidence intervals, Bayesian methods, or effect sizes.

Cherry-Picking Data: selectively choosing data points that support a desired outcome or hypothesis while ignoring data that contradicts it. For example, showing an upward sales trend over the first few months of a year, while omitting the data that showed sales declined for the rest of the year.

Computing P-value: A Numerical Example

A university claims that the average SAT score for its incoming students is 1080. A sample of 56 freshmen at the university is drawn, and the average SAT score is found to be $\bar{x}=1044$ with a sample standard deviation of $s=94.7$ points. Find the p-value.

Suppose our hypothesis in this case is

$H_o:\mu =1080$

$H_1:\mu \ne 1080$

The standardized test statistic is:

$$\begin{array}{rl}Z& =\frac{\bar{x}–{\mu }_o}{\frac{s}{\sqrt{n}}}\\ & =\frac{1044-1080}{\frac{94.7}{\sqrt{56}}}=-2.85\end{array}$$

From the alternative hypothesis, the test statistic is two-tailed, therefore, the p-value is given by

$$\begin{array}{rl}P(z\le -2.85orz\ge 2.85)& =2\times P(z\le -2.85)\\ & =2\times 0.0022=0.0044\end{array}$$

Deciding to Reject the Null Hypothesis

A very small p-value would lead us to reject the null hypothesis while a high p-value would not Since the p-value of a test is the probability of randomly drawing a sample at least as contrary to $H_o$ as the observed sample, one can think of the p-value as the probability that we will be wrong if we choose to reject $H_o$ based on our sampled data. The p-value, then, is the probability of making a Type I Error.

Recall that the maximum acceptable probability of making a Type-I Error is the significance level ($\alpha$), and it is usually determined at the outset of the hypothesis test. The rule that is used to decide whether to reject $H_o$ is:

• Reject $H_o$ if $p\le \alpha$
• Do not reject $H_o$ if p > \alpha$

Practical Example: Calculating P-Values in Python & R

from scipy import stats

# Two-sample t-test  

t_stat, p_value = stats.ttest_ind(group_A, group_B)

print(f"P-value: {p_value:.4f}") 

# Two-Sample t-test

result <- t.test(group_A, group_B)

print(paste("P-value:", result$p.value))

Best Practices for Using P-Values

• Pre-specify significance levels (e.g., $alpha=0.05$) before testing.
• Report effect sizes and confidence intervals alongside p-values.
• Avoid dichotomizing results (“significant” vs “not significant”).
• Consider Bayesian alternatives when appropriate.

Conclusion

P-values are powerful but often misused. By understanding their definition, interpretation, and limitations, you can make better data-driven decisions.

Want to learn more?

• American Statistical Association’s Statement on p-values
• “The Cult of Statistical Significance” by Ziliak & McCloskey
• P-value Definition and Interpretation
• P-value interpretation and misinterpretation

Try Permutation Combination Math MCQS