Effect Size and Statistical Significance

Statistical significance is important but not only the most important consideration in evaluating the results. Because statistical significance tells only the likelihood (probability) that the observed results are due to chance alone. It is important to consider the effect size when you obtain statistically significant results.

Effect size is a quantitative measure of some phenomenon. For example,

  • Correlation between two variables
  • The regression coefficients ($\beta_0, \beta_1, \beta_2$) for the regression model, for example, coefficients $\beta_1, \beta_2, \cdots$
  • The mean difference between two or more groups
  • The risk with which something happens

The effect size play an important role in power analysis, sample size planning and in meta-analysis.

Since effect size is an indicator of how strong (or how important) our results are. Therefore, when you are reporting results about statistical significant for an inferential test, the effect size should also be reported.

For the difference in means, the pooled standard deviation (also called combined standard deviation, obtained from pooled variance) is used to indicate the effect size. The effect size ($d$) for the difference in means by Cohens’s is

$d=\frac{mean\, of\, group\,1 – mean\,of\,group\,2}{SD_{pooled}}$

Cohen’s provided the rough guidelines for interpreting the effect size.

If $d=0.2$ the effect size will be considered as small.

For $d=0.5$ the effect size will be medium.

and if $d=0.8$ the effect size is considered as large.

Note that statistical significance is not the same as the effect size. The statistical significance tells how likely it is that the result is due to chance, while effect size tells how important the result is.

Also note that the statistical significance is not equal to economic, human, or scientific significance.

For effect size of dependent sample $t$-test, see the post effect size for dependent sample t-test

See the a short video on Effect Size and Statistical Significance

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Discover more from Statistics for Data Analyst

Subscribe now to keep reading and get access to the full archive.

Continue reading

Scroll to Top