If you recall from your stats 101 class p-values represent the probability that a quantitative finding reflects actual reality, such as some sort of difference between groups. Statisticians use arbitrary cutoffs to define “statistical significance.” P < 0.05 is a very common cutoff, meaning that if your result achieves a p-value of 0.05 or less, then the finding is said to be “statistically” significant, or, there is a 95% chance that your groups are different (or your therapy made a difference, or there was a change in functioning, ect.). There is great controversy between statisticians and researchers regarding statistical significance. I read a paper recently titled, “The Difference Between Statistically Significant or Not Is Not, Itself, Statistically Significant.” Best. Title. Ever. Basically, if p = 0.05 is significant, then what to say about p = 0.08? Is there a dichotomy, in reality, between a finding which is statistically significant versus one which is not? What about p = .05000000000001? And so the debate rages.
I have only a limited understanding of statistics and of probability theory in general and am not necessarily interested in commenting on the debate itself. However, interested as I am in epistomology especially with regards to political philosophy, and philosophy of science, I found this article fascinating.
It lists multitudes of ways of describing findings which are p > 0.05. Many of them express something along the lines of “pretty much significant, but not really.” For example,
approached near significance (p=0.06)
approached our criterion of significance (p>0.08)
approached significant (p=0.11)
approached the borderline of significance (p=0.07)
approached the level of signiﬁcance (p=0.09)
approached trend levels of significance (p0.05)
approached, but did reach, significance (p=0.065)
approaches but fails to achieve a customary level of statistical significance (p=0.154)
approaches statistical significance (p>0.06)
approaching a level of significance (p=0.089)
approaching an acceptable significance level (p=0.056)
approaching borderline significance (p=0.08)
From the standpoint of argument, it is interesting that there are so many creative ways to nudge a reader toward believing your point of view. This perspective informs my skepticism toward grandiose public policy based on “science.” In practice within social and political spheres, “facts” proclaimed by “science” [especially neuroscience] reflect essential and unquestionable reality. In theory, “science” can only suggest tentative hypotheses, not infrequently supported by evidence “approaching a level of statistical significance.”
Science sure has its place. There are definitely ways to qualify and quantify statistical and scientific findings. Science is a powerful tool to investigate the natural world. Science and technology enrich our lives and can be forces for good. But science has its limits. Any competent scientist will readily admit that scientific findings are tentative and that science, in general, while very powerful for studying natural phenomena, is ultimately limited in its quest for truth. A scientist who fails to acknowledge as much is either ignorant or a charlatan. And no, that’s not a false dichotomy.