An item from the folks at IES that may have some interest among readers of this space.  I know I’m particularly interested in the discussion, as it is these kinds of myths that underpin so much of how educational research is view.  Another example would be the “gold standard” of research that is housed in the What Works Clearinghouse.

From the IES Director: Statistically Significant Doesn’t Mean Meaningful Starting with our very first statistics course, most of us were taught that random variation can lead us to misidentify a difference between groups or a change over time when there is no meaningful difference or change. All measurement includes some amount of random error, which means randomness can fool us into putting too much stock into apparent differences that do not reflect meaningful differences in true values. To minimize these mistakes, we’re taught to calculate p-values to assess “statistical significance.” Many of us were led to believe that a p-value < .05—which serves as the bright line for statistical significance in education and many other fields—indicates that there is under a 5 percent likelihood that the differences we see in our data are due to chance. Unfortunately, that’s not what p-values mean at all. As the American Statistical Association has been warning for years, a p-value doesn’t directly translate into the probability that a finding is due to chance. (Readers should check out the ASA statement and its accompanying commentary.) Moreover, p-values and tests of statistical significance say nothing about the size of an effect or whether a difference is educationally meaningful. In a large sample, a difference that is statistically significant might be trivial; in a small sample, substantively important differences might not reach statistical significance. How can we do better? Below, Brian Gill and I discuss how Bayesian statistics can help us better understand recent NAEP results while moving the field away from the p<.05 bright line that leads to misinterpretations of the data. Read the full blog. The Institute of Education Sciences, a part of the U.S. Department of Education, is the nation’s leading source for rigorous, independent education research, evaluation, statistics, and assessment.

By visiting Newsflash you may also sign up to receive information from IES and its four Centers NCES, NCER, NCEE, & NCSER to stay abreast of all activities within the Institute of Education Sciences (IES).