Forget 0.005 and use JASP, instead

Seeking to address the lack of research reproducibility due to the high rate of false positives in the literature, Benjamin et al. (2017a,b) propose a pragmatic solution, despite many of the proponents believing that the proposal is nonsense, anyway (that is, it is a quick fix, not a credible one from a Bayesian perspective). Notwithstanding its potential impact, Benjamin et al.’s (2017) proposal is a mere tweak to NHST yet it is not recognizant of the entire engine of science, which has motivated a large mass of authors to offer their own counter-arguments. These counter-arguments touch on philosophy of science, methodology, NHST as pseudoscience, unwarranted conclusions, descriptives approaches to data analysis, frequentist testing, Bayes Factors testing, Bayesian hypothesis testing, and replicability (see https://psyarxiv.com/t2fn8).

Despite Benjamin et al.’s best intentions, their proposal only reinforces the NHST pseudoscientific approach with a stricter bright-line threshold for claiming evidence it cannot possibly claim—irrespective of how well it calibrates with Bayes factors under certain conditions—while dismissing potential consequences such as the stifling of research and its publication, an increase in false negatives, an increase in misbehaviour, and a continuation of pseudoscientific practices. Furthermore, theirs is a frequentist solution many of their Bayesian proponents do not even believe in, born out of a false belief of lacking equally simple but better-suited alternatives (Machery, 2017).

In reality, an equally simple and better suited alternative exists, perhaps the only one the authors could be entitled to make from their Jeffresian perspective, hence our own recommendation: Use JASP, the stand-alone, free-to-download, R-based statistical software with user-friendly interface, developed by Wagenmakers’ team (https://jasp-stats.org). JASP allows for analysing the same dataset using frequentist (Fisher’s tests of significance) and Bayesian tools (Jeffreys’s Bayes factors) without necessarily forcing a mishmash of philosophies of science. JASP allows for the Popperian (also Meehl’s, Mayo’s) approach to severely testing a null hypothesis but also to ascertain the credibility of tests based on the observed data. JASP allows researchers to qualitatively calibrate their significance tests to Bayes factors but also the possibility of calibrating Bayes factors to frequentist tests so as to ascertain their error probabilities. JASP is an existing simple and easy application that shows two interpretations of the same data, which aligns well with the training undertaken by frequentist and Jeffreysian researchers alike while allowing them the opportunity to learn the alternative philosophy—irrespective of which they will choose for publication—and may as well help diminish the misinterpretation of results and the reaching of unsupported conclusions. In so doing, it may even help improve replicability.