Major Compulsory Revisions

My main criticism is that there could be a more thorough discussion of how to test for mean/location difference between two samples. I don’t think the Wilcoxon-Mann-Whitney U test is the last word in this.

First the Student T test. There is a version of the Student T test that allows for different variances between the two distributions. The T test assays whether the difference between sample means is likely to reflect a real difference in means of the two samples’ underlying distributions or may just be stochastic. The p-value produced may be problematic for small sample size or very skewed distributions. However, by the central limit theorem, the sample mean will tend towards Gaussian distributed for large sample size regardless of the sample distribution, so generally it is pretty robust to non-normality. For instance the author’s Fig 1A and 1B look Gaussian, I see no reason not to use a Student T test to test for mean difference. For more on its robustness and why it is robust see – e.g. http://thestatsgeek.com/2013/09/28/the-t-test-and-robustness-to-non-normality/

Second The Wilcoxon-Mann-Whitney U test. The Wilcoxon-Mann-Whitney U test being non-parametric (a rank based test) will be less powerful than something that takes the actual values into account, e.g. the Student T test. A more serious issue is that the null hypothesis of the Wilcoxon-Mann-Whitney U test is not straightforward. It isn’t a test of mean difference unless the two distributions have the same shape, which rather defeats the point. See e.g. - http://thestatsgeek.com/2014/04/12/is-the-wilcoxon-mann-whitney-test-a-good-non-parametric-alternative-to-the-t-test/

Third, the KS test. The KS test suffers some floor and ceiling effects, i.e. its test statistic D tends to be small when in either tail. This has been noticed and tests that adjust for this have been developed which are more powerful than the KS test. See - Beware the Kolmogorov-Smirnov test! Eric Feigelson and G. Jogesh Babu, https://asaip.psu.edu/Articles/beware-the-kolmogorov-smirnov-test

Off the top of my head, another way of testing for mean difference would be a two stage approach. First, use some test you may have doubts about to discard the clearly insignificant candidates. To the small subset of candidates remaining a more rigorous assumption free computer intensive test can be applied. I was thinking of bootstrap permuting case control status to obtain a sample of the test statistic from which a p-value can be read. I don’t know how valid this idea is, e.g. see http://en.wikipedia.org/wiki/Sequential_analysis

Minor Problems

See attached for suggested edits to doc

Level of interest: An article of importance in its field

Quality of written English: Needs some language corrections before being published

Declaration of competing interests: I declare that I have no competing interests