Review for "Science with no fiction: measuring the veracity of scientific reports by citation analysis"

Completed on 17 Sep 2017 by Thomas Munro . Sourced from http://www.biorxiv.org/content/early/2017/08/09/172940.

Login to endorse this review.


Comments to author

The idea of validating this metric against the Reproducibility Project is excellent, but I think more examples are needed, especially examples where apparently reproducible claims later turned out to be wrong, as discussed by neuroskeptic. The "replication crisis" was after all the inspiration for the Reproducibility Project in the first place. A classic example is Millikan's incorrect value for the charge of the electron, whose apparently high reproducibility slowly declined over several decades:
https://hsm.stackexchange.c...

From your tree viewer, it appears that you have values for many other studies, which would strengthen the paper if added.

I like your idea of plotting the result over time. The trajectory may well give early warning of problems, as in the Millikan case.

Calculating this metric would be much less work using a search engine that delivers the sentence in which the citation appears, the "citation context". The Colil database does this, but only for a subset of Pubmed Central. It's a powerful proof of concept, though.

http://colil.dbcls.jp

Citeseerx seems to give the context for some citations, but not to a useful extent. The old Microsoft Academic Search used to offer this, but the new Microsoft Academic web interface does not; the citation contexts are only available via the API, which the typical researcher is not going to use. This is a tragic waste, since the new version has excellent coverage; I find it often detects citations that google scholar misses. If you approached them, this paper might help them see the value of that feature enough to re-implement it. Perhaps you could suggest a collaboration. The R-factor is much more likely to take off if it's easily calculated. I think it's worth mentioning these services in the paper to help people who'd like to try it.

As neuroskeptic pointed out, the name "R-factor" is likely to be confused with the R-index. I think a more descriptive name would help to avoid confusion, e.g. "Published replication rate" or quotient. Also, rather than use subscripts, which are not self-explanatory, it could be given as a fraction, e.g. "1/11(9%)".