Three critical indicators of the quality of evidence – often widely overlooked – are whether the paper is published, where it is published, and who the authors are. A published paper has to go through rigorous peer review to check its scientific accuracy. The very top journals have the highest standards, using leading scholars at the world’s best business schools to scrutinise a manuscript. As Managing Editor of the Review of Finance, Europe’s leading finance journal, I reject 97% of manuscripts. The 3% not rejected are not immediately accepted either. Instead, their status is “revise-and-resubmit”. The reviewers communicate concerns that the authors need to address, and the paper can still be rejected at the next round. It is not unusual for a paper to take five years to be published after its first draft. A hard slog for the authors, but it helps ensure that the published results are correct.
As an example of how rigorous checks can overturn a paper’s conclusions, consider a witness’s submission to the Parliamentary Inquiry on Corporate Governance. This witness argued for disclosure of pay ratios, citing a finding “that firm productivity is negatively correlated with pay disparity between top executive and lower level employees.” The assertion was based on an unpublished 2010 draft. Having gone through peer review and after methodology was improved, the 2013 published version reverses the preliminary findings and concludes “that firm value and operating performance both increase with relative pay”. In addition to highlighting the importance of peer review, this example also shows how a partisan observer can find any paper to support a given pre-existing viewpoint, even an unpublished draft when a published version is available.
The second indicator is the quality of the journal in which a paper has been published. That a journal calls itself “peer-reviewed” is far from sufficient to guarantee its rigour, since there is a vast range in the quality of reviewing standards. Journal quality can easily be checked by looking at one of the freely-available lists – for example the Financial Times Top 50. Aa reader does not need to be an academic “insider” to do this.
Of course, every paper starts out unpublished. How do we gauge quality of a new paper? The third indicator is the credentials of the authors, such as the quality of their institution and their track record of past top-tier publications – credentials we would check for an expert witness in a trial. Again, lists of the top universities are freely available. Last November, a study on executive pay hit national headlines – even though every single person I asked had not heard of the business school that released the study. This is not elitism, but simply a desire to use the best evidence. We would listen to a medical opinion from the Royal Marsden Hospital more than a hospital we have never heard of. An equally serious issue is that none of the newspapers who covered the study had even read it: it was not out yet. But, they were happy to take the authors’ word for it, potentially because its findings played to national mood on executive pay.
Peer review is not perfect. Mistakes are made. But it is better to go with something checked than something unchecked. When considering treatment options for a medical condition, a patient would want to consider the world’s best evidence on the treatments’ success, conducted by the top scientists and thoroughly checked. We should apply the same rigour when considering the health of the economy.