Differing Views

I have a problem and I am not sure how to resolve it.

How do we ensure that the papers that we publish present the most up-to-date information and analysis?

Academic publication meets strict standards, one of which requires that papers, once submitted, are sent to referees (usually two) who read the paper, check that the research is up-to-date, comment on the significance, and note any omissions, errors or muddled writing. Most people will ask a colleague or two to read a paper before submission – it is better to find out about weaknesses at this point in my opinion.

I’ve acted as a referee myself on frequent occasions and I hope my comments are useful. When the journal allows it, I prefer my name to be known to the authors (though I may not know who they are), because it will allow them to understand my point of view, and if necessary check the precise meaning of my comments. I don’t really believe in saying things that I’d not discuss with someone face to face.

In general, I find that the comments of referees on my papers always result in stronger papers. They see things from a wider point of view than I do because they have not been bound up with a particular project for the previous months (or years); they highlight things that, while obvious to me, are not obvious to others; they point out areas where my writing is unclear; and they are great at suggesting references that I have overlooked. It might be annoying to have to unpick your writing once you think you have signed it off, but in the end it is  worthwhile.

But – I am sure you can hear a ‘but’ coming…

But, just once in a while it all goes wrong and that shakes my faith in the system. I had a paper a while ago that was refereed by three people (I’m not sure why that was, it is the first time I’ve come across it, but perhaps that is the new standard). Curiously, each identified totally different weaknesses in the paper. The optimistic side of me would see that as a validation that my point of view, while not everyone’s, did not contain any total howlers. Unfortunately, that is not how journal editors work: they tend to be more negative so that in this case it merely tripled the weaknesses.

You can see their point: except that in many instances these particular referees disagreed with each other. One thought that the stone tools might be particularly early, another was disappointed that I had not explored the possibility that they represented a survival of that technology into late prehistory. In actual fact there is no evidence in Scotland for the early or late instance of this technique at all, though I suppose if we were hidebound we would never discover anything new. One was concerned that I had not undertaken a Bayesian analysis of the (poorly contexted) radiocarbon dates; this raised the vision of Patrick Ashmore who taught me so much about the unreliability of dates based on uncertain contexts, something that I’m not sure even the most sophisticated of Bayesian work can remedy. I could go on, but I think you get my point.

Of course, it is possible to argue your case with an editor, but in my experience this is rarely successful; editors tend to assign academic precedence to referees rather than authors, even when it is the latter who have been studying a particular subject or site. And, I always have that niggling feeling – ‘what if they are right’. Having my work questioned makes me doubt myself. I know I should be more resilient, but my inclination is to go through the comments and try to cover each one in text. This might lead to some strange discussion of issues that most people would not regard as relevant, but it does make everything blindingly obvious. Sometimes the level of detail is such that one is left with the lurking feeling that you should have added the referees as co-authors.

Being a referee is a big commitment. For every journal there is an army of unpaid referees, reading, thinking and commenting. We have to thank them. But it is not a perfect system. Occasionally there are scores to be paid; or simply the desire to let off steam after you have had to deal with some picky referee yourself; sometimes arms are twisted to referee something where you really don’t have the expertise. Usually, these things show up and, of course, that is the reason that the double referee system has been developed. But as long as editors bow to the referee’s opinion without any thought, then the system is flawed.

I’m not sure how to improve it. We need to ensure academic excellence. But I’m coming across more and more examples of refereeing that is somehow not quite working. I’m hoping that with more open dialogue we might be able to return to the system where the referees work to ensure the significance and quality of publications, without rewriting them on behalf of the authors.