Once again, I'm going to "get meta" on that recent paper on blogs as a channel of scientific communication I mentioned in my last post. Here, the larger question I'd like to consider is how peer review -- the back and forth between authors and reviewers, mediated (and perhaps even refereed by) journal editors -- does, could, and perhaps should play out.
Prefacing his post about the paper, Bora writes:
First, let me get the Conflict Of Interest out of the way. I am on the Editorial Board of the Journal of Science Communication. I helped the journal find reviewers for this particular manuscript. And I have reviewed it myself. Wanting to see this journal be the best it can be, I was somewhat dismayed that the paper was published despite not being revised in any way that reflects a response to any of my criticisms I voiced in my review.
Bora's post, in other words, drew heavily on comments he wrote for the author of the paper to consider (and, presumably, to take into account in her revision of the manuscript) before it was published.
Since, as it turns out, the author didn't make revisions addressing Bora's criticisms that ended up in the published version of the paper, Bora went ahead and made those criticisms part of the (now public) discussion of the published paper. He still endorses those criticisms, so he chooses to share them with the larger audience the paper has now that it has been published.
Later, in a comment on his own post, he adds:
Nobody can publish my comments without my consent, but I can do it myself.
I was not the editor of the paper in the sense people usually think of. I did not communicate with other reviewers, I did not see other reviewer's notes, I do not know if they suggested acceptance, acceptance with minor/major revisions, or rejection, and I was not a part of the decision to accept or not. I suggested major revisions, then a second round of review. That did not happen, so I am free to post my comments - both those I made at the time and additional comments.
I also have a little bit of gray area here. This being the first manuscript I did for JCOM I did not fully understand my role. I was asked to suggest some names for reviewers (I have no idea who reviewed in the end). I did not understand that this is where my job ends, so I sent in my review as well. It is, thus, an unofficial review. They (editors) read it, but they did not need to use my comments if they did not want to do that. This was more like sharing my thoughts with them. But the comments being unofficial makes them even more 'mine' and free to post online.
As Bora was the "editor" of the paper rather than an official referee of the paper, it's not clear whether the journal editors overseeing the fate of this submission actually forwarded Bora's critiques onto the author, or if they did forward the critiques to the author but indicated that they wouldn't count. Myself, if I were the author of the manuscript, I think I'd want more prepublication feedback, not less, on the theory that this would help me produce a stronger paper.
So maybe, the official referees didn't, as a group, make critiques that raised the points Bora's did. Maybe the majority of however many reviewers there were gave effusively positive reviews saying, "This is fabulous as it stands, and should be published without further ado!"
I am in a position to tell you that not all of the substantive critiques raised by people asked to peer review the manuscript were adequately addressed (at least for values of "adequate" that the referees would endorse) before the paper was published. I was one of the referees in question.
I doubt that any of this is terribly unusual, but it brings into focus some interesting questions about the role of peer review. Is it intended merely as a filter against the most atrocious methodological crimes, the most blatant crackpottery? As a gatekeeper for the literature, keeping out submissions that do not meet the very highest standards (or, alternatively, that do not satisfy the reviewers' biases)? As a preview of the kind of engagement from other scholars the author can expect the submission to elicit if published as-is?
Do the journal editor, the peer reviewers, and the author in a case like this have a shared understanding of what peer review is supposed to accomplish, or of how reviewer comments ought to be used, whether by the author or by the journal editor?
In cases where the outcome suggests that there wasn't agreement among journal editor, peer reviewers, and author about how to use referee reports, how ought the parties to deal with this disagreement? For example, if one has labored over comments only to have them ignored, how should one respond? Indulge me in a quick poll:
If we treat referee reports as a preview of coming attractions, a microcosm of the kinds of criticisms -- and disagreements -- other scholars are likely to have, then being the "hard referee" whose judgment is overruled may be all in a day's work. Indeed, if peer review is meant primarily to alert the author of a manuscript to the sorts of battles she is likely to have to wage to defend her work, maybe it isn't even so much of a problem is journal editors end up substituting their (positive) judgment for the (negative) judgments of the peer reviewers whose reviews they have solicited. (There, though, I think it would be better if peer reviewers were given the straight story on how their reviews are, or are not, to be used.)
On the other hand, if getting one's manuscript published in a peer reviewed journal is taken to be a mark of something else -- of having produced officially recognized Reasonable Knowledge Claims Based on Sound Methodology -- and if the constructive engagement between referees and author in the process of peer review is the mechanism that is somehow taken to be responsible for conferring this special status on the published work that results -- then maybe we have cause to worry about authors, or journal editors, who don't actually engage with the criticisms the referees raise. (Please note that "engaging" with criticisms does not always mean accepting them. Sometimes engagement involves mounting a counterargument -- but this requires acknowledging that reasonable people might raise the criticisms you are answering.)
What kind of duties does the manuscript author has, as far as dealing with issues raised in referee reports?
What kind of duty do the journal editors have to ensure that the manuscript author actually engages with those issues, and that this engagement leaves its trace in the version of the manuscript that is eventually published?