7/1/2010 Author: Panos Y. Papalambros
Volume: 132(7) – July, 2010
Who would wish to be blind and kept at arm’s length? Would we wish that for a JMD reviewer?
Effecting a good review process for each JMD submission is my most important function as editor. I have had occasions to comment on this in previous editorials (A Good Review, December 2008; Who is a JMD Reviewer? February 2010). It is a long-held JMD and ASME practice that authors are not informed of the identity of the reviewers (a “masked” or “blind” review) but do know the identity of the associate editor. Occasionally, I am asked whether we should have the identity of the associate editor masked from the authors (a “blind plus” review?) or the authors’ identities masked from the reviewers (a “double-blind” review). The recent emergence of “arm’s length” evaluators in academic promotion cases has also brought up considerations for an “arm’s length” reviewer system. My preamble questions give you a hint on my positions, so let me explain why.
It is well documented that the peer review process is imperfect, just as all of us humans are imperfect. At this point in time, this process appears better than the alternatives—although internet-based mass dissemination may change this in the not-too-distant future. Masking the reviewers’ identity offers a degree of separation between author and reviewer, and a general avoidance of improper personalization of the inevitable critique. But there may be some proper personalization that reviewers as members of the community may and do exercise: critique of a younger researcher’s submission may put a stronger emphasis on helping the author write a high quality successful paper; seasoned authors may be criticized more harshly for not living up to their usual standards. This phenomenon is actually the opposite than what the proponents of double-blind reviews think, namely, that senior authors get a better treatment due to their status and their friends in the community. In my experience that is not true, and blind review really helps here.
There is also evidence that double-blind reviews do not work that well in masking the authors’ identity most of the time due to included references, topic, or style of writing. Can’t we really guess? A fun study showed that using automatic matching techniques, based on discriminative self-citations, identified authors correctly 40-45% of the time; had 60% accuracy for the top-10% most prolific authors, and 85% accuracy for authors with 100 or more prior papers (Hill, S. and Provost, F. The myth of the double-blind review?: Author identification using only citations. SIGKDD Explor. Newsl. 5, 2, Dec. 2003). I bet we would do better than the machines!
Double-blind reviews do create a perception of a fairer process at face value, but I think that in most cases the impression is only skin deep. Then there is the question about how far blindness can go. Should we mask the authors from the associate editors also? What about the chief editor? The potential for bias exists at all levels, but increasing blindness is an unlikely remedy. In fact, we could go the other way: no blindness at all, as it is practiced in some software engineering journals, where the system is based on volunteering to the editors any conflicts of interest by both authors and reviewers. I have known of JMD reviewers who send their reviews to the authors directly, as well as to the editors, so as to expedite the revision process.
An altogether different issue is application of the arm’s length principle. It is telling that the principle is a legal one in origin, and has thus crept into the academic promotion deliberations. The arm’s length principle is the condition or the fact that the parties to a transaction are independent and on an equal footing (wikipedia). Some have proposed using this principle in selecting paper reviewers. Interestingly, the principle comes from contract law; namely, laws that govern relations among individuals where agreements are to be kept with specific remedies for breach of promises. But you can find it also in tort law remedies for civil wrongdoings not arising out of contractual obligations. In simpler terms more appropriate for the present context, arm’s length is the description of an agreement made by two parties freely and independently of each other, and without some special relationship, such as being a relative, having another deal on the side or one party having complete control of the other (thefreedictionary.com). Finally, another interesting definition comes from the business world: A transaction between affiliated firms made purely on commercial basis (with) both firms trying to maximize their advantage, and neither firm accommodating or favoring the other in any way ((businessdictionary.com). This is more fitting to our situation; just substitute: firms = researchers, commercial = scholarly.
The rationale for selecting unbiased reviewers is obvious. In a relatively close community, like ours in JMD, we clearly avoid obvious conflicts of interest. Yet, if we stretch our arms too far we may find only reviewers who would be disinterested, disengaged or not sufficiently knowledgeable, and thus not good choices for the task, as they would readily tell us. This is not uncommon in academic promotion review exercises where the value added is rather meager. In JMD, the associate editors play the major role in the delicate balancing of these requirements. When in doubt, the editor will provide further advice and guidance, so that the entire editorial board operates under similar practices. For this reason also, I favor openness in the identity of the associate editor allowing direct communication with the authors.
In the end, many of these issues seem to be of greatest interest to the life sciences, medical and pharmaceutical communities. There, the outcome of peer reviews can have a very high impact not just on people’s lives but also on how large sums of money get dispersed. Generally, we do not face this problem at JMD —and that’s the good and the bad news!
