Journal Editorial Practices and an Anecdote

I was interested to read the recent Daily Nous conversation about journal editorial practices. Much of the discussion there focused on whether double-anonymous refereeing (where the authors and referees are ignorant of each others' identities) was good enough, or whether triple-anonymous methods (where the editors are also unaware of the authors' identities) are necessary for a fair system.

I was interested to see many journals' editors describing their methods, and how they try to avoid bias. One question I didn't see addressed, however, concerns oversight. Is there any system in place to confirm whether journals are actually run the way they're said to? I would consider this a somewhat paranoid question, except that I know firsthand of at least one high-profile journal which at least sometimes, at the discretion of the editor, suspends its official policy of double-anonymous refereeing.

My experience happened shortly after I finished my PhD, when I was a postdoc. I knew at the time it was a problematic situation, but I decided it probably wasn't prudent to make a fuss at that stage in my career. I'd been intending to wait until I was tenured, but I think that given the conversations happening in the discipline at the moment, now is an appropriate time for the story to come to light. This is the story of "Quantifiers and Epistemic Contextualism," one of my first papers defending contextualism, which was eventually published in Philosophical Studies.

I apologize for the length of the narrative. I'd make more of a point of being concise, but I want to make sure that I explain exactly what happened, without editorial summarizing. (I'll express some opinions after telling the story.)

So here's the story.

After defending my PhD in 2008, and beginning work as a postdoc in St Andrews the same year, I started to develop some of the ideas from my dissertation into publishable papers. I submitted one of them, a defence of contextualism, to Philosophical Studies in March of 2009. That journal uses an online program for submissions where an author can watch the progress of his submission as it goes through various stages—'editor assigned', 'reviewer assigned', 'review completed', etc. I noticed in July that four months had passed and it still just said 'editor assigned'. (I knew from a previous paper that it ought to be progressing through other stages.) So I submitted a query through the online submission, asking whether my paper might have fallen through the cracks.

I received a response to that question in the form of a personal email from Stewart Cohen, the editor of the journal. Cohen wrote that "the reason your paper is listed as editor assigned, is that I'm going to review it myself." He said he hoped to get a chance to review it soon, and invited me to email him directly if I had any further questions. This happened in July. (Potentially relevant background not everyone might know: Cohen wrote several influential papers in defence of contextualism in the 1980s and 1990s; he is one of the prominent figures in this subfield of epistemology.)

In September 2009, Cohen was hired as a part-time professorial fellow at Arché, the same institution where I was working.

In October, Cohen came to St Andrews in connection with his new job. He spoke briefly with me about my submission then, saying that he generally liked it, but had some particular concerns. He sent me a lengthy email detailing them, and we had a rather involved back-and-forth exchange on various of the substantive philosophical questions. At the conclusion of this exchange, after I'd agreed to make various changes to the paper, we'd gotten down to just one major remaining point of disagreement. In my paper, I argued that my neo-Lewisian form of contextualism avoids a challenge that Cohen himself had leveled against Lewis, concerning his ability to handle both Gettier cases and lottery cases. Cohen had argued, in a 1998 paper, that Lewis can't have it both ways; but I suggested in my submission that the issue could be avoided. Cohen was unconvinced by my treatment of this part of the paper, but invited me to work on it further. He wrote to me in an email:

Let's call it a conditional acceptance. I'm not sure you've made any advance in applying the RA view to Gettier cases and so I don't want to commit to publishing that part of the paper. You may convince me that I'm wrong, but even if you don't, you can always just cut that part. So unless you think you'll refuse to cut it, even if I'm not satisfied, you can list it as forthcoming.

In November 2009, I submitted (via direct email to Cohen) a revised version of the paper. We corresponded briefly about it in January when Cohen was visiting Arché again, but he didn't read and respond to my new draft until April 2010, when he wrote back, saying that he was still unconvinced by my treatment of lotteries and Gettier cases. We had another email exchange, which continued until his next visit to Arché in May. On May 24, we met over coffee about whether the section in question ought to be published. By the end of that meeting, he did agree to publish it, and on May 25 I received an official acceptance from the journal, and an invitation to upload my final version. The paper did end up being published, including the revised version of the section defending Lewis from Cohen.

Ok, those were 'just the facts'. From here on out, I'm editorializing. These seem to me the important problematic points to emphasize:

• The editor of Philosophical Studies seems to be treating submissions in his own area of research interest very differently from the way he treats other submissions, sometimes deciding to referee papers himself. As far as I know, Cohen was the sole referee for my submission.
• At least sometimes, refereeing at Philosophical Studies isn't even double-anonymous. (Indeed, in this case it wasn't even single-anonymous.) I spent two hours sitting with my referee over coffee, arguing about whether his concern about my paper was correct. (This is at odds with their explicit policy of a "double-blind review procedure".)
• There is lots of research showing the potential for various sorts of bias to come into play when one isn't making these decisions via anonymous review. But this is an extremely clear instance in which privilege worked to my advantage. I was a Rutgers PhD, with a job at a prestigious research institution, where my colleague literally down the hall (well, down the hall and around several corners) was the editor, who was also the referee. I was also the kind of person willing to be bold enough to argue with a senior figure who was repeatedly suggesting that my idea wasn't suitable for publication. (It helped that I was already on a first-name basis with him before the story began.) These factors, which had nothing to do with the quality of my submission, played a huge role in getting this early publication. And even with the advantages I had, I was still a fresh PhD without a tenure-track position—my desperation for a publication, combined with the strange relationship I was in with a powerful figure, was disconcerting to say the least. I very much doubt that Cohen understood how awkward my position was, or how uncomfortable it made me.
• In addition to worries about anonymous review, I think it's also potentially problematic that Cohen's biggest concerns about my paper had to do with the particular section where I engaged critically with his own work. I don't think for a minute that he was motivated by a desire not to have critiques of his work published. (And I'm not also totally sure that his worries were unfounded—I think my paper probably improved when I took his worries more seriously.) But I do think that it's easy to be biased in favour of our own views, and that this is something a journal should be very careful to avoid. It wouldn't be practical or particularly desirable, I think, to prohibit referees who are being criticized in submissions—but I do think that this is a situation that calls for an editor (who is not the referee!) to observe particularly carefully, and take the referee's advice in the appropriate context. I don't know to what degree my experience represents a pattern, but I think it would be a very poor feature of a journal if submissions criticizing the ideas of the editor were quite likely to (a) be refereed by the editor himself, and (b) be scrutinized particularly critically. There is a very nearby possible world where I simply deleted the section of my paper critical of Cohen. 

On the whole I'm very confident in asserting that in at least one instance, Philosophical Studies engaged in seriously problematic editorial procedures. I don't know how atypical my story is.

I also want to explicitly note that this story occurred five years ago. I see that Jennifer Lackey and Wayne Davis were both added as Associate Editors of Philosophical Studies in 2011, after my paper was accepted in its final form. (Cohen is still the editor-in-chief.) I don't know how much change that represents. For all I know, it may be that a system is now in place such that stories like mine couldn't occur again. But I'd like to know. I'd like to call for a statement from Philosophical Studies about its editorial practices. (Phil Studies was not one of the journals whose practices were described in the Daily Nous post linked above.) In particular, does the editor still have the discretion to decide to referee a paper himself, with full knowledge of the author? Does the journal sanction the kind of back-and-forth discussion between editor–referee and author I describe? If not, are there procedures in place that prevent it from happening?

I'm not an anonymous-review absolutist; if a journal prefers not to have an anonymous refereeing system, I'm OK with that. But transparency is important; if Philosophical Studies is not a journal run with a commitment to anonymous review, then the philosophical community should probably be thinking of that journal in a very different way than it does.