While I can see some risk that I could be suffering a moral blindspot here, I feel like, as someone who's recently been speaking out about philosophy journal editorial practices, and also someone who worked closely for a couple of years with Nous and PPR, that I should make a point of making public my opinion that those journals seem, from all the evidence available to me, to be very well-run. I'm unaware of any editorial misconduct there, and I do not suspect that there has been any. I think the discipline would be a better place if more editors were more similar to Ernie Sosa.
I don't expect anybody to take my opinion as significant evidence on this matter, but I wanted to at least put myself on the record.
Saturday, February 07, 2015
Friday, February 06, 2015
Maximizing the Value in Life
I like Amanda MacAskill's piece on sleeping in. Amanda argues very sensibly that while we think life is the kind of good where it's better to have more of it, we don't measure the relevant quantity by subtracting the date of birth from the date of death; the value in life comes from experiencing it. So ceteris paribus, if you spend more of your life asleep, you're missing out on some of what's good about life. This seems completely right. (Whether Amanda's advice to sleep less is good depends on the degree to which the quality of waking life would be degraded, which of course will vary between individuals.)
I have wondered for a little while, however, about a further development of a point like this. If it is the experience of life that is valuable, such that more of the experience is better than less, even if the biological lifespan is the same, it seems to me that there's also a case to be made that it's better to have more of the experience, even holding fixed the amount of time in which one is conscious. It's a familiar phenomenon that sometimes, the passage of time feels faster than at other times. If I'm sitting at home playing Dominion, I could spend eight hours and barely notice it. But if I'm taking photographs in a city I've never been to before, those eight hours feel much more full of my life.
I suspect, then, that for reasons much like the ones Amanda articulates in her piece, we have some reason not merely to sleep less, but to engage in those activities that slow perceived time down, and avoid those that speed it up. This suggests, for instance, that it might be a great idea to travel more, or that it might be a terrible idea to have children. Ceteris paribus.
I have wondered for a little while, however, about a further development of a point like this. If it is the experience of life that is valuable, such that more of the experience is better than less, even if the biological lifespan is the same, it seems to me that there's also a case to be made that it's better to have more of the experience, even holding fixed the amount of time in which one is conscious. It's a familiar phenomenon that sometimes, the passage of time feels faster than at other times. If I'm sitting at home playing Dominion, I could spend eight hours and barely notice it. But if I'm taking photographs in a city I've never been to before, those eight hours feel much more full of my life.
I suspect, then, that for reasons much like the ones Amanda articulates in her piece, we have some reason not merely to sleep less, but to engage in those activities that slow perceived time down, and avoid those that speed it up. This suggests, for instance, that it might be a great idea to travel more, or that it might be a terrible idea to have children. Ceteris paribus.
Thursday, February 05, 2015
Challenges for Anonymous Review
I'm continuing to think about journals. There are a lot of serious problems, and it's really unclear how to fix them. I want to lay out another problem I haven't seen remarked on before. This one's more about referees and authors than editors. I'll start with a problem that has been remarked on before:
As has recently been observed several times, even with the best editorial practices in the world, it's not very difficult for a referee who isn't committed to anonymous review to google the title of a submission and learn the author. If, like most of us, you don't submit your paper until after it's been around in various forms for a while—you've presented it at workshops, posted drafts online for comments, etc.—the anonymity of the review process is contingent on the whim of the referee. It's possible to take some steps here as an author—you can change the title prior to submission, for example. This will make it harder for a referee to discover your identity, but likely not impossible.
Here's the twist on the problem: the reason that anonymous refereeing is important is to avoid bias—preferential treatment of some submissions (those by established figures, white people, native English speakers, men, people educated at prestigious departments, etc.) over others. So while it's possible for authors to go to extraordinary measures to protect their anonymity, in many cases, it's against their own selfish interests to do so. Suppose for the purpose of argument—and this strikes me as fairly plausible—that I am someone who is more likely to be unfairly benefitted, than unfairly harmed, by a breakdown in anonymous review. I'm at least most of the kinds of people mentioned in that parenthetical. (I've never quite figured out whether I'm a white person or not.) When I submit my paper to a journal, the journal has a rule, and the profession has a norm, that I should anonymise my paper. I remove self-citations or put them into the third person, I omit acknowledgements, etc. But now I have a choice to make: in addition to removing that straightforward information from my submission, should I take my draft down from my website, and change the title? Should I paraphrase key sections that are too similar to the abstract of a talk I gave last year that's still up on a workshop website? If anonymous review were very important to me, I could go to extra effort to preserve it. But if I'm thinking selfishly—and I do think it's reasonable for individuals participating in this system to be making these kinds of decisions selfishly—I'm not going to be very motivated to do so.
It is only those who need the protection from negative bias who are incentivised to go to great lengths to ensure anonymous review. And so now we get to the next layer of the problem: if, as seems likely, the people most incentivised to ensure anonymous review are more likely than others to take the steps to render their submissions ungooglable, then ungooglibility becomes evidence a submission by someone referees are biased against. If a referee decides to google the paper, and finds nothing, the referee can take this to be evidence against the paper's having been written by someone "important".
I can only see two solutions to this problem; both seem very difficult to implement. The first is that authors don't treat these kinds of extraordinary measures as optional—journal policies or disciplinary norms could make clear that authors must not include drafts online, or references to titles, or submit papers with language too similar to language on workshop websites, etc. This seems like a very intrusive requirement, but if the rules could be articulated clearly enough, it might do the job. The other is the obvious one: that referees don't go seeking this information out. This requires sound judgment on the part of referees, and is hard to enforce, but it would also work. The difficulty here, of course, is that it's so hard for editors to get good referees (or any referees), because there's so little incentive to do the job well, ethically, or at all.
I don't know how to make the economics work, but I feel like a lot of things would be much easier if there were some way we could pay referees for their work.
As has recently been observed several times, even with the best editorial practices in the world, it's not very difficult for a referee who isn't committed to anonymous review to google the title of a submission and learn the author. If, like most of us, you don't submit your paper until after it's been around in various forms for a while—you've presented it at workshops, posted drafts online for comments, etc.—the anonymity of the review process is contingent on the whim of the referee. It's possible to take some steps here as an author—you can change the title prior to submission, for example. This will make it harder for a referee to discover your identity, but likely not impossible.
Here's the twist on the problem: the reason that anonymous refereeing is important is to avoid bias—preferential treatment of some submissions (those by established figures, white people, native English speakers, men, people educated at prestigious departments, etc.) over others. So while it's possible for authors to go to extraordinary measures to protect their anonymity, in many cases, it's against their own selfish interests to do so. Suppose for the purpose of argument—and this strikes me as fairly plausible—that I am someone who is more likely to be unfairly benefitted, than unfairly harmed, by a breakdown in anonymous review. I'm at least most of the kinds of people mentioned in that parenthetical. (I've never quite figured out whether I'm a white person or not.) When I submit my paper to a journal, the journal has a rule, and the profession has a norm, that I should anonymise my paper. I remove self-citations or put them into the third person, I omit acknowledgements, etc. But now I have a choice to make: in addition to removing that straightforward information from my submission, should I take my draft down from my website, and change the title? Should I paraphrase key sections that are too similar to the abstract of a talk I gave last year that's still up on a workshop website? If anonymous review were very important to me, I could go to extra effort to preserve it. But if I'm thinking selfishly—and I do think it's reasonable for individuals participating in this system to be making these kinds of decisions selfishly—I'm not going to be very motivated to do so.
It is only those who need the protection from negative bias who are incentivised to go to great lengths to ensure anonymous review. And so now we get to the next layer of the problem: if, as seems likely, the people most incentivised to ensure anonymous review are more likely than others to take the steps to render their submissions ungooglable, then ungooglibility becomes evidence a submission by someone referees are biased against. If a referee decides to google the paper, and finds nothing, the referee can take this to be evidence against the paper's having been written by someone "important".
I can only see two solutions to this problem; both seem very difficult to implement. The first is that authors don't treat these kinds of extraordinary measures as optional—journal policies or disciplinary norms could make clear that authors must not include drafts online, or references to titles, or submit papers with language too similar to language on workshop websites, etc. This seems like a very intrusive requirement, but if the rules could be articulated clearly enough, it might do the job. The other is the obvious one: that referees don't go seeking this information out. This requires sound judgment on the part of referees, and is hard to enforce, but it would also work. The difficulty here, of course, is that it's so hard for editors to get good referees (or any referees), because there's so little incentive to do the job well, ethically, or at all.
I don't know how to make the economics work, but I feel like a lot of things would be much easier if there were some way we could pay referees for their work.
Monday, February 02, 2015
Journal Editorial Practices and an Anecdote
I was interested to read the recent Daily Nous conversation about journal editorial practices. Much of the discussion there focused on whether double-anonymous refereeing (where the authors and referees are ignorant of each others' identities) was good enough, or whether triple-anonymous methods (where the editors are also unaware of the authors' identities) are necessary for a fair system.
I was interested to see many journals' editors describing their methods, and how they try to avoid bias. One question I didn't see addressed, however, concerns oversight. Is there any system in place to confirm whether journals are actually run the way they're said to? I would consider this a somewhat paranoid question, except that I know firsthand of at least one high-profile journal which at least sometimes, at the discretion of the editor, suspends its official policy of double-anonymous refereeing.
My experience happened shortly after I finished my PhD, when I was a postdoc. I knew at the time it was a problematic situation, but I decided it probably wasn't prudent to make a fuss at that stage in my career. I'd been intending to wait until I was tenured, but I think that given the conversations happening in the discipline at the moment, now is an appropriate time for the story to come to light. This is the story of "Quantifiers and Epistemic Contextualism," one of my first papers defending contextualism, which was eventually published in Philosophical Studies.
I apologize for the length of the narrative. I'd make more of a point of being concise, but I want to make sure that I explain exactly what happened, without editorial summarizing. (I'll express some opinions after telling the story.)
So here's the story.
After defending my PhD in 2008, and beginning work as a postdoc in St Andrews the same year, I started to develop some of the ideas from my dissertation into publishable papers. I submitted one of them, a defence of contextualism, to Philosophical Studies in March of 2009. That journal uses an online program for submissions where an author can watch the progress of his submission as it goes through various stages—'editor assigned', 'reviewer assigned', 'review completed', etc. I noticed in July that four months had passed and it still just said 'editor assigned'. (I knew from a previous paper that it ought to be progressing through other stages.) So I submitted a query through the online submission, asking whether my paper might have fallen through the cracks.
I received a response to that question in the form of a personal email from Stewart Cohen, the editor of the journal. Cohen wrote that "the reason your paper is listed as editor assigned, is that I'm going to review it myself." He said he hoped to get a chance to review it soon, and invited me to email him directly if I had any further questions. This happened in July. (Potentially relevant background not everyone might know: Cohen wrote several influential papers in defence of contextualism in the 1980s and 1990s; he is one of the prominent figures in this subfield of epistemology.)
In September 2009, Cohen was hired as a part-time professorial fellow at Arché, the same institution where I was working.
In October, Cohen came to St Andrews in connection with his new job. He spoke briefly with me about my submission then, saying that he generally liked it, but had some particular concerns. He sent me a lengthy email detailing them, and we had a rather involved back-and-forth exchange on various of the substantive philosophical questions. At the conclusion of this exchange, after I'd agreed to make various changes to the paper, we'd gotten down to just one major remaining point of disagreement. In my paper, I argued that my neo-Lewisian form of contextualism avoids a challenge that Cohen himself had leveled against Lewis, concerning his ability to handle both Gettier cases and lottery cases. Cohen had argued, in a 1998 paper, that Lewis can't have it both ways; but I suggested in my submission that the issue could be avoided. Cohen was unconvinced by my treatment of this part of the paper, but invited me to work on it further. He wrote to me in an email:
Ok, those were 'just the facts'. From here on out, I'm editorializing. These seem to me the important problematic points to emphasize:
I also want to explicitly note that this story occurred five years ago. I see that Jennifer Lackey and Wayne Davis were both added as Associate Editors of Philosophical Studies in 2011, after my paper was accepted in its final form. (Cohen is still the editor-in-chief.) I don't know how much change that represents. For all I know, it may be that a system is now in place such that stories like mine couldn't occur again. But I'd like to know. I'd like to call for a statement from Philosophical Studies about its editorial practices. (Phil Studies was not one of the journals whose practices were described in the Daily Nous post linked above.) In particular, does the editor still have the discretion to decide to referee a paper himself, with full knowledge of the author? Does the journal sanction the kind of back-and-forth discussion between editor–referee and author I describe? If not, are there procedures in place that prevent it from happening?
I'm not an anonymous-review absolutist; if a journal prefers not to have an anonymous refereeing system, I'm OK with that. But transparency is important; if Philosophical Studies is not a journal run with a commitment to anonymous review, then the philosophical community should probably be thinking of that journal in a very different way than it does.
I was interested to see many journals' editors describing their methods, and how they try to avoid bias. One question I didn't see addressed, however, concerns oversight. Is there any system in place to confirm whether journals are actually run the way they're said to? I would consider this a somewhat paranoid question, except that I know firsthand of at least one high-profile journal which at least sometimes, at the discretion of the editor, suspends its official policy of double-anonymous refereeing.
My experience happened shortly after I finished my PhD, when I was a postdoc. I knew at the time it was a problematic situation, but I decided it probably wasn't prudent to make a fuss at that stage in my career. I'd been intending to wait until I was tenured, but I think that given the conversations happening in the discipline at the moment, now is an appropriate time for the story to come to light. This is the story of "Quantifiers and Epistemic Contextualism," one of my first papers defending contextualism, which was eventually published in Philosophical Studies.
I apologize for the length of the narrative. I'd make more of a point of being concise, but I want to make sure that I explain exactly what happened, without editorial summarizing. (I'll express some opinions after telling the story.)
So here's the story.
After defending my PhD in 2008, and beginning work as a postdoc in St Andrews the same year, I started to develop some of the ideas from my dissertation into publishable papers. I submitted one of them, a defence of contextualism, to Philosophical Studies in March of 2009. That journal uses an online program for submissions where an author can watch the progress of his submission as it goes through various stages—'editor assigned', 'reviewer assigned', 'review completed', etc. I noticed in July that four months had passed and it still just said 'editor assigned'. (I knew from a previous paper that it ought to be progressing through other stages.) So I submitted a query through the online submission, asking whether my paper might have fallen through the cracks.
I received a response to that question in the form of a personal email from Stewart Cohen, the editor of the journal. Cohen wrote that "the reason your paper is listed as editor assigned, is that I'm going to review it myself." He said he hoped to get a chance to review it soon, and invited me to email him directly if I had any further questions. This happened in July. (Potentially relevant background not everyone might know: Cohen wrote several influential papers in defence of contextualism in the 1980s and 1990s; he is one of the prominent figures in this subfield of epistemology.)
In September 2009, Cohen was hired as a part-time professorial fellow at Arché, the same institution where I was working.
In October, Cohen came to St Andrews in connection with his new job. He spoke briefly with me about my submission then, saying that he generally liked it, but had some particular concerns. He sent me a lengthy email detailing them, and we had a rather involved back-and-forth exchange on various of the substantive philosophical questions. At the conclusion of this exchange, after I'd agreed to make various changes to the paper, we'd gotten down to just one major remaining point of disagreement. In my paper, I argued that my neo-Lewisian form of contextualism avoids a challenge that Cohen himself had leveled against Lewis, concerning his ability to handle both Gettier cases and lottery cases. Cohen had argued, in a 1998 paper, that Lewis can't have it both ways; but I suggested in my submission that the issue could be avoided. Cohen was unconvinced by my treatment of this part of the paper, but invited me to work on it further. He wrote to me in an email:
Let's call it a conditional acceptance. I'm not sure you've made any advance in applying the RA view to Gettier cases and so I don't want to commit to publishing that part of the paper. You may convince me that I'm wrong, but even if you don't, you can always just cut that part. So unless you think you'll refuse to cut it, even if I'm not satisfied, you can list it as forthcoming.In November 2009, I submitted (via direct email to Cohen) a revised version of the paper. We corresponded briefly about it in January when Cohen was visiting Arché again, but he didn't read and respond to my new draft until April 2010, when he wrote back, saying that he was still unconvinced by my treatment of lotteries and Gettier cases. We had another email exchange, which continued until his next visit to Arché in May. On May 24, we met over coffee about whether the section in question ought to be published. By the end of that meeting, he did agree to publish it, and on May 25 I received an official acceptance from the journal, and an invitation to upload my final version. The paper did end up being published, including the revised version of the section defending Lewis from Cohen.
Ok, those were 'just the facts'. From here on out, I'm editorializing. These seem to me the important problematic points to emphasize:
- The editor of Philosophical Studies seems to be treating submissions in his own area of research interest very differently from the way he treats other submissions, sometimes deciding to referee papers himself. As far as I know, Cohen was the sole referee for my submission.
- At least sometimes, refereeing at Philosophical Studies isn't even double-anonymous. (Indeed, in this case it wasn't even single-anonymous.) I spent two hours sitting with my referee over coffee, arguing about whether his concern about my paper was correct. (This is at odds with their explicit policy of a "double-blind review procedure".)
- There is lots of research showing the potential for various sorts of bias to come into play when one isn't making these decisions via anonymous review. But this is an extremely clear instance in which privilege worked to my advantage. I was a Rutgers PhD, with a job at a prestigious research institution, where my colleague literally down the hall (well, down the hall and around several corners) was the editor, who was also the referee. I was also the kind of person willing to be bold enough to argue with a senior figure who was repeatedly suggesting that my idea wasn't suitable for publication. (It helped that I was already on a first-name basis with him before the story began.) These factors, which had nothing to do with the quality of my submission, played a huge role in getting this early publication. And even with the advantages I had, I was still a fresh PhD without a tenure-track position—my desperation for a publication, combined with the strange relationship I was in with a powerful figure, was disconcerting to say the least. I very much doubt that Cohen understood how awkward my position was, or how uncomfortable it made me.
- In addition to worries about anonymous review, I think it's also potentially problematic that Cohen's biggest concerns about my paper had to do with the particular section where I engaged critically with his own work. I don't think for a minute that he was motivated by a desire not to have critiques of his work published. (And I'm not also totally sure that his worries were unfounded—I think my paper probably improved when I took his worries more seriously.) But I do think that it's easy to be biased in favour of our own views, and that this is something a journal should be very careful to avoid. It wouldn't be practical or particularly desirable, I think, to prohibit referees who are being criticized in submissions—but I do think that this is a situation that calls for an editor (who is not the referee!) to observe particularly carefully, and take the referee's advice in the appropriate context. I don't know to what degree my experience represents a pattern, but I think it would be a very poor feature of a journal if submissions criticizing the ideas of the editor were quite likely to (a) be refereed by the editor himself, and (b) be scrutinized particularly critically. There is a very nearby possible world where I simply deleted the section of my paper critical of Cohen.
I also want to explicitly note that this story occurred five years ago. I see that Jennifer Lackey and Wayne Davis were both added as Associate Editors of Philosophical Studies in 2011, after my paper was accepted in its final form. (Cohen is still the editor-in-chief.) I don't know how much change that represents. For all I know, it may be that a system is now in place such that stories like mine couldn't occur again. But I'd like to know. I'd like to call for a statement from Philosophical Studies about its editorial practices. (Phil Studies was not one of the journals whose practices were described in the Daily Nous post linked above.) In particular, does the editor still have the discretion to decide to referee a paper himself, with full knowledge of the author? Does the journal sanction the kind of back-and-forth discussion between editor–referee and author I describe? If not, are there procedures in place that prevent it from happening?
I'm not an anonymous-review absolutist; if a journal prefers not to have an anonymous refereeing system, I'm OK with that. But transparency is important; if Philosophical Studies is not a journal run with a commitment to anonymous review, then the philosophical community should probably be thinking of that journal in a very different way than it does.
Subscribe to:
Posts (Atom)