Challenges for Anonymous Review

I'm continuing to think about journals. There are a lot of serious problems, and it's really unclear how to fix them. I want to lay out another problem I haven't seen remarked on before. This one's more about referees and authors than editors. I'll start with a problem that has been remarked on before:

As has recently been observed several times, even with the best editorial practices in the world, it's not very difficult for a referee who isn't committed to anonymous review to google the title of a submission and learn the author. If, like most of us, you don't submit your paper until after it's been around in various forms for a while—you've presented it at workshops, posted drafts online for comments, etc.—the anonymity of the review process is contingent on the whim of the referee. It's possible to take some steps here as an author—you can change the title prior to submission, for example. This will make it harder for a referee to discover your identity, but likely not impossible.

Here's the twist on the problem: the reason that anonymous refereeing is important is to avoid bias—preferential treatment of some submissions (those by established figures, white people, native English speakers, men, people educated at prestigious departments, etc.) over others. So while it's possible for authors to go to extraordinary measures to protect their anonymity, in many cases, it's against their own selfish interests to do so. Suppose for the purpose of argument—and this strikes me as fairly plausible—that I am someone who is more likely to be unfairly benefitted, than unfairly harmed, by a breakdown in anonymous review. I'm at least most of the kinds of people mentioned in that parenthetical. (I've never quite figured out whether I'm a white person or not.) When I submit my paper to a journal, the journal has a rule, and the profession has a norm, that I should anonymise my paper. I remove self-citations or put them into the third person, I omit acknowledgements, etc. But now I have a choice to make: in addition to removing that straightforward information from my submission, should I take my draft down from my website, and change the title? Should I paraphrase key sections that are too similar to the abstract of a talk I gave last year that's still up on a workshop website? If anonymous review were very important to me, I could go to extra effort to preserve it. But if I'm thinking selfishly—and I do think it's reasonable for individuals participating in this system to be making these kinds of decisions selfishly—I'm not going to be very motivated to do so.

It is only those who need the protection from negative bias who are incentivised to go to great lengths to ensure anonymous review. And so now we get to the next layer of the problem: if, as seems likely, the people most incentivised to ensure anonymous review are more likely than others to take the steps to render their submissions ungooglable, then ungooglibility becomes evidence a submission by someone referees are biased against. If a referee decides to google the paper, and finds nothing, the referee can take this to be evidence against the paper's having been written by someone "important".

I can only see two solutions to this problem; both seem very difficult to implement. The first is that authors don't treat these kinds of extraordinary measures as optional—journal policies or disciplinary norms could make clear that authors must not include drafts online, or references to titles, or submit papers with language too similar to language on workshop websites, etc. This seems like a very intrusive requirement, but if the rules could be articulated clearly enough, it might do the job. The other is the obvious one: that referees don't go seeking this information out. This requires sound judgment on the part of referees, and is hard to enforce, but it would also work. The difficulty here, of course, is that it's so hard for editors to get good referees (or any referees), because there's so little incentive to do the job well, ethically, or at all.

I don't know how to make the economics work, but I feel like a lot of things would be much easier if there were some way we could pay referees for their work.