Over the recent decades, arXiv has created a culture of open-access and early dissemination that has never existed at this scale in science. At the same time, in some disciplines, the peer-review process has become double-blind. Today, there are research communities, such as mine, that are at the confluence of those two phenomena.
This post discusses the possible solutions to have a smooth interplay between arXiv eraly dissemination and double-blind peer-review.
Note that this post is not a discussion about the pros and cons of arXiv or of double-blind peer-review. Note also that I focus on arXiv because it’s the biggest preprint server, but the whole discussion holds for any other service (PeerJ, etc).
Why submitting to journals and conferences with double-blind peer-review?
In some research communities, you simply don’t have the choice: all journals or conferences, or the most prestigious ones use a double-blind peer-review process.
Why posting on arXiv?
Posting on arXiv is all about early dissemination: posting on arXiv means to be read early, to get feedback, and to be cited early. In addition, posting on arXiv means getting a timestamp on an idea or an experiment in order to be able to claim precedence later on.
Let’s respectively call them the early-dissemination requirement and timestamping requirement.
What is the opposition?
When one post to arXiv:
the paper is indexed by Google, Google Scholar and other academic search engines, which means that if a reviewer does a bibliographic search about the paper, she would find it. Let’s call it the reviewer-on-internet problem (P1).
the paper is announced on arXiv email digests and arXiv RSS feeds, which means that reviewers who monitor those channels will accidentally unblind the paper. Let’s call it the accidental-unblinding problem (P2)
We must be explicit
Status-quo is not an option, we must do something. Otherwise, it will happen that some reviewers reject papers only because they were posted on arXiv, which would be a direct attack against scientific conversation and early dissemination. I don’t want this to happen.
Whatever the solution is, as Charles Sutton emphasizes, the solution must be super clearly and unambiguously explained for both authors and reviewers.
What are the solutions?
Process-based solutions
Here is a review of potential solutions:
S0: Go back to single-blind
Pros: the problem disappears
Cons: one looses double-blind and/or it is impossible to revert such a community change.
S1: Not post to arXiv before the end of the double-blind peer-review process
Pros: the problem disappears
Cons: the peer-review process is too long, too long with respect to scientific dissemination and timely timestamping of ideas. One cannot waits months before posting a hot paper on arXiv.
S2: Clarify the reviewer timeline We could ask reviewers to not search for the paper extensively until they have already formed an opinion. They could first post their opinion in the review system and then update it if they find something unblinding when searching (after submitting their first review). The reason is that, in many cases, the paper under review already appears somewhere on the internet: in a talk abstract, in a blog post, in a project web page, etc. (solution suggested by Robert Feldt).
S3: Introduce a blackout-period. A blackout-period means that authors are not allowed to post on arXiv for X days before the submission and Y days after the submission. This solution has the advantage that it is purely process-based with no changes on arXiv at all.
Pros: Pragmatic solution with no special software development.
Cons: The tricky question is about the appropriate values for X and Y. If they are too high, they clearly conflict with the requirements of early dissemination and timestamping.
S4: Work-around Given that people (hence reviewers) monitor their category of interest, the announce would not be a problem if the paper is in another category. So the idea is to publish one’s paper in another category. For instance, I publish in software engineering and my potential reviewers monitor CS.SE. Instead of registering my paper in CS.SE, I register it somewhere else.
Pros: pragmatic solution with no special software development
Cons: it somehow conflicts with the arXiv moderation process (arXiv moderators are expert in their respective category).
Tool-assisted solutions, with added support for double-blind in arXiv
S5: Temporary anonymization : add support for temporary anonymization in arXiv (eg in the metadata and announces)
Pros: it solves the the reviewer-on-internet problem, and may even partially solves the accidental-unblinding problem.
Cons: it breaks the noble and old concept of “scientific conversation”. Charles Sutton in his post clearly argues against it as well (note that Charles’ Anti-Recommendation is only about this specific kind of support, not the other ones below).
S6: Delayed publication: add support for delayed publication for specific papers.
Pros: This complies with the timestamping requirement.
Cons: It delays early dissemination. It prevents early buzz and citations in hot areas. The duration of the delay is an important parameter to agree on.
S7: Disabled notification: add support for disabling email and announces for specific papers.
Pros: it solves the the accidental-unblinding problem, it keeps the good properties of early dissemination.
Cons: it does not solve the reviewer-on-internet problem (P1). So it must be combined with S2.
All those solutions require adhesion and development effort from the arXiv team.
Recommendation
On the process side, posting reviews before going on the Internet (S2) is an excellent idea.
On the tool side, I have a preference for adding support for disabling email and announces for specific papers (S7), because it specifically targets the accidental-unblinding problem (P2) while staying fully compliant with the early-dissemination and timestamping requirements.
Also, proper support for double-blind in arXiv will encourage some communities to improve their arXiv presence (and the corollary, it will prevent them to do bad choices that would prevent early dissemination).
Acknowledgement
I’d like to thank Charles Sutton, Jim Entwood, Tommy Ohlsson and Robert Feldt for their inputs on that topic.
See also
- Making unblinding manageable: Towards reconciling prepublication and double-blind review
- Open-science and Double-blind Peer-Review