Registered Replication Reports - Stay Tuned!

Almost exactly one year ago, Alex Holcombe and I met in Florida (while at the Vision Sciences conference) to discuss our concerns with the state of the scientific literature and the problems with the incentive structure for publishing. Now, we're excited to report that our schemes are about to bear fruit. 

Tomorrow, we will be announcing the first of what we hope will be many Registered Replication Report protocols to be published at APS's journal, Perspectives on Psychological Science. I'll post details here tomorrow morning - stay tuned!

I thought it might be interesting, before that announcement, to give a little of the backstory for how this new initiative came about. 

Motivations

For good reasons, many journals favor novelty and originality. With adequately powered studies, that preference might be okay, but most psychology studies are woefully underpowered, meaning that the effect size estimates they provide are relatively unstable and noisy. Moreover, the emphasis on null hypothesis significance testing (NHST) means that statistically significant results are treated as robust, even when a study is underpowered. We felt that the real emphasis should be on estimating the actual size of effects in the world, and for that purpose, statistical significance is the wrong metric. In fact, as this video from Geoff Cumming shows, the p-value from an individual study is not particularly diagnostic of the size of the underlying effect (unless the sample size is huge):

The problem with relying on statistical significance is amplified by a slew of questionable practices, including p-hacking, optional stopping, data peeking, etc. that have been addressed recently. Although these issues with NHST have been well documented for decades with little change in practices from researchers, Alex and I had both encountered increased interest in confronting these problems and changing how the field does business. The problem was one of incentives.

When dealing with inference from a study to the true size of an effect in reality, direct replications are needed. Only through multiple, direct replication using a shared protocol can we arrive at an accurate meta-analytic effect size estimate that overcomes the inferential shortcomings of any individual study. (The one exception might be studies that are so highly powered that they effectively do not require an inference to generalize to the full population.) Meta-analysis can help, but they are subject to uncontrolled differences in procedures, file drawer problems, etc. Yet, the incentives against publishing direct replications have been enormous. 

Publication is the currency of our field, but direct replications have been undervalued relative to novelty. Anyone who has tried to publish a direct replication that did not find the same effect as the original knows the challenges of overcoming biases in the publication process. There are many reasons a replication might produce a smaller effect, and those reasons often provide a rationale for rejection. 

These factors, taken in combination, have produced a proliferation of exciting, counter-intuitive, new findings with almost no published direct replications to verify the size of these effects in reality and few publications of studies that disconfirm original findings.  Our goal was to establish a new journal, one dedicated to publishing direct replications of important results, that could help to change those incentives. We felt the field needed a journal that focused not on publishing "null results" but on publishing the outcome of direct replications regardless of their outcome. 

We wanted our new journal to adhere to the following principles:

• Use vetted, pre-registered plans for the study procedure and analysis so that each study would be an accurate, direct replication of an original result
• Have all studies use the same protocol, with the results published regardless of the outcome
• Identify studies that have high replication value, those that are theoretically or practically important but that have not been directly replicated
• Publish the findings in an open-access format so that they could provide a definitive assessment of the size of an effect that would be accessible both to researchers and to the public (and media)
• Make all data from each study publicly available as well
• Emphasize the estimation of the population size of an effect rather than the statistical significance (succeed/failure) of individual replication attempts

The journal would provide an outlet for direct replication, providing an incentive for conducting such studies. If successful, it could induce broader changes in the field. For example, it might help to stem the use of questionable research practices that lead to significant but possibly spurious results. If you used questionable research practices to obtain a spiffy new finding, the prospect of having multiple labs attempt direct replications would give you pause. We hoped that the presence of such a journal might lead researchers to verify their own results before publishing, ideally using a pre-registered design and analysis and greater power. We viewed this journal not as an effort to debunk iffy findings (although that might happen), but as an effort to shore up our science for the future.

APS answers the call

Alex and I did a lot of legwork in developing our idea for the journal. We explored possible publishers and outlets, consulted colleagues, developed a mission statement, guiding principles, procedures for the submission and review process, lists of possible editorial board members, possible journal names, etc. By mid-July of last year, we were about ready to enter the final stages of development. At that point, we decided to seek guidance from experienced editors and leaders of academic societies to explore the factors that might lead to the success or failure of a new journal. I wrote to several publishing luminaries and colleagues to seek their advice, including Alan Kraut and Roddy Roediger at APS. All had excellent suggestions and feedback. 

Unexpectedly, a few weeks later Alan Kraut asked me to attend a small meeting in St. Louis to discuss the state of publishing in our field with some of the publishing board members and others at APS. As it turned out, APS had been actively working on a set of initiatives to improve reporting and publishing practices, with Eric Eich (editor in chief of Psychological Science) taking the lead. I couldn't attend the start of the meeting due to prior obligations, but I arrived at dinner after the rest of the attendees had been meeting all day. It was there that I learned that the group had discussed the ideas that Alex and I had developed, and they thought APS should adopt them. I was happily stunned. That APS wanted to address the issues in publishing head on has given me more cause for optimism about the state of our science than I have had for years. 

As with any large-scale new society initiative, we had to overcome some resistance, but the process was remarkably smooth given the scope of what we are trying. For some time, it was unclear whether the new registered replication reports would be published independently or incorporated into one of APS's journals. Then, after it became clear that hosting it in an established journal would give the best odds of success, it was unclear which journal would house it. In the end, with Bobbie Spellman's advocacy as editor in chief, we decided that it would be published in Perspectives on Psychological Science. The journal has become the go-to place for discussions of scientific practices in psychology, and it provides a natural home for this experimental initiative.

Once it become clear that Perspectives would host the Registered Replication Reports, Alex and I took on roles as associate editors and refined our materials to better fit their new home. We publicly launched the Registered Replication Reports initiative in March, and the month or so since then, we received a number of excellent proposals, many of which are wending their way through the process of devising and vetting an accurate protocol for a direct replication. Over the coming months, we will be announcing a number of accepted protocols. And, we hope that many researchers in the field will contribute their own direct replications to these projects. 

The first approved protocol, developed with the aid of the original author, will be announced tomorrow! Stay tuned.