A step too far: What replications do and do not imply

An article that appearing in Nature yesterday focused on a recent paper in PLoS ONE that reported a failure to replicate a classic social priming study by Ap Dijksterhuis. Unfortunately, the headline and thrust of that Nature piece incorrectly linked a failed replication attempt to the noted case of Stapel's fraud. It inappropriately impugned an entire discipline (social psychology) and country (the Netherlands) based on a single paper that was unable to replicate a few related priming effects. That's not just a step, but an entire staircase too far.

Replication failures do NOT indicate fraud. They are a part of the normal scientific process. Even if the original effect is completely true, some proportion of studies will fail to replicate it. And, even if it is false, one negative result does not prove that. Replications can fail for many reasons (and spurious positive results can occur as well). Replications like the PLoS paper add evidence. Perhaps the original effect is a little weaker than previously thought. Or, perhaps it is more sensitive to subtle differences in method. Or, perhaps the replication attempts were not done correctly. Or, perhaps the original study was not done correctly. Only with multiple, independent, direct replications can we better estimate the true underlying effect. A single result, positive or negative, does not provide definitive evidence. It is informative. Just not that informative.

I spoke the author of the Nature piece for 20 minutes when she was researching it. I didn't comment on the latest replication controversy. Instead, I talked with her about how psychology is taking the lead on addressing replicability and best practices. I talked about the great new initiatives at APS and other journals, and the changing incentives that will promote publication of direct replication attempts. Those changing incentives will improve the state of our science going forward, and they might generalize to other fields as well. Unfortunately, none of those positive comments made it into the piece.

If you want to read more, I'd recommend what  had to say about the controversy.

Update: Fixed some formatting.

Update 2: I fixed the typo in the title of the post (to -> too) almost immediately after publishing, but blogger won't let me fix the link. I guess it's permanent. Blech.