Wednesday, June 5, 2013

Continuing the "diablog" with Rolf Zwaan -- still more thoughts

Posted by
+Rolf Zwaan just continued our "diablog" (love that term) on reliability and replication. (Rolf -- sorry for slightly misrepresenting your conclusion in my last post, and thanks for clarifying.) At this point, I think we're in complete agreement on pretty much everything in our discussion. I thought I'd comment on one suggestion in his post that was first raised by +Etienne LeBel in the comments on Rolf's first post and that Rolf discussed in his most recent post.

The idea of permitting additional between subjects conditions on registered replication reports is an interesting one. As Rolf notes, that won't work for within-subject designs as the new conditions would potentially affect the measurement of the original conditions. I have several concerns about permitting additional conditions for registered replication reports at Perspectives, but I don't think any of them necessarily precludes additional conditions. It's something the other editors and I will need to discuss more. Here are the primary issues as I see them:  

  • The inclusion of additional conditions should not diminish the sample size for the primary conditions. Otherwise, it would lead to a noisier effect size estimate for the crucial conditions, undermining the primary purpose of the replication reports. Given subject pool constraints and our desire to measure the crucial effects with a maximimal sample size, that could be a problem, particularly at smaller schools.
  • The additional condition must in no way affect the measurements in the primary condition. That is, subjects in the primary conditions could not be aware of the existence of an additional condition. Some measures would need to be taken to avoid any interactions among subjects. That's already something we account for in most designs, so I don't see this as a major impediment.
  • The additional conditions could not be reported alongside the primary analyses in the printed journal article. The issue here is that we want the final published article to report the same measures and tests for each individual replication attempt. Otherwise, the final report will become unwieldy, with each of the many participating labs reporting different analyses. That would hinder the ability of readers to assess the strength of the primary effect under study.
If we do decide to permit additional between-subjects conditions, analyses of those conditions could be reported on the OSF project pages created for each participating lab. There are no page limits for those project pages, and each lab could discuss their additional conditions more fully. I will make sure the other editors (+Alex Holcombe and +Bobbie Spellman) and I discuss this possibility.




No comments:

New comments are not allowed.

Subscribe to: Post Comments (Atom)