Guides · Replication and generalization

Designing a replication study: a step-by-step plan

Choose an important finding with a real reason for doubt, pin down the exact effect and its original size, and obtain or rebuild the materials. Record every difference from the original, plan the sample for an effect smaller than the published one, preregister the analysis and the success criterion, and judge the result against both zero and the original estimate.

Step 1: choose a finding worth the effort

Replications cost time and participants, so the target should matter for theory or for practice. An effect that already has several close replications is a less urgent target than one that has none.^[1]

The strength of the original evidence is a useful signal of where doubt is warranted. In the large psychology reproducibility project, whether a replication succeeded was better predicted by the strength of the original evidence than by characteristics of the original or replication teams.^[2]

Before committing, you can screen a set of published results for warning signs. A toolbox developed for consumer research applies heuristics such as p-curve, z-curve and caliper tests to the statistics reported in articles to flag patterns consistent with publication bias or p-hacking.^[3]

A long record of conflicting replications can itself justify one more attempt. The authors of a replication of single-exposure musical conditioning argued that earlier mixed outcomes left the question open, and that the accumulated criticism told them exactly which design problems to fix.^[4]

Step 2: define the effect and get the materials

Write down the precise effect you intend to replicate, with its original effect size, confidence interval and sample size, where and in which country the study ran, what kind of sample it used and how it was administered. In complex designs decide which result is the target, for example the interaction or one of its simple effects, because that choice drives the rest of the design.^[1]

Ask the original authors for their materials. If they are unavailable, rebuild the procedure from the published method section on the assumption that the original study was run rigorously, and ask experts in the area whether your version matches it.^[1]

Stimuli tied to a time or culture may need replacing with functionally equivalent ones, ideally developed with the original authors and checked in a pilot study. A French replication of a product-placement study ran pretests showing that French viewers perceived the original American film clips in roughly the same way as the original participants did.^[1][5]

Step 3: decide how close to stay and record every difference

For each element of the study, including instructions, measures, stimuli, procedure, location, payment and participant population, classify the replication as exact, close or conceptually different from the original and justify the choice. Readers can then judge whether each deviation was reasonable and where the study sits between close and conceptual, and future work can use the recorded differences to find boundary conditions.^[1]

Anticipate differences that could change the effect, such as participants' social class or mood, and measure them at the end of the study so they can be tested as moderators without disturbing the replication. Include manipulation checks and measures of any preconditions the theory requires, so that a failure to create the right test conditions can be told apart from a failed prediction.^[1][6]

Published replications show two practical ways to present closeness. A key account management study worked in stages, first imitating the original design and measures with new data and only then adding new constructs, and an IKEA-effect replication included an appendix table setting out the similarities and differences between its study and the two originals.^[7][8]

Varying several conditions at once is acceptable when the aim is generalization, since success then covers more ground, but if the result fails you will not know which change was responsible. Adding some conditions that should not matter is a cheap way to confirm that they indeed do not.^[9]

Step 4: plan the sample for a smaller effect than the published one

Published effects tend to overstate true effects. Replication effects in the psychology project averaged about half the original size, and because publication bias favors large estimates, a power analysis based on the published effect gives less power than it appears to.^[2][10]

Aim for power of at least .80 and preferably up to .95, erring high. One suggested alternative is a sample 2.5 times the original. If earlier replications exist, base the calculation on their meta-analytic average, and if one lab cannot reach the required sample, pool data collection with other labs.^[1]

A worked example: the musical conditioning replicators recalculated the original effect conservatively, because the original study had dropped participants in a way that could inflate it. Their power analysis called for 107 participants to reach .95 power, and they collected 182, 224 and 127 in their three experiments.^[4]

Step 5: fix the plan before collecting data

Preregistration means stating the research questions and analysis plan before seeing the outcomes, which keeps the tests you predicted separate from patterns noticed afterwards. Practical variants exist for situations short of the ideal, such as when the data already exist.^[11]

For a replication, register the methods, the documented differences from the original, the analysis plan and the definition of a successful replication. A replication template on the Open Science Framework supports this in two parts, one completed before data collection and one after.^[1]

The plan should close off the flexibility that inflates false positives: decide the rule for stopping data collection in advance, list every variable collected, report every condition, and show results with and without any excluded observations or covariates.^[12]

Step 6: set the success criterion in advance

Calling a replication a success or failure only because it reached significance is not enough. Report two tests: whether the replication effect differs from zero, and whether it differs from the original effect. Together they give four possible verdicts.^[1]

• Success: different from zero, in the same direction as the original, and similar to or larger than it.^[1]
• Informative failure: not different from zero or opposite in direction, and significantly different from the original.^[1]
• Practical failure: significantly different from both zero and the original, which in this scheme means a real but smaller effect.^[1]
• Inconclusive: significantly different from neither, so the estimate is too imprecise to decide.^[1]
• Meta-analytic check: pool the original and the replication. If the effects are not heterogeneous and the pooled effect is significant, the original conclusion stands; if the pooled effect is not significant, it is in doubt; if they are heterogeneous, look for a moderator. One published failure to replicate an anchoring effect had an estimate fully consistent with the original; the replication simply had less power.^[10]
• Other criteria exist, such as asking whether the replication estimate falls inside the original's confidence interval. Under that rule the musical conditioning replication counted its first experiment as a failure even though the effect was significant, because it was reliably smaller than the original.^[4]

Step 7: report so the result can be pooled

Report the replication effect size and its confidence interval, the test against the original, and the limitations of both studies. Make the data, analysis code and every analysis available so others can evaluate the attempt or try their own, with exceptions only where data protection requires them.^[1]

When replication is part of an original paper, report all relevant studies including non-replications, with documented data and code, and add a short statement of what the results imply for the generality of the earlier work.^[6]

Keep the conclusion proportionate. Even a large, carefully staged replication is one more data point, and its authors caution that a single replication should not be the last word on a topic.^[7][1]

The evidence behind this

• Research methods pitfalls: ratio variables and censored data
• Marketing knowledge: how it is made and when to trust it

References

1. Mark J. Brandt, Hans IJzerman, Ap Dijksterhuis, Frank J. Farach, Jason Geller, Roger Giner‐Sorolla, James A. Grange, Marco Perugini, Jeffrey R. Spies, Anna van 't Veer (2013). The Replication Recipe: What makes for a convincing replication?. Journal of Experimental Social Psychology.
2. Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science.
3. Susanne Jana Adler, Lukas Röseler, Martina Katharina Schöniger (2023). A toolbox to evaluate the trustworthiness of published findings. Journal of Business Research.
4. Ivar Vermeulen, Camiel J. Beukeboom (2015). Effects of Music in Advertising: Three Experiments Replicating Single-Exposure Musical Conditioning of Consumer Choice (Gorn 1982) in an Individual Setting. Journal of Advertising.
5. Felipe Pantoja, Patrícia Rossi, Adilson Borges (2015). How Product-Plot Integration and Cognitive Load Affect Brand Attitude: A Replication. Journal of Advertising.
6. Oleg Urminsky, Berkeley J Dietvorst (2024). Taking the Full Measure: Integrating Replication into Research Practice to Assess Generalizability. Journal of Consumer Research.
7. Dennis Herhausen, Björn Sven Ivens, Robert Spencer, Michael Weibel (2021). Key account management configurations and their effectiveness: A quasi-replication and extension. Industrial Marketing Management.
8. Marko Sarstedt, Doreen Neubert, Kati Barth (2016). The IKEA Effect: A Conceptual Replication. Journal of Marketing Behavior.
9. R. Murray Lindsay, Andrew S. C. Ehrenberg (1993). The Design of Replicated Studies. The American Statistician.
10. John Lynch, Eric T. Bradlow, Joel Huber, Donald R. Lehmann (2015). Reflections on the replication corner: In praise of conceptual replications. International Journal of Research in Marketing.
11. Brian A. Nosek, Charles R. Ebersole, Alexander C. DeHaven, David T. Mellor (2018). The Preregistration Revolution. Proceedings of the National Academy of Sciences.
12. Joseph P. Simmons, Leif D. Nelson, Uri Simonsohn (2011). False-Positive Psychology. Psychological Science.

Bibliographic details only. Follow the DOI to read each study at its publisher.