Guides · Research credibility and open science

Publication bias and inflated effects: how to read a literature

Published effects tend to be too large because significant results are more likely to be written up and published, and because flexible analysis can turn noise into significance. Bias-corrected reanalyses show how big the gap can be: a nudge meta-analysis average near d = 0.43 shrank to roughly zero after correction, and advertising elasticities fell more than fivefold. Before citing an average effect, check the share of significant results, whether small studies report bigger effects, clustering just past the significance threshold, and whether corrected estimates agree.

Why published effects run high

Two filters sit between a study and its citation. The first is selective publication: novel, positive and clean results are better rewarded, so null findings are more likely to stay in the file drawer. The second is flexible analysis, where choices about exclusions, covariates and outcomes are made after seeing the data. Both push the published record toward significant results.^[1][2]

When only significant estimates get through, small studies that happened to overestimate the effect are the ones that clear the bar, so the average of published estimates sits above the true value. DellaVigna and Linos found this pattern in academic nudge trials, where less precise studies reported larger effects. The flexibility filter is powerful on its own: in the simulations of Simmons and colleagues, combining four common analytic choices produced a false positive in about 61 percent of studies of a null effect.^[3][4]

How large the gap can be

• Replication: the Open Science Collaboration repeated 100 psychology studies. Of the originals, 97 percent were significant, but only 36 percent of the replications were, and the mean replication effect (r = 0.197) was about half the original mean (r = 0.403).^[5]
• Scale: DellaVigna and Linos compared 126 trials covering 23 million people, run by two US nudge units that kept records of every trial, with nudge trials published in academic journals. The published trials averaged an 8.7 percentage point effect on take-up, the nudge-unit trials 1.4 points. Selective publication, made worse by low statistical power, explained about 70 percent of the difference.^[3]
• Power: in the same comparison the median treatment arm had 484 participants in the academic sample against 10,006 in the nudge-unit sample, so most academic trials could detect only large effects.^[3]
• Consumer research: Adler and colleagues cite a tally in which only about 10 percent of 44 replication attempts of consumer research findings were successful.^[6]

Worked example: the nudging meta-analysis debate

Mertens and colleagues pooled choice architecture studies across behavioural domains and reported an average effect of d = 0.43, with a 95 percent interval of 0.38 to 0.48. They found that interventions changing the structure of choices outperformed those that only changed information, and they reported moderate publication bias.^[7]

Three letters in the same journal reanalysed the corrected dataset the authors had shared. Maier and colleagues applied robust Bayesian meta-analysis, which averages over several selection and regression models. The combined effect fell to 0.04, with a credible interval from 0 to 0.14 and a Bayes factor near one, meaning the data could no longer tell an effect from no effect. They found evidence against any effect for information and assistance nudges, an undecided result for structural nudges, and strong heterogeneity, so some nudges may still work.^[8]

Szaszi and colleagues pointed out that the original paper's own sensitivity analysis already put the adjusted average between 0.08 under severe bias and 0.31 under moderate bias. Three further correction methods gave 0.01, 0.07 and 0.08. They argued that the unexplained variation across studies matters more than the average, and named defaults as one of the few nudges with reason to expect sizeable effects.^[9]

Bakdash and Marusich plotted the raw effect distributions by domain. Four domains showed a sharp cut-off at or just below zero with long tails of large positive effects, a shape consistent with suppressed unfavourable results. Nearly two thirds of all effects fell below the overall pooled mean, so the mean did not describe a typical study.^[10]

The lesson is not that nudges never work. The corrections disagree on the exact size, but all three letters conclude the unadjusted average overstates the typical effect and hides wide variation, which matches the much smaller effects DellaVigna and Linos found at scale. The debate was possible because the original authors shared their data and code.^{[8][9][3][10]}

A marketing example: advertising elasticities

Sethuraman, Tellis and Briesch pooled 751 short-term and 402 long-term brand advertising elasticities from 56 studies and reported averages of 0.12 and 0.24. These were simple averages, already well below earlier meta-analytic means of 0.22 and 0.41.^[11][12]

Korkames, Stanley and Stremersch reanalysed 538 elasticities. Simple averages on their data matched the earlier literature, but weighting by precision cut them sharply, and a regression correction for funnel asymmetry gave 0.0008 in the short term and 0.03 in the long term, more than five times smaller and closer to results from randomized experiments. They attribute the drop to better methods that address misspecification and aggregation as well as publication selection, and they still find elasticities approaching 0.10 under favourable conditions.^[12]

Checks you can run on a literature

Adler and colleagues built a toolbox of rules of thumb that work from the test statistics reported in published papers. Their procedure is to define the effect of interest, choose one focal test per independent sample under a disclosed selection rule, extract the statistics, and read several diagnostics together.^[6]

• Count the significant results. If almost every focal test is significant, ask where the rest went. In the 2022 volume of the Journal of Consumer Psychology, 92 percent of 150 focal tests were significant; the psychology originals later replicated were 97 percent significant.^[6][5]
• Plot effect size against precision. If smaller or noisier studies report bigger effects, the funnel is asymmetric and selection is likely.^[3][12]
• Look just past the threshold. A caliper test compares how many results fall just above and just below the significance cut-off; for mediation, check whether confidence intervals cluster just clear of zero.^[6]
• Run both a p-curve and a z-curve. For the 2022 journal sample, the p-curve pointed to a real effect with 85 percent estimated power, but that estimate is an upper bound; the z-curve, which models selection, expected only 13 percent of studies to be significant if all were rerun and 41 percent of significant results to replicate.^[6]
• Be wary of small studies with perfect results. Simmons and colleagues advise reviewers that underpowered studies where every pattern comes out as predicted deserve more scrutiny, not less.^[4]
• Compare several bias corrections and report the range when they disagree, as the nudge letters did.^[8][9][12]
• Look at the spread, not just the mean: how much effects vary and what share of studies fall below the pooled average.^[10][9]
• Seek evidence that escaped the filter, such as large replications, registered studies and complete trial registries.^[5][3][1]

Limits of these checks

The toolbox diagnoses a well-defined set of studies, not a single study, and its results are unreliable when only a handful of studies exist. It cannot see unpublished work or practices such as dropping variables before publication. Adler and colleagues also warn against using it to declare a theory false: weak evidence is not the same as a false effect, and a finding flagged for likely bias is better treated as a candidate for rigorous replication.^[6]

Corrections rest on assumptions about how selection works, and methods can disagree: estimates in the nudge letters ran from 0.01 to 0.31 depending on method and assumed severity. When you write a review, report the uncorrected and corrected estimates side by side and say which you rely on and why.^[9][8]

For practitioners the stakes are budgets. Adler and colleagues note that managers may invest in interventions drawn from research streams with questionable evidence. When a trial run at scale with every result recorded exists, as in the nudge-unit data, it is free of the selective publication that inflates a published average.^[6][3]

The evidence behind this

• Research methods pitfalls: ratio variables and censored data
• Marketing knowledge: how it is made and when to trust it
• Choice architecture and decision biases: what the research shows
• Advertising incrementality: how to know whether ads worked

References

1. Brian A. Nosek, Charles R. Ebersole, Alexander C. DeHaven, David T. Mellor (2018). The Preregistration Revolution. Proceedings of the National Academy of Sciences.
2. Lachlan Deer, Susanne Jana Adler, Hannes Datta, Natalie Mizik, Marko Sarstedt (2024). Toward Open Science in Marketing Research.
3. Stefano DellaVigna, Elizabeth Linos (2022). RCTs to Scale: Comprehensive Evidence From Two Nudge Units. Econometrica.
4. Joseph P. Simmons, Leif D. Nelson, Uri Simonsohn (2011). False-Positive Psychology. Psychological Science.
5. Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science.
6. Susanne Jana Adler, Lukas Röseler, Martina Katharina Schöniger (2023). A toolbox to evaluate the trustworthiness of published findings. Journal of Business Research.
7. Stephanie Mertens, Mario Herberz, Ulf J. J. Hahnel, Tobias Brosch (2022). The effectiveness of nudging: A meta-analysis of choice architecture interventions across behavioral domains. PNAS.
8. Maximilian Maier, Frantis̆ek Bartos̆, T. D. Stanley, David R. Shanks, Adam J. L. Harris, Eric‐Jan Wagenmakers (2022). No evidence for nudging after adjusting for publication bias. Proceedings of the National Academy of Sciences.
9. Barnabás Szászi, Anthony Higney, Aaron Charlton, Andrew Gelman, Ignazio Ziano, Balázs Aczél, Daniel G. Goldstein, David S. Yeager, Elizabeth Tipton (2022). No reason to expect large and consistent effects of nudge interventions. Proceedings of the National Academy of Sciences.
10. Jonathan Z. Bakdash, Laura R. Marusich (2022). Left-truncated effects and overestimated meta-analytic means. Proceedings of the National Academy of Sciences.
11. Raj Sethuraman, Gerard J. Tellis, Richard A. Briesch (2011). How Well Does Advertising Work? Generalizations from Meta-Analysis of Brand Advertising Elasticities. Journal of Marketing Research.
12. Joseph Korkames, T. D. Stanley, Stefan Stremersch (2025). Meta-analysis of advertising effectiveness: New insights from improved bias corrections. International Journal of Research in Marketing.

Bibliographic details only. Follow the DOI to read each study at its publisher.