Guides · Methods and evidence

Causal claims from observational data in marketing research

On its own, usually not. When observational estimates of advertising effects were checked against randomized experiments on the same campaigns, they missed the experimental lift by a factor of three in half of the Facebook studies, and richer data with thousands of user-level features did not close the gap. Causal wording is still common in cross-sectional research and readers take it at face value, so use a design with credible identifying variation where you can and associational wording where you cannot.

Causal wording is common where the design cannot carry it

A large-scale analysis of 194,631 social science articles with purely cross-sectional designs found that 46% of titles or abstracts made causal claims about their own results between 1980 and 2024. The share held near 20% until 2000 and passed 60% by 2024. Business articles had the highest rate of the five disciplines studied, at 84% in 2024, and journals with higher impact used causal wording more often than lower-impact ones.^[1]

The same study shows why this matters. In an experiment with 1,105 participants, readers of the original abstracts often concluded that the study had produced causal evidence. Rewriting the abstract in associational terms, or adding a short label naming the design, reduced that tendency but did not remove it: about half of readers still used causal language when summarizing the rewritten abstracts. Summaries written by language models went further, adding causal claims to findings that the authors had stated only as associations.^[1]

What randomized experiments reveal about observational estimates

Advertising is one of the few fields where observational methods have been scored against randomized experiments on the same data. Using 15 experiments at Facebook covering 500 million user-experiment observations, researchers rebuilt the estimates that matching and regression methods would have produced. Those estimates mostly overstated the experimental lift and sometimes understated it, and in half of the studies they missed the lift in purchase outcomes by a factor of three. No single method was reliably best, and the methods came closer for registrations and page views than for purchases.^[2]

A follow-up across 663 Facebook experiments gave two modern methods, double machine learning and stratified propensity score matching, access to more than 5,000 user-level features, far more than most advertisers hold. Median experimental lifts were 29%, 18% and 5% for upper, middle and lower funnel outcomes. The double machine learning estimates had medians of 83%, 58% and 24%, and propensity matching did worse. The authors concluded that even this rich data could not reliably recover a campaign's causal effect.^[3]

Paid search shows the size of the gap in money terms. In eBay's data, a simple regression of sales on search spending implied a return above 4,100%, and adding day and region controls still left more than 1,600%. The experiment put the average return at minus 63%. When eBay stopped brand keyword ads, almost all the lost paid clicks came back through organic search results.^[4]

Why selection defeats the usual fixes

The core problem is that exposure is not random. Search ads are clicked by people who already intend to buy, so clicks and purchases move together whether or not the ad changed anything. Brand keyword ads are an extreme case, because they reach users who were already navigating to the site.^[4]

Targeting creates the same problem in display and social advertising. Across 25 retail and brokerage experiments, the targeted nature of advertising produced selection bias severe enough to cripple the observational methods in common use. In the Facebook studies, a measure of user activity mainly explained who was exposed rather than who converted, and for several campaigns an observational model would have needed extra covariates stronger than all the available data combined to remove the bias.^[5][2]

Outside advertising the logic is the same. A cross-sectional design cannot establish which variable came first, and without randomization it remains open to confounding and reverse causality. Method choices also move the answer: a meta-analysis of 1,532 word-of-mouth effects found that studies which did not control for promotions overstated the effect of online reviews on sales, and that the type of endogeneity correction used changed the estimate.^[1][6]

Experiments are better, but not free

Randomized designs remove selection bias but bring their own limits. In 25 large field experiments representing $2.8 million in digital ad spending, the median confidence interval on return on investment was more than 100 percentage points wide, because individual sales are very noisy relative to the cost of the ads. An informative test can need more than 10 million person-weeks, which puts it out of reach for many firms.^[5]

Design choices matter too. Control groups that see charity ads are costly and become invalid when the platform optimizes delivery separately for the charity ad and the focal ad, because the two groups then contain different kinds of people. The ghost ads method instead logs when the focal ad would have been shown to a control-group user, which restores a valid comparison. Randomized ad testing is also not new: a 1995 meta-analysis pooled 389 split-cable television tests in which matched household panels differed only in the advertising they received.^[7][8]

When observational data can support a causal conclusion

Observational data is not automatically ruled out. Quasi-experimental designs such as instrumental variables or regression discontinuity can support causal conclusions from observational data, provided the authors state strong assumptions openly and argue a credible identification strategy. What cannot carry a causal claim is variation across units at a single point in time with nothing that isolates the cause.^[1]

Instrumental variables are not a default fix. A critical review of their use in marketing argues that the instruments in the empirical literature are rarely valid and that the estimators can carry large finite-sample bias and sampling error. Before using them, researchers should show that endogeneity is a first-order problem and that the instruments are strong and valid; otherwise it is better to measure the omitted variable directly, and only randomized variation is a valid instrument without further assumptions.^[9]

Advocates of empirics-first research in marketing make a similar case. Associations are useful for describing a new phenomenon and for motivating causal work, but where possible researchers should deal with endogeneity and selection and use experimental variation from lab, field or natural experiments. With archival data, quasi-experimental variation is the thing to look for. For some strategic questions causal clarity will not be available, and then it should not be claimed.^[10]

Descriptive findings can stand as descriptive

A well-documented association is a contribution in its own right. The empirics-first framework treats robust patterns as knowledge that can later inform theory, and asks researchers to test them for resilience with other measures, other estimation methods, extra controls, other endogeneity corrections and subsamples. Failed resilience tests are useful because they show where a pattern stops holding.^[10]

Journal policy can change how findings are written. A journal that explicitly discourages causal claims showed far lower causal wording than other outlets, though causal language still appeared in 12.9% of its articles. The researchers who reported this point out that the comparison is itself correlational, since careful authors may simply choose that journal.^[1]

A wording and design checklist

These checks follow from the studies above and apply to archival, platform and survey work alike.^[1][2]

• Match verbs to the design. Words such as impact, effect, increases or influences describe causation; with cross-sectional data, report associations and predictions instead.^[1]
• Check the whole paper, not just the abstract. Articles with carefully worded abstracts often made unhedged causal claims in the introduction and discussion, far more than in the methods section.^[1]
• Name the design in the abstract. A short methodological label measurably reduced readers' tendency to read causal evidence into correlational results.^[1]
• Say how units came to be treated. For advertising data, state how the platform selected who saw the ad and why your controls can or cannot account for it.^[3][5]
• Treat very large observational returns as a warning sign. The paid search estimates above fell from over 4,100% to below zero once the variation was experimental.^[4]
• Where a test is feasible, randomize and report the full confidence interval, even when it is wide.^[5][7]
• If you use a language model to summarize papers, ask for caution explicitly. A careful prompt reduced causal overstatement in most of the five models tested, while simplifying or practical prompts increased it in all of them.^[1]
• If you use instrumental variables, argue why endogeneity matters for your estimate and show that the instruments are strong and plausibly valid, rather than relying on lagged marketing variables as instruments.^[9]

The evidence behind this

• Advertising incrementality: how to know whether ads worked
• Research methods pitfalls: ratio variables and censored data
• Marketing knowledge: how it is made and when to trust it

References

1. Calvin Isch, Timothy G. Dorr, Neil Fasching, Grace Jennings, Duncan J. Watts (2026). Quantifying the prevalence and impact of overreaching causal claims in social science. Nature Human Behaviour.
2. Brett R. Gordon, Florian Zettelmeyer, Neha Bhargava, Dan Chapsky (2019). A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook. Marketing Science.
3. Brett R. Gordon, Robert Moakler, Florian Zettelmeyer (2023). Close Enough? A Large-Scale Exploration of Non-Experimental Approaches to Advertising Measurement. Marketing Science.
4. Thomas Blake, Chris Nosko, Steven Tadelis (2015). Consumer Heterogeneity and Paid Search Effectiveness: A Large-Scale Field Experiment. Econometrica.
5. Randall A. Lewis, Justin M. Rao (2015). The Unfavorable Economics of Measuring the Returns to Advertising*. The Quarterly Journal of Economics.
6. Ana Babić Rosario, Francesca Sotgiu, Kristine De Valck, Tammo H.A. Bijmolt (2015). The Effect of Electronic Word of Mouth on Sales: A Meta-Analytic Review of Platform, Product, and Metric Factors. Journal of Marketing Research.
7. Garrett A. Johnson, Randall A. Lewis, Elmar I. Nubbemeyer (2017). Ghost Ads: Improving the Economics of Measuring Online Ad Effectiveness. Journal of Marketing Research.
8. Leonard M. Lodish, Magid Abraham, Stuart Kalmenson, Jeanne Livelsberger, Beth Lubetkin, Bruce Richardson, Mary Ellen Stevens (1995). How T.V. Advertising Works: A Meta-Analysis of 389 Real World Split Cable T.V. Advertising Experiments. Journal of Marketing Research.
9. Peter E. Rossi (2014). Even the Rich Can Make Themselves Poor: A Critical Examination of IV Methods in Marketing Applications. Marketing Science.
10. Peter N. Golder, Marnik G. Dekimpe, Jake T. An, Harald J. van Heerde, Darren S. U. Kim, Joseph W. Alba (2022). Learning from Data: An Empirics-First Approach to Relevant Knowledge Generation. Journal of Marketing.

Bibliographic details only. Follow the DOI to read each study at its publisher.