ScholarSignal
Menu
Sign inEvaluate an idea

Evidence · Advertising

Advertising incrementality: how to know whether ads worked

Incrementality is the share of sales that would not have happened without the ad, and randomised experiments are the only reliable way to measure it. Large field experiments at Facebook and eBay show that standard observational methods, even with thousands of user features, can badly overstate lift, and that branded search ads can have no measurable short-term benefit. Even experiments with millions of users are often too noisy to pin down a return on investment.

Definition

Advertising incrementality is the causal effect of an ad or campaign: the difference in an outcome such as sales between people who were exposed and otherwise comparable people who were not. It differs from attribution, which assigns credit for conversions that did happen, including those that would have happened anyway. The standard way to estimate it is a randomised experiment with a holdout group.

What the research finds

More ad weight alone rarely lifts sales

A meta-analysis of 389 split-cable TV experiments found that increasing advertising weight raised sales in only a minority of tests. Effects were larger for new products and for changes in copy or strategy, and where short-term effects appeared they tended to persist.[1]

Observational methods overstate lift

Comparing 15 randomised Facebook experiments with the estimates that common observational methods would produce from the same data showed that matching and regression often failed to recover the true effect. A follow-up across 663 experiments, using more than 5,000 user-level features, found median experimental lifts of 29%, 18% and 5% for upper, middle and lower funnel outcomes, while the better of two advanced non-experimental methods estimated 83%, 58% and 24%.[2][3]

Paid search often harvests demand that exists anyway

Field experiments at eBay show that returns from paid search are a fraction of non-experimental estimates, because search clicks and purchase intent go together. Brand keyword ads had no measurable short-term benefit. For other keywords, new and infrequent users responded, but frequent users who would have bought anyway absorbed most of the spend, making average returns negative.[4]

Precise measurement is expensive

In 25 large field experiments with US retailers and brokerages, the median confidence interval on return on investment was more than 100 percentage points wide. Individual sales are so volatile relative to ad cost that an informative experiment can need more than 10 million person-weeks, which puts proper measurement out of reach for many firms.[5]

Better experimental designs lower the cost

The ghost ads method identifies control-group users who would have seen the focal ad, so advertisers no longer need to pay for placebo charity ads to create a comparison group. Statistical power and sample-size formulas have also been worked out for platform lift studies, including multi-cell tests, although that work is a preprint.[6][7]

Targeting effects depend on the buyer's stage and social context

In a field experiment by an online travel firm, retargeted ads showing the exact products a person had viewed performed worse on average than generic ads, and only caught up once browsing showed the person's preferences had narrowed. In experiments with 5.3 million Facebook users, adding a friend's name to an ad raised click-through, more so for closer ties; this study is a conference paper.[8][9]

Privacy rules change measured effectiveness

Using 3.3 million survey takers randomly exposed to 9,596 display campaigns, researchers found that display ads became far less effective at changing stated purchase intent in the EU after privacy laws restricted tracking. The loss was largest on general-content sites and for small, plain ads without video, audio or interactive features.[10]

How strong is the evidence?

The core findings come from large randomised field experiments in leading journals, which is strong evidence for the settings studied. Most run on one platform (Facebook) or one marketplace (eBay), and platform-run experiments measure effects on that platform's own outcomes; totals across channels are still mostly modelled rather than measured. Experiments capture short-run lift well, while long-run brand effects remain hard to identify experimentally. The experiment-design work on lift studies is a preprint, the social-cue study is a conference paper, and the privacy study measures stated purchase intent rather than sales.

References

  1. Leonard M. Lodish, Magid Abraham, Stuart Kalmenson, Jeanne Livelsberger, Beth Lubetkin, Bruce Richardson, Mary Ellen Stevens (1995). How T.V. Advertising Works: A Meta-Analysis of 389 Real World Split Cable T.V. Advertising Experiments. Journal of Marketing Research.
  2. Brett R. Gordon, Florian Zettelmeyer, Neha Bhargava, Dan Chapsky (2019). A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook. Marketing Science.
  3. Brett R. Gordon, Robert Moakler, Florian Zettelmeyer (2023). Close Enough? A Large-Scale Exploration of Non-Experimental Approaches to Advertising Measurement. Marketing Science.
  4. Thomas Blake, Chris Nosko, Steven Tadelis (2015). Consumer Heterogeneity and Paid Search Effectiveness: A Large-Scale Field Experiment. Econometrica.
  5. Randall A. Lewis, Justin M. Rao (2015). The Unfavorable Economics of Measuring the Returns to Advertising*. The Quarterly Journal of Economics.
  6. Garrett A. Johnson, Randall A. Lewis, Elmar I. Nubbemeyer (2017). Ghost Ads: Improving the Economics of Measuring Online Ad Effectiveness. Journal of Marketing Research.
  7. C. H. Bryan Liu, Elaine M. Bettaney, Benjamin Paul Chamberlain (2018). Designing Experiments to Measure Incrementality on Facebook. arXiv.
  8. Anja Lambrecht, Catherine E. Tucker (2013). When Does Retargeting Work? Information Specificity in Online Advertising. Journal of Marketing Research.
  9. Eytan Bakshy, Dean Eckles, Rong Yan, Itamar Rosenn (2012). Social Influence in Social Advertising: Evidence from Field Experiments. Proceedings of the 13th ACM Conference on Electronic Commerce.
  10. Avi Goldfarb, Catherine E. Tucker (2010). Privacy Regulation and Online Advertising. Management Science.

Bibliographic details only. Follow the DOI to read each study at its publisher.

Related topics

A synthesis of 10 studies, written by ScholarSignal. Last reviewed 2026-09-26.

Building on this evidence?

ScholarSignal checks a new research idea against the closest existing work and an editor rubric, and tells you whether the contribution is new enough for the journals you target.

Evaluate my idea for free