Guides · Methods and evidence

Survey measurement pitfalls in marketing research

Survey measures mislead most often when predictor and outcome come from the same respondents at the same time, when a convenient proxy stands in for the construct, and when attitudes or intentions are read as if they were behavior. Editors cite common method variance as a reason for desk rejection, and published comparisons show that stated intentions, recommendation scores and copy-test scores track behavior only loosely. Build indices and screens with these weaknesses in mind.

Common method variance in single-source surveys

In a 2016 editorial, the editor-in-chief of the Journal of the Academy of Marketing Science listed, among the common reasons for desk rejection, models built mostly on main-effect hypotheses and tested with cross-sectional survey data, because common method variance often undermines most of those main-effect findings. The concern applies whenever both the independent and the dependent variables are self-reported by the same respondents.^[1][2]

A review of method biases in behavioral research traces common method variance to four sources: the same rater answering for predictor and outcome, a shared measurement context, a shared item context, and features of the items themselves. In the studies it summarizes, method variance averaged 15.8 percent of the variance in marketing measures, the lowest of the fields compared.^[3]

The same review ranks procedural remedies before statistical ones: separate the measurement of predictor and criterion in time, place, psychologically or by method, protect respondents' anonymity, and reduce evaluation apprehension. It also warns that Harman's single-factor test, still widely reported, does nothing to statistically control for the bias.^[3]

Meta-analytic evidence points the same way. Across studies of market orientation and business performance, those that measured performance with subjective assessments reported stronger correlations than those that used objective performance data, about .28 against .22 after correction.^[4]

Remedies are mostly procedural. Researchers who acknowledge the problem point to multi-source data, objective outcome measures, separating predictor and outcome measurement in time, and statistical checks such as marker variables, rather than relying only on a single-factor test.^[2]

Measure the construct you mean

Measures should be developed, not simply declared. A classic procedure for marketing constructs starts by specifying the domain of the construct, generates and purifies a pool of items, and assesses reliability and validity before the measure is used to test hypotheses.^[5]

A convenient measure can quietly narrow a concept. Brand salience has usually been measured as top-of-mind recall with the product category as the only retrieval cue. Romaniuk and Sharp argue that this is too narrow: salience is a brand's propensity to be noticed or come to mind in buying situations, which depends on links to many cues, so a single category prompt misses much of it.^[6]

Broader measures can carry more information. A salience measure based on links between the brand and a set of attributes was related to the probability of switching suppliers in subscription markets, a relationship that held across three studies, two markets and two data collection methods.^[7]

Competing instruments sometimes turn out to be equally valid. A meta-analysis of 42 effect sizes from 17 studies found SERVQUAL and SERVPERF equally good predictors of overall service quality; adapting SERVQUAL to the study context improved it, while adapting SERVPERF did not. The authors suggest choosing by purpose, the longer scale for diagnosis and the shorter one for efficiency.^[8]

Attitude and image responses follow usage

In brand image surveys, whether someone links a brand to an attribute depends largely on whether they use the brand. Users give more image responses than former users or non-users, so bigger brands collect more responses on almost every attribute. Without a benchmark for the number of users, a rise in image scores can reflect nothing more than a larger user base or a different sample.^[9]

The form of a question can decide whether a relationship appears at all. Across 188 brands in 11 industries, absolute satisfaction ratings explained customers' share of wallet very weakly, whereas satisfaction measured relative to competing brands was a strong predictor.^[10]

Stated intentions, recommendation scores and copy tests

Intentions predict behavior only loosely. A review of customer metrics describes a three-wave study in which 36.0% of consumers who said they would definitely or probably buy a nondurable product actually tried it, against 12.6% of those who said they would not. The link also depends on the response scale, the time horizon and the lag between the question and the purchase.^[11]

Calibrated probability scales can do better. Estimates from the 11-point Juster purchase probability scale were compared with panel data for 16 brands in three categories and generally came out in the right order and of the right magnitude. They were not error-free: brand penetration was consistently underestimated in two categories and overestimated in the third, and the authors describe simple purchase recall as an unreliable benchmark.^[12]

Popular single-question metrics deserve the same caution. In five years of business-to-business data with 2,785 responses matched to revenue, Net Promoter promoters did spend more in the current year, but they were a minor and inconsistent source of growth among existing customers, which mostly came from a general rise across the whole base. In 389 split-cable television tests, standard recall and persuasion copy-test scores showed no strong relationship with the sales effect of the advertising.^[13][14]

Constructed variables that distort analysis

Many survey and accounting indices are built after data collection, and some constructions create problems of their own.^[15][16]

• Ratio indices. Two ratios that share a component can be correlated simply because of that shared term, and using a ratio as the dependent or an independent variable in regression can produce misleading or even reversed inferences. The recommendation is to use a ratio only inside a full model that also includes the variables it is made from.^[15]
• Unfinished durations. When some respondents are still customers at the time of the survey, their tenure is incomplete. The product-limit estimator uses such incomplete observations to estimate how long relationships last, but it assumes that the reason observation stopped is unrelated to the duration itself, an assumption that needs checking.^[16]

Respondents who are not who they seem

Online samples face a newer threat. A preprint commentary reports that an autonomous AI agent passed 99.8% of 6,000 attention checks and argues that such answers look like careful human responding, so screens built to catch inattentive people miss them. A second preprint showed that a commercial language model could produce scores above clinical cutoffs on validated screening questionnaires, so passing a screener is no longer proof of an eligible human respondent. Both are preprints, but they suggest attention checks and timing filters should not be the only quality control.^[17][18]

A checklist for questionnaire design and review

Reviewers and authors can use these questions to judge whether a survey measure can support the claim built on it.^[1][6]

• Do the predictor and the outcome come from different sources or different times? If not, what other evidence rules out common method variance?^[1][4]
• If predictor and outcome come from the same respondents, did you separate them in time or format, and do you rely on more than Harman's single-factor test to argue the bias is small?^[3]
• Was each multi-item measure developed and checked for reliability and validity, or adopted without evidence that it captures the construct you mean?^[5]
• Does the measure cover the construct as defined, or only its most convenient part?^[6]
• Are brand attitude or image scores benchmarked against the number of users each brand has?^[9]
• Is satisfaction measured in absolute terms when the outcome depends on comparison with competitors?^[10]
• Are intentions or recommendation scores used as stand-ins for behavior without a validation against actual purchases?^[11][13]
• Does any regression use a ratio without also including its components?^[15]
• What besides attention checks shows that respondents are real people?^[17]

The evidence behind this

• Research methods pitfalls: ratio variables and censored data
• Customer satisfaction and loyalty metrics: NPS, CRM and the evidence
• Mental and physical availability: what the research shows
• Synthetic respondents: can LLMs stand in for survey participants?

References

1. Robert W. Palmatier (2016). Editorial: the past, present, and future of JAMS. Journal of the Academy of Marketing Science.
2. Oliver Roll, Jan Luca Geerties (2025). Unlocking value-based pricing: the moderating roles of pricing capabilities and contingency factors in B2B markets: an empirical approach. Journal of Revenue and Pricing Management.
3. Philip M. Podsakoff, SCOTT B. MACKENZIE, Jeong Yeon Lee, Nathan Philip Podsakoff (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies.. Journal of Applied Psychology.
4. Paul D. Ellis (2006). Market Orientation and Performance: A Meta‐Analysis and Cross‐National Comparisons*. Journal of Management Studies.
5. Gilbert A. Churchill Jr. (1979). A Paradigm for Developing Better Measures of Marketing Constructs. Journal of Marketing Research.
6. Jenni Romaniuk, Byron Sharp (2004). Conceptualizing and Measuring Brand Salience. Marketing Theory.
7. Jenni Romaniuk, Byron Sharp (2003). Brand Salience and Customer Defection in Subscription Markets. Journal of Marketing Management.
8. François A. Carrillat, Fernando Jaramillo, Jay Prakash Mulki (2007). The validity of the SERVQUAL and SERVPERF scales. International Journal of Service Industry Management.
9. Jenni Romaniuk, Byron Sharp (2000). Using Known Patterns in Image Data to Determine Brand Positioning. International Journal of Market Research.
10. Alexander Buoye, Yuliya Komarova Loureiro, Sertan Kabadayi, Mohammad G. Nejad, Timothy L. Keiningham, Lerzan Aksoy, Jason Allsopp (2016). Is share of wallet exclusively about making customers happy or having more customers? Exploring the relationship between satisfaction and double jeopardy. Journal of service management.
11. Sunil Gupta, Valarie A. Zeithaml (2006). Customer Metrics and Their Impact on Financial Performance. Marketing Science.
12. Malcolm Wright, Anne Sharp, Byron Sharp (2002). Market statistics for the Dirichlet model: Using the Juster scale to replace panel data. International Journal of Research in Marketing.
13. Philip Mecredy, Malcolm Wright, Pamela M. Feetham (2018). Are Promoters Valuable Customers? An Application of the Net Promoter Scale to Predict Future Customer Spend. Australasian Marketing Journal (AMJ).
14. Leonard M. Lodish, Magid Abraham, Stuart Kalmenson, Jeanne Livelsberger, Beth Lubetkin, Bruce Richardson, Mary Ellen Stevens (1995). How T.V. Advertising Works: A Meta-Analysis of 389 Real World Split Cable T.V. Advertising Experiments. Journal of Marketing Research.
15. Richard A. Kronmal (1993). Spurious Correlation and the Fallacy of the Ratio Standard Revisited. Journal of the Royal Statistical Society. Series A (Statistics in Society).
16. Edward L. Kaplan, Paul Meier (1958). Nonparametric Estimation from Incomplete Observations. Journal of the American Statistical Association.
17. Zhicheng Lin (2025). Synthetic respondents and the illusion of human data.
18. Kianté Fernandez, Laura Berner, Blair R K Shevlin (2026). The threat of synthetic respondents extends to clinical mental health screening.

Bibliographic details only. Follow the DOI to read each study at its publisher.