Guides · Methods and evidence

LLM synthetic respondents in marketing research: evidence and limits

Language models prompted with personas can reproduce some average survey answers, but they tend to understate variation, can change the conclusions of regression analyses, shift with prompt wording and model updates, and fail to reproduce several well-known behavioral effects. The strongest current case is for upstream work such as pretesting items and piloting, benchmarked against human data. A separate and growing risk is AI-completed responses inside human online samples, where most of the evidence so far comes from preprints.

Two different problems

Silicon sampling means prompting a language model to answer as a respondent with given characteristics and treating the output as data from that kind of person. A review for consumer and marketing researchers found that comparisons of silicon and human samples have produced mixed results that vary considerably by domain.^[1]

The second problem is involuntary: real online samples that contain answers produced or assisted by AI. The two call for different responses. Silicon sampling is a design choice that can be benchmarked and disclosed, while contamination enters data that researchers believe is human.^[2][3]

The evidence base is young and uneven. Several key sources below are preprints or conference papers rather than peer-reviewed journal articles, and they are identified as such where they appear.^[4][5][2]

Where silicon samples match human data

At the level of averages, the match can be close. When ChatGPT was given personas built from real respondents in the 2016 and 2020 American National Election Studies and asked for feeling thermometer ratings of 11 social and political groups, its average scores corresponded closely to those of the human respondents.^[6]

A conference paper on random silicon sampling reported that a model conditioned only on the demographic make-up of a group produced opinion distributions on facial recognition technology that closely resembled U.S. polls. The marketing review lists further successes, including personality markers, framing effects and political attitudes.^[7][1]

Individual-level prediction has also been tested. A preprint dataset of 2,058 U.S. participants who answered about 500 questions over four waves was used to build digital twins from their earlier answers. On held-out tasks the twins were right 71.7% of the time, against 59.2% for random guessing and 81.7% for the same people answering again, and the twin data reproduced 6 of the 10 classic experimental results that the human data had reproduced.^[5]

Where they fail

The same election-study comparison showed the limits behind a good average. Synthetic answers varied less than real ones, and when thermometer scores were regressed on respondent characteristics, 48% of the coefficients from the ChatGPT data differed significantly from those estimated on the human data. Small changes in prompt wording shifted the distribution of answers, and the same prompt gave significantly different results over a three-month period.^[6]

Behavioral effects are a particular weak point. The marketing review reports failures to reproduce the endowment effect, mental accounting and the sunk cost fallacy, and a prospect theory replication in which ChatGPT mostly maximized expected payoff instead of showing the human pattern of risk aversion for gains and risk seeking for losses. In a rerun of 14 Many Labs 2 studies, the model reproduced just over a third, and in 6 studies it gave nearly identical answers every time, leaving no variation to analyze.^[1]

Performance also depends on who is being simulated. The review cites evidence that models mimic answers from non-Western countries considerably less well, and the random silicon sampling study found that accuracy varied by demographic group and question topic, which its authors linked to social biases in the model.^[1][7]

Why results are hard to reproduce

A preprint review of language model research in cognitive science names the recurring obstacles: closed models whose training data cannot be inspected, and the absence of conventions for reporting prompts and instructions, which act as new settings that change results. Model updates add a further moving part, as the three-month drift in the election-study replication shows.^[4][6]

For this reason the marketing review recommends keeping a full record of model settings, inputs and outputs, preferably in a public repository, and probing whether results survive changes in wording or question order.^[1]

Where synthetic data can help

The review sees the clearest value upstream, in qualitative pretesting and pilot studies whose output researchers can check for themselves. Examples include generating candidate items for a new scale, pretesting survey wording and developing stimuli or vignettes before a human study.^[1]

For main studies the bar is higher. The review advises using silicon samples only for topics well covered in training data, avoiding narrow target groups and very recent events, benchmarking every silicon sample against a relevant human sample on both averages and variation, and discarding samples that return a single correct answer.^[1]

AI-completed responses in human samples

A preprint commentary argues that contamination is already widespread. It cites a demonstration in which an autonomous AI agent passed 99.8% of 6,000 attention checks while giving internally consistent answers, and reports that about a third of survey participants say they use AI assistance. Its central point is that such answers look like careful responding, so screens built to catch inattentive humans do not catch them.^[2]

A second preprint tested validated clinical screening questionnaires. Given only brief diagnostic personas, a commercial model produced 2,106 synthetic respondents whose scores on seven instruments were largely congruent with their assigned diagnoses, rose with assigned severity and exceeded clinical cutoffs, without any instrument content or scoring rules in the prompt. Scores above a screening threshold therefore cannot be taken as proof of a genuine participant.^[3]

Practical rules for researchers

These rules follow from the studies above; because parts of the evidence are preprints, treat them as current best judgment rather than settled standards.^[1][2]

• Do not report silicon samples as evidence about human behavior without a human benchmark on the same measures.^[1][6]
• Compare variances and relationships between variables, not just means; a matching average can coexist with different regression conclusions.^[6]
• Record the model version, settings, full prompts and outputs, and test several wordings.^[1][4]
• Be most cautious with behavioral biases, narrow populations and non-Western samples.^[1]
• For online human samples, treat convenience panels as exploratory and use stronger recruitment for confirmatory tests, as the contamination commentary proposes.^[2]
• Do not rely on attention checks or screening thresholds alone to establish that a respondent is human.^[2][3]

The evidence behind this

• Synthetic respondents: can LLMs stand in for survey participants?
• Research methods pitfalls: ratio variables and censored data
• Marketing analytics and AI: what peer-reviewed research shows

References

1. Marko Sarstedt, Susanne Adler, Lea Rau, Bernd H. Schmitt (2024). Using large language models to generate silicon samples in consumer and marketing research: Challenges, opportunities, and guidelines. Psychology and Marketing.
2. Zhicheng Lin (2025). Synthetic respondents and the illusion of human data.
3. Kianté Fernandez, Laura Berner, Blair R K Shevlin (2026). The threat of synthetic respondents extends to clinical mental health screening.
4. Desmond C. Ong (2024). GPT-ology, Computational Models, Silicon Sampling: How should we think about LLMs in Cognitive Science?. arXiv.
5. Olivier Toubia, George Z. Gui, Tianyi Peng, Dennis J. Zhang (2025). Twin-2K-500: A dataset for building digital twins of over 2,000 people based on their answers to over 500 questions. arXiv.
6. James Bisbee, Joshua D. Clinton, Cassy Dorff, Brenton Kenkel, Jennifer Larson (2024). Synthetic Replacements for Human Survey Data? The Perils of Large Language Models. Political Analysis.
7. Seungjong Sun, Eungu Lee, Dongyan Nan, Xiangying Zhao, Wonbyung Lee, Bernard J. Jansen, Jang Hyun Kim (2024). Random Silicon Sampling: Simulating Human Sub-Population Opinion Using a Large Language Model Based on Group-Level Demographic Information. arXiv preprint.

Bibliographic details only. Follow the DOI to read each study at its publisher.