Evidence · Research methods

Synthetic respondents: can LLMs stand in for survey participants?

Large language models prompted with demographic personas can reproduce some group-level survey distributions, but accuracy varies considerably by topic, subgroup and domain, so the best-supported use is pretesting and pilot work rather than replacing human samples. A second problem is growing: AI agents can complete online surveys, pass almost all attention checks and give coherent answers, so standard quality screens no longer show that a panel is human. Much of this evidence is new and not yet peer reviewed.

Definition

Synthetic respondents are survey or experiment answers generated by a large language model instead of a person. In silicon sampling, a researcher deliberately prompts the model to answer as someone with given demographic traits and treats the output as a stand-in for a human sample. A related but separate risk is contamination: real online panels that contain AI-generated or AI-assisted answers without the researcher knowing.

What the research finds

Group opinions can be mimicked from demographics alone

Prompting a language model with only the demographic make-up of a group produced response distributions remarkably close to US public opinion polls on facial recognition technology. How well it worked varied by demographic group and question, which the author links to social biases built into the model.^[1]

Agreement with human samples varies by domain

A review for consumer and marketing research found that studies comparing silicon and human samples give results that vary considerably across domains. It recommends silicon samples mainly for early stages such as qualitative pretesting and pilot studies, sets out stricter conditions for using them in main studies and flags ethical issues.^[2]

Inference from language models has reproducibility problems

A review of how researchers use language models to study cognition, including as simulated participants, points to closed versus open models, the invisibility of training data and the need for conventions on prompts and instructions, which act like new settings that change results.^[3]

Benchmarks with real individuals are emerging

A public dataset records answers from 2,058 US participants to 500 questions across four waves, about 2.42 hours per person, with a repeat wave that sets a test-retest baseline. It is designed as ground truth for testing when simulated personas match the real people they are built from.^[4]

AI agents pass the checks built to catch bad respondents

An autonomous AI agent passed 99.8% of 6,000 attention checks while producing internally consistent, hypothesis-confirming answers, and roughly a third of survey participants report using AI assistance. Because the resulting bias looks like signal rather than noise, screens designed to catch careless humans do not catch careful-looking AI.^[5]

Clinical screening instruments are also exposed

A simulation study generated 2,106 synthetic personas from 13 diagnostic profiles at three severity levels with one commercial model and ran them through seven validated mental health screening instruments. It reports that such personas can get through screeners that serve as gatekeepers for study eligibility.^[6]

Defences lie in design, not detection

The proposed response is procedural: statistical methods that allow for contamination, sampling stratified by the role the evidence will play, treating online convenience samples as exploratory and demanding robust recruitment for confirmatory tests.^[5]

How strong is the evidence?

Only one study reviewed here is a peer-reviewed marketing journal article, a review with usage guidelines. The rest are preprints, a conference paper or commentary, and the contamination figures come from a single preprint commentary. Tests of silicon sampling so far cover few topics and few models, and models change quickly, so accuracy results may not carry over to newer versions. No study here checks whether synthetic respondents reproduce known regularities in buying behaviour rather than echoing what the model already encodes, and how widespread AI answers are in commercial marketing panels has not been measured.

References

Seungjong Sun, Eungu Lee, Dongyan Nan, Xiangying Zhao, Wonbyung Lee, Bernard J. Jansen, Jang Hyun Kim (2024). Random Silicon Sampling: Simulating Human Sub-Population Opinion Using a Large Language Model Based on Group-Level Demographic Information. arXiv preprint.
Marko Sarstedt, Susanne Adler, Lea Rau, Bernd H. Schmitt (2024). Using large language models to generate silicon samples in consumer and marketing research: Challenges, opportunities, and guidelines. Psychology and Marketing.
Desmond C. Ong (2024). GPT-ology, Computational Models, Silicon Sampling: How should we think about LLMs in Cognitive Science?. arXiv.
Olivier Toubia, George Z. Gui, Tianyi Peng, Dennis J. Zhang (2025). Twin-2K-500: A dataset for building digital twins of over 2,000 people based on their answers to over 500 questions. arXiv.
Zhicheng Lin (2025). Synthetic respondents and the illusion of human data.
Kianté Fernandez, Laura Berner, Blair R K Shevlin (2026). The threat of synthetic respondents extends to clinical mental health screening.

Bibliographic details only. Follow the DOI to read each study at its publisher.

Building on this evidence?

ScholarSignal checks a new research idea against the closest existing work and an editor rubric, and tells you whether the contribution is new enough for the journals you target.

Evaluate my idea for free