Synthetic Users vs Human Respondents: What NN/g’s 604-Person Study Found

Can you swap real research participants for synthetic users vs human respondents and still trust the answer? NN/g ran that experiment and the result is more interesting than a simple yes or no: the synthetic users reached the same conclusion as 604 humans, but their answers were far less varied. That gap between “right answer” and “right texture” is where most AI research plans quietly go wrong.

NN/g published the comparison on 16 October 2026, reusing a question from its earlier UX terminology survey and posing it to 120 synthetic respondents generated through the Synthetic Users platform in February 2026.

What they actually compared

The question was open-ended: name the professional field that best describes someone who gathers insights about people’s needs and turns them into recommendations for how a product should be built and improved. 604 tech and design professionals answered it in the human survey. 120 synthetic respondents answered the same prompt later.

Synthetic Users assigned each AI respondent a profile — job title, age, location, organization type, years of experience, cultural background, and Big Five personality traits — based on a research plan NN/g supplied with inclusion and exclusion criteria. Both datasets were coded for keywords rather than forced into single categories, because some respondents named more than one field.

Where the two datasets agreed

UX terminology dominated in both. 84% of synthetic respondents used UX-related terms, compared with about 70% of human respondents. NN/g calls the 14-percentage-point gap statistically significant, but notes it does not change the practical conclusion of the original study: no alternative term has displaced UX as the dominant name for this work.

Job role mattered in both datasets too — but “UX” stayed the most prevalent term regardless.

Where they diverged: mode collapse

The synthetic responses contained far fewer distinct answers than the human ones. NN/g labels this mode collapse: a narrower range of terms from the AI group, even though the headline percentage pointed the same direction.

Two structural differences complicate the comparison. First, Synthetic Users simulates interviews, so AI respondents answered as if speaking to an interviewer and often gave longer responses than the humans’ survey answers. Second, the platform sometimes reworded the question — asking about “people’s needs, behaviors, and experiences” instead of the original phrasing. NN/g notes that even small wording changes can prime respondents, and that researchers have limited control over how questions are presented on these platforms.

Sample composition also differed by design: 67% of the human dataset consisted of UX professionals, versus only 20% of the synthetic dataset, because NN/g set role quotas for the AI sample to broaden representation. The human recruitment through NN/g’s own Practitioner Panel likely inflated familiarity with UX terminology in that group.

What this means for your next study

NN/g’s own framing is blunt: synthetic-user research is still not user research. For a product designer deciding whether to skip recruitment this sprint, the study suggests a narrower use.

Use synthetic users when you need a fast sanity check on framing — a first pass at how a question might land, or a rough map of terminology before you commit recruitment budget. The conclusion matched here, so a directional read is defensible.

Do not use them when variation is the point. If you are looking for edge cases, unexpected phrasing, or the long tail of how real people describe a problem, mode collapse is exactly the failure mode. The synthetic answers were narrower, and narrowing is the opposite of what exploratory research is for.

Watch the question wording. If your platform rephrases prompts, you no longer control one of the basic variables of your study. Ask how the tool presents questions before you trust its output.

How this sits against running a real card sort

The same logic applies to other generative shortcuts. Interaction-design.org’s card sorting lesson makes a parallel argument about AI-generated information architecture: AI can draft a plausible category structure in seconds, but it reproduces common conventions and your prompt’s assumptions rather than revealing how your users actually think. The suggested workflow there — let AI draft, then test with real participants — matches what NN/g’s numbers imply for surveys.

The honest read: NN/g’s study is a point in favor of synthetic users for conclusion-level questions, and a warning about them for everything else. If your decision only needs the headline number, the AI answer may be good enough. If your decision depends on hearing how differently people phrase the same idea, recruit humans.

Sources

• NN/g — Synthetic Users vs. Human Respondents: Same Conclusion, Less Variation
• Interaction Design Foundation — What Is Card Sorting?