Synthetic Users vs Generic LLMs

To get a good product sense, you need to have a good understanding of your potential customers. Creating custom synthetic users (SU) to test your work will complement what you gather from actual humans. But be wary of applying ‘Generic LLMs’ out of the box! Not surprisingly SUs outperform generic models in many regards. Here is what your internal insights agent can deliver in contrast to ChatGPT, Claude or Gemini:

→ More realistic user codes that match your past human research.
→ Consistent behaviour across different testing scenarios.
→ Fewer implausible assumptions leading to deeper, segment-specific insights (like price anchors and value seekers) that generic models missed.

The value of custom, proprietary data, whether for an internal SU or with seeding data in services like Synthetic Users.

Watch out for critical limitations:
both solutions are not a substitute but complementary to human research. Despite their speed and scale, synthetic users cannot truly provide:
→ Emotional context and personal lived experiences.
→ Cultural trends and genuine human preferences.
→ Brand Trust/Perception for new websites and offering.
→ Price value sensitivity, a core human factor, can also be an ambivalent signal.

Synthetic users are indeed here to stay in product. Assume that they will only become more nuanced and versatile - especially the more context data you can give them.

Lessons for your own product:
→ create a set of custom synthetic users
→ use synthetic users as a supplement, a broad hypothesis generator
→ Narrow down the solution space.
→ Deepen the context with real human studies.