How accurate are synthetic audiences?

The source material decides

Researchers at Stanford built AI profiles of 1,052 real people from two-hour interviews. Profiles given both the interview and survey answers matched the people’s answers 86 per cent as well as the people themselves did two weeks later. With the interview alone it was 83 per cent. Profiles given only a demographic description reached 74 per cent.

The conclusion is simple: a profile that only knows age, gender and place of residence is guessing. A profile with rich source material about the group is reasoning.

Where the limits are

The Nürnberg Institut für Marktentscheidungen compared 500 real American consumers with 500 AI-generated respondents from a general language model, with no source material of its own. The model was right about awareness of well-known brands, but overestimated how many considered them and underestimated purchases of lesser-known ones. The answers were also more positive and less varied than the humans’.

That is exactly the difference between asking a general AI and asking a profile built on data about the group. It is also why synthetic audiences should not be used to calculate shares.

What you can rely on, and not

• Rely on the direction: which version is strongest, which objections come and where the groups come apart.
• Do not rely on percentages. They belong in a survey with real respondents.
• Always ask for the source material. An answer that cannot be followed back to data about the group is a guess.

How Svava works with it

Every profile in Svava is built on hundreds of data points or source-backed observations about the group. Under every answer there is a report showing which source material was linked to it, and the profile sheet states how many observations and sources the profile rests on.