Synthetic data is everywhere, but is it any good?

The market research sector has a problem: You don’t pick up your damn phone anymore. Some eight in 10 of us don’t answer when an unknown number calls, according to the Pew Research Center, a shift that has had a knock-on effect on pollsters’ ability to get us to share our thoughts. Online surveys, too, can be easily gamed, and because they require people to opt in by physically visiting a website, they can be even easier to ignore than phone surveys. That’s where AI can help. Across the polling and consumer research industries, firms are using artificial intelligence to manufacture synthetic survey responses, creating plausible answers from fake people to stand in for, or pad out, real ones. Qualtrics, the experience-management giant, now offers synthetic panels that take a survey as an input and produce record-level responses designed to be statistically modeled the same way as responses from 1,000 humans, according to Ali Henriques, the company’s executive director of market research. The system leans heavily on Qualtrics’ own data: A publicly available base model contributes between 5 and 10 of the final result, with the remaining 95-plus drawn from the firm’s commissioned research and aggregated, anonymized client data, stripped of brands and no more than 18 months to two years old to keep it relevant. It’s not just Qualtrics. In May, Gallup, the 90-year-old pollster, disclosed a partnership with Simile, an AI company founded by Stanford researchers, to build “agents” from in-depth interviews with around 1,000 members of its probability-based panel. But Gallup, which didn’t respond to an interview request, has been careful to say simulated responses won’t be used to produce its published population estimates, and has pledged never to present them as human answers. “Our work on simulated responses is not a departure from that commitment,” the company said in its blog post announcing the partnership. “It is built on top of it.” Such caution is needed, says Jason Miklian, a research professor at the Center for Global Sustainability at Norway’s University of Oslo, who is studying the synthetic research space. “While synthetic data can give you an incredible snapshot of conventional wisdoms of what sorts of things people have generally believed over time,” he says, “it’s incredibly bad at generating anything surprising.” The surprises, he points out, are the valuable bits: the new knowledge that drives scholarship or business decisions. Miklian sees synthetic data as useful for pressure-testing a survey before spending money administering it to real people, or for questions whose answers would have looked the same five or 10 years ago. But some worry about mission creep. Sean Westwood, a political scientist at Dartmouth College and director of its Polarization Research Lab, worries firms selling silicon sampling will rarely disclose the model or the success metrics against which they ought to be benchmarked. “’We use GPT-5’ is just not a method,” he says. “Silicon sampling launders bias as data,” Westwood says, arguing that stereotypes subsumed into training data can quickly become consensus opinions when scaled up. Some companies are using AI to scale up their systems: French pollster Ifop offers a product called DataBoost AI, which it says can “transform small sub-samples into robust bases using statistical levers”. In one recent example criticized by French statisticians on Bluesky, Ifop used the tech to turn a sample of 116 real interviews with middle- and high-school teachers into a group of 580 teachers. Ifop did not respond to an interview request. Westwood argues that because AI models work in a non-deterministic way, introducing random errors with each run, researchers can’t use traditional statistical techniques to calculate uncertainty in a real sample. Increasing sample sizes, he argues, sacrifices the ability to understand what is actually being measured. The University of Oslo’s Miklian fears a “creep” of synthetic responses into what was once human-driven political polling, and potentially a feedback loop in which synthetic surveys amplify existing assumptions, then become ammunition for anyone wanting to challenge real election results that fail to match them. Qualtrics, for its part, is eager to try to ensure that doesn’t happen in its areas of research. “We’re making a concerted effort to educate the market that this is not a replacement,” says Henriques, the company’s market research director. She’s spent the last year and a half thinking about synthetic respondents, and sees a line between modeling behavior and reproducing life. “All of these pieces start to come together in a really interesting way that’s understanding just the human,” she says. “But I don’t believe we’ll be able to fully simulate those really lived experiences.”