First-hand information for everyone
Main finding: Large language models (LLMs) used to emulate voters can match individual reported votes about 85% of the time while producing group-level errors that reach 10 percentage points or more (difference between emulated and reported vote shares). (F1, F2)
"Individual accuracy" is the share of respondents for whom the emulated vote equals the reported vote. "Misrepresentation" is the gap between a group’s emulated and reported vote shares. (F8)
How the study was done: The authors emulated electorates from the American National Election Studies (ANES) for 2012, 2016, and 2020, including every respondent who reported a two-party presidential vote and had complete profile data (n = 3,793; 1,978; 5,141). Profiles listed race/ethnicity, gender, age, education, ideology, and political attention. (F5)
Which models and controls were tested: They compared a closed-source API model (Claude Sonnet 4.6) with an open-weight model (Llama-3.1-8B-Instruct). They evaluated multiple levels of control: base prompts, added prompt instructions, a decomposed reasoning setup, and—for Llama—activation-level control using contrastive activation addition (CAA). They varied the number of runs per voter and sampled with temperature τ = 0.7. (F6, F7)
Concrete results: On ANES 2020 both models achieved high individual accuracy (Claude 0.848; Llama 0.858). Yet both models overestimated the Democratic share for voters who self-identify as Native American by about 13.5 points (13.5 and 13.6 points, respectively), with 95% intervals excluding zero. Population-level errors remained small while some group errors were large. (F2, F1)
Systematic patterns: Misrepresentation was largest and most consistent for gender and education. Claude’s emulated gender gap was about three times the reported gap across elections. Llama’s emulated gender gap was about 2.3–3 times the reported gap. (F13)
Effects of removing profile attributes: Removing gender from the profile reduced the largest gender-group error by 22–42%. Removing education reduced the largest education-group error by up to 64%. By contrast, removing self-reported race increased the mean group error by over 40%. These changes occurred without lowering individual accuracy. (F3)
Tests of mitigation and control: Common alignment strategies did not reliably reduce misrepresentation. Repeated queries, aggregation, fairness instructions, and activation-level control failed to eliminate group errors. In some cases a fairness instruction substantially increased the worst-group error (for example, doubling the worst racial error in some settings). Activation-level control (CAA) on Llama shifted many voters toward the Democratic candidate, raising the population-level error from 1.4 to 6.6 points. It also worsened worst-group errors for gender and education. Individual accuracy remained essentially unchanged. (F4, F10, F11)
Aggregation note: Averaging predicted probabilities (probability aggregation) has a formal guarantee that it cannot be worse than the worst single predictor, but in practice it produced only small changes (at most about 1.5 points) and did not systematically reduce worst-group errors because model mistakes were often repeated across runs. (F9)
Limitations and uncertainty: Some large group errors rest on small sample sizes and are therefore noisy. The largest racial error (Native American) is based on the smallest racial group in the dataset (n = 89) and is the least precise estimate. Racial-group errors also varied across elections and across the two models. (F12, F14)
Significance and caveats: The results show that high overall fidelity to reported votes can coexist with substantial subgroup misrepresentation. This pattern was robust across datasets, two models, and multiple control strategies. At the same time, small group sizes and variability across elections and models limit the precision and generality of specific subgroup estimates. (F1, F13, F12)