First-hand information for everyone
PerSo is a benchmark that measures both cognitive and behavioral parts of social intelligence in large language models using real persuasion dialogues. It tests whether models can (1) pick which reply will persuade someone and (2) forecast the persuadee’s belief, emotion, and intent. (A “delta” is a Reddit signal awarded when a comment changes the original poster’s view.)
The dataset was built from Reddit’s ChangeMyView (CMV) threads. Each example pairs the OP’s initial post with two different persuader replies: one that later received a delta and one that did not. The successful reply was also required to have a higher upvote score. Both OP follow-up replies had to be at least 100 tokens for the cognitive task. This produced 872 qualifying threads, from which 200 were randomly sampled for the final evaluation.
PerSo’s cognitive task uses 4-option multiple-choice questions that forecast the persuadee’s upcoming belief, emotion, or intent. The choices were generated by an LLM-in-the-loop pipeline (Gemini 2.5 Pro) and checked by human annotators. The behavioral task gives models the OP post and the two candidate replies (the OP’s later replies are hidden). Models rate each reply on a four-point persuasiveness scale; the higher-rated reply is taken as the model’s choice. Accuracy is the reported metric for both tasks. Human participants (n=3) completed the Tromsø Social Intelligence Scale (TSIS) for contextual comparison; the small sample limits statistical reliability.
Concrete results. On the behavioral task, the best-performing model (Qwen-3.5-397B, without “thinking”) reached 76% accuracy. The best human participant reached 75%; the gap is 1 percentage point.
On the cognitive tasks, models struggled most with emotion forecasting. Emotion was the hardest of the three dimensions; several models scored near or below the 25% chance level on emotion. The top reported model on the cognitive dimensions (Sonnet-4.6) remained behind the best human by small margins: about 2 percentage points on belief, 6 on emotion, and 4.5 on intent. Overall, the best human participant stayed ahead across the three cognitive dimensions.
Scaling and reasoning mode. Performance often increased with model size within families such as Gemma and Qwen, but this pattern did not hold for all families or variants. Enabling a step-by-step “thinking” mode had mixed effects: it improved emotion inference for some Qwen sizes but tended to lower belief accuracy across Qwen variants. Effects on intent and behavioral accuracy were inconsistent.
Quality checks and validation. A 10% random subset was manually reviewed by three annotators. For cognitive answer selection (annotators saw the final OP reply), inter-annotator agreement was reported as substantial: average pairwise Cohen’s κ = 0.911 for belief, 0.623 for emotion, and 0.709 for intent. The MCQs and distractors were rated for quality as part of this validation.
Caveats. Human reference results come from only three participants, which limits the statistical reliability of model-vs-human comparisons. PerSo is built only from English-language CMV threads and focuses on single-turn exchanges; how results generalize to other communities, languages, cultures, or multi-turn dialogues is unknown.
In sum, PerSo offers a linked behavioral-and-cognitive evaluation on real persuasion data. It finds that a top LLM can match or slightly exceed the best observed human on selecting persuasive replies, while humans remain ahead at forecasting internal states—especially emotion.