First-hand information for everyone
Researchers introduce MP-Bench, the first benchmark made to test voice agents as active participants in multiparty conversations. The benchmark focuses on two human skills that matter in group talk: knowing when to take or yield a turn (turn-taking awareness) and producing responses that fit the flow of the conversation (response appropriateness). It also includes comprehension-style question-and-answer checks to see whether agents correctly understand who said what.
To build MP-Bench, the team created scenarios with multiple speakers and designed objective tests that ask the agent whether it should speak next, what a good reply would be, and factual questions about the discussion. They ran the benchmark on 12 voice agents, including systems that operate in real time. The benchmark deliberately used distinct speaker voices so that the test would focus on conversational ability rather than on the technical task of separating similar voices.
The results show clear limits today. Real-time voice agents scored at or below 22% on the multiparty comprehension questions. For turn-taking — deciding when to speak, interrupt, or stay silent — agents performed near chance levels, close to a 50% baseline. In short, current real-time agents often do not understand who is speaking well enough to join a group conversation reliably.
The authors also ran an ablation where speaker voices were made more similar. Abilities that depend on telling voices apart fell dramatically. Comprehension fell to about 4%, speaker-name accuracy dropped to 0%, and discussion appropriateness declined (reported from roughly 75% to 45% in their numbers). Turn-taking remained near or below chance. These tests show that when voice separation gets harder, multiparty understanding collapses.
Why this matters: many real-world interactions are multiparty — meetings, family chats, or group events. For a voice agent to be useful in those settings it must track who is speaking, follow the flow, and choose the right moments to talk. MP-Bench highlights that current agents are far from this ability and points to specific weaknesses to improve.
The authors frame MP-Bench as a first step. They made methodological choices — such as using distinct voices — to isolate the core multiparty challenge and note that this decision affects what the benchmark measures. They also report using generative AI tools only in supporting roles like writing polish and code debugging, and they examined judge robustness across model families. These caveats mean MP-Bench exposes a clear gap in agent behavior but is not the final word on every aspect of real-world multiparty interaction.