arXiv News

First-hand information for everyone

Site

AboutPrivacy PolicyTerms of Use

© 2026 arXiv News

Large‑language models easily impersonate users when text is seeded with fake structure and content | arXiv News

arXiv News

EnglishJapanese

Switch language

EnglishJapanese
Loading account…

First-hand information for everyone

Back to front page
CryptographyEnglishPublished

Large‑language models easily impersonate users when text is seeded with fake structure and content

November 12, 2026arXiv: 2610.40396v1

Recent experiments show that many state‑of‑the‑art chat models can be tricked into believing that malicious text coming from a tool output is actually written by the user. The attackers combine false special tokens with crafted content, deceiving the model’s role‑tracking system. In the AgentDojo workspace family, success rates for such role‑spoofing attacks ranged from 63.1 % to 103.6 % across a broad set of frontier LLMs, with GPT‑5.4 achieving a 96.2 % injection‑to‑control ratio. This high success illustrates how challenges that blend structural spoofing with content alterations are far more potent than either alone.

Table of injection versus control success rates for role‑spoofing attacks across several large language models, displaying numbers for each model's control, injection, and ratio columns. GPT‑5.4 has a ratio of 96.2%, and the ratios span 63.1% to 103.6%.
ModelControlInjectionRatio
GPT-5.449.8%47.8%96.2%
Claude Opus 4.834.8%22.0%63.1%
Claude Opus 528.6%29.7%103.6%
Claude Sonnet 565.5%43.2%65.9%
Gemini 3 Pro66.6%61.7%92.6%
Gemini 3.1 Pro46.9%45.2%96.3%
Figure 1. Figure 1 shows, for seven frontier LLMs, the proportion of times a role‑spoofing injection causes the model to follow the malicious instruction relative to a user‑issued instruction (control). The table lists the control rates, injection rates, and the injection‑to‑control ratio per model; GPT‑5.4 attains a 96.2 % ratio, while ratios across models range from 63.1 % to 103.6 %.Edelman, Ezra, Edelman, Benjamin, Goel, Surbhi, Malach, Eran · arXiv:2610.40396v1 · Figure 1: · CC BY 4.0

To counter this threat, researchers developed a lightweight detector that uses linear probes trained on the model’s internal activations. The probes learn the model’s belief about who authored each token. When the model mistakenly attributes a token in a tool response to the user, the probe flags this as a spoofing attempt. Across many held‑out test cases—including injections that fool the model’s own attribution—the probes achieve near‑perfect separation between clean and injected examples. Their performance is best when the training data contains uniformly random tokens, suggesting that the probes capture general patterns of role attribution rather than specific attack signatures.

The detection method shows a very low false‑positive rate (below 0.1 %) while maintaining high true‑positive rates across diverse domains and injection types. These results demonstrate the feasibility of real‑time monitoring of user‑versus‑tool token origin without requiring expensive fine‑tuning or changes to the underlying model architecture.

Chart plotting true‑positive versus false‑positive rates for different linear probes, highlighting low false‑positive rates (below 0.1 %) and high true‑positive rates for detecting injected text.
Figure 5. Figure 5 presents receiver‑operator curves for linear probes trained to detect role spoofing. The blue line denotes the authorial‑role probe; the purple line represents a probe from Ye et al. For injected samples, the curves show true‑positive rates (y‑axis) versus false‑positive rates (x‑axis) up to 1 %. The illustration demonstrates that the probes achieve high true‑positive rates while keeping false‑positive rates below 0.1 %.Edelman, Ezra, Edelman, Benjamin, Goel, Surbhi, Malach, Eran · arXiv:2610.40396v1 · Figure 5: · CC BY 4.0

Source paper

Understanding and Mitigating Role Spoofing in LLMs

CryptographyArtificial Intelligence
View on arXiv