A public interactive benchmark called Multimodal‑SEA was released to measure how well personal AI agents can resist social‑engineering attacks that use forged text, images, documents, and web pages. The benchmark contains 608 expert‑designed tasks drawn from real fraud cases. These tasks cover 14 domains, 152 attack scenarios, and four difficulty levels and are split between Chinese and English environments.

The evaluation framework uses a multimodal attack harness that provides the model with task information, controls how an attack unfolds, and supplies the multimodal content that the attacker tries to use. The harness relies on simulated applications and records tool‑execution logs to verify whether an agent actually performed a requested action.

When the harness was run against seven widely used multimodal models, every one of them succeeded in at least some attacks. Attack success rates ranged from 7.1 % to 34.5 % across the 608 benchmark tasks.

Adding multimodal content to the attacks raised success from 19.9 % to 29.2 %. At the same time, proactive verification coverage increased from 54.7 % to 65.1 %, showing that simply improving verification does not stop deception.

Enforcing stronger safety instructions lowered attack success rates—for example, GPT‑5.6‑Sol‑SP dropped from 34.8 % to 10.8 %. However, these instructions also increased multimodal deception miss rates from 50.4 % to 63.5 %, indicating a trade‑off between refusing harmful actions and recognizing deception.

Some models exhibit excessive refusal of legitimate requests. Claude Opus 5 refuses 32.1 % of verified legitimate requests, while MiniMax‑M3 refuses 51.2 %.

The study highlights that current multimodal agents are still susceptible to sophisticated, multimodal social‑engineering attacks and that safety mechanisms must balance refusal and deception detection.