arXiv News

First-hand information for everyone

Site

AboutPrivacy PolicyTerms of Use

© 2026 arXiv News

arXiv News

EnglishJapanese

Switch language

EnglishJapanese
Loading account…

First-hand information for everyone

Back to front page
Natural Language ProcessingEnglishPublished

A new multilingual safety dataset shows that English‑only testing underestimates how unsafe language models can be in other languages

November 12, 2026arXiv: 2610.40516v1

The MSafeTemp collection contains 1.8 million prompts in 60 languages that jump‑start research on safer multilingual AI. It is built by a three‑stage pipeline: the system first spots harmful words in a prompt, removes them to create a template, then translates the surrounding text. A language‑model judge checks that the translation still fits the template, and finally culturally authentic entities fill the slots. A figure illustrates these stages and the creation of generic harmful, benign and culturally localized variants.

Figure diagram of the MSafeTemp pipeline: template extraction → translation → judging → slot filling
Figure 1. Pipeline diagram showing the stages of template extraction, translation, slot filling, and judging that produced the 1.8 million multilingual prompts.Neuwinger, Max, Yukhymenko, Hanna, Schlag, Imanol · arXiv:2610.40516v1 · Figure 1: · CC BY 4.0

When the authors tested several popular models, they found that attacks that failed in English were successful after the prompts were translated. Moreover, attacks that localized a prompt to a specific culture failed in ways that pure translation could not reveal. A second figure shows that the dataset really does cover 60 different languages.

Table listing numbers of seeds, languages, templates, and prompt counts for WildGuard and HarmBench
TrackSeedsLanguagesTemplatesHarmfulBenignPost-hoc
WildGuard2,50044109,425/110,0001,537,628186,639+20,52897,120
HarmBench2006011,976/12,000173,74329,598+3,6829,463
Table 1. Table of MSafeTemp statistics confirming coverage of 60 languages across WildGuard and HarmBench datasets.Neuwinger, Max, Yukhymenko, Hanna, Schlag, Imanol · arXiv:2610.40516v1 · Table 1: · CC BY 4.0

The study highlights a key limitation: the judging step is performed by automated language‑model judges, which may introduce bias. The authors plan a full human‑evaluation study in the future.

Source paper

How I Aligned Your Model: Scaling Up Multilingual Safety Training with Templates

Natural Language ProcessingArtificial IntelligenceMachine Learning
View on arXiv
A new multilingual safety dataset shows that English‑only testing underestimates how unsafe language models can be in other languages | arXiv News