Most AI Safety Tests Run in English. Four Indian Languages Now Have Their Own.

A safety refusal that holds in English can collapse in Hindi, Bengali, Marathi, or Punjabi once the same request is wrapped in an authority claim, a logical appeal, or one of four other persuasive frames, according to a new open benchmark called IndicSafeEval.

The preprint lays out a 4-language by 6-strategy by 10-category matrix (7,200 adversarial prompts in total) built to test whether multilingual models break the same way English ones do. The four languages are Hindi, Bengali, Marathi, and Punjabi. The six persuasive strategies are Logical Appeal, Authority Endorsement, Misrepresentation, Anchoring, Priming, and Confirmation Bias. The ten categories cover safety-critical content such as medical, legal, and self-harm advice.

Safety performance depends on both the target language and the persuasive framing, and the worst breakdowns are not random. They cluster in the risk categories where social pressure is most effective, so a deployment's defense is the interaction between language and frame, not the refusal rate alone.

The benchmark is black-box and limited to open-source models, so closed frontier systems (GPT-4o, Claude, Gemini) are not measured here. Coverage is four of the 22 scheduled Indian languages, leaving Tamil, Telugu, Malayalam, and others untested. The work is a preprint, not peer-reviewed, and specific per-model failure rates should be read as provisional.

The dataset, prompt templates, and judge code are public on GitHub, so anyone can rerun the 7,200-prompt matrix against their own open-source model before shipping to Hindi, Bengali, Marathi, or Punjabi speakers.