Gender bias varies sharply across leading language models

An audit of ten large language models from nine vendors found inconsistent gender effects across stereotype attribution and moral-judgment tasks. Some systems favored protecting women from hypothetical harm, others favored men, and several responded identically regardless of gender.

For developers, the variation makes model-specific testing essential. A bias result from one provider cannot predict another provider’s behavior, and replacing a model may change both the direction and magnitude of disparities in an application.

Same prompts, ten systems

The study, titled Gender bias across LLMs is common and highly heterogeneous (arXiv), addresses a gap left by earlier research that tested relatively few models. Its authors ran two controlled experiments across systems released between April 2025 and June 2026:

• Claude Sonnet 4.6
• Claude Fable 5
• GPT-5.5
• Gemini 3.1 Pro
• Llama 4 Scout
• Grok 4.1 Fast
• Mistral Small 4
• Microsoft Copilot
• DeepSeek V4-Flash
• Qwen3.6

The researchers defined bias as a systematic difference between matched prompts after changing a gender cue. That measurement captures asymmetric outputs, though it cannot identify whether the behavior comes from training data, post-training, safety policies, or prompt interpretation.