Across ImplicitBBQ tests, large language models were far more likely to choose a person as a criminal when religion was signalled through a practice such as praying five times a day than when a prompt explicitly identified someone as Muslim or Christian. Researchers from IIIT Hyderabad and IIT Kharagpur report the pattern in the ImplicitBBQ preprint, which introduces a question-answering benchmark for demographic identities conveyed through cultural practices and personal attributes.

Across 11 models, including eight open-weight systems with downloadable parameters and three closed services, mean bias in ambiguous contexts was more than six times higher with implicit cues than with explicit labels. Safety instructions and chain-of-thought prompts reduced average bias but left frequent failures to abstain. Few-shot examples produced the largest improvement, while caste bias remained high under every prompting strategy tested.

Names blur more than they reveal

Earlier work often measured implicit bias by replacing a demographic label with a stereotypically associated name, such as Muslim with Mohammed or woman with Julia. The authors identify four problems with that method:

• Names provide no consistent signal for dimensions such as age or socioeconomic status.
• Name-to-identity mappings vary across cultures. Jean is commonly read as female in English and male in French, while Sameer can signal Hindu or Muslim identity depending on the community.
• A single name can imply gender, religion, region and ethnicity simultaneously, obscuring which association affects the model.
• Researchers often assume these mappings without validating them with the relevant communities.

ImplicitBBQ represents identity through characteristic-based cues such as wears a janeu, referring to a sacred thread associated with some Hindu communities; prays namaz; has wrinkles; or shops at discount stores. Human annotators validated the demographic association of every retained cue.

From 140 cues to 8,000 tests

The dataset extends the