A new Stanford study finds that benchmark scores measure both model capability and compatibility with a static evaluation prompt. Replacing HELM’s standard prompt with a small set of structured prompting methods produced an average 6% performance gain and changed model rankings on five of seven benchmarks.

The research paper integrates DSPy, Stanford’s declarative prompting framework, with HELM, the Center for Research on Foundation Models’ evaluation suite. The authors test four frontier language models and two open-source models on general and medical tasks, then compare the structured-prompt results with HELM’s existing baseline scores.

One prompt can tilt the field

HELM, short for Holistic Evaluation of Language Models, typically runs each model with the same static prompt configuration. Model behavior depends heavily on instructions, formatting, demonstrations, and requests for intermediate reasoning, so a shared prompt can favor models whose training better matches its wording and structure.

That sensitivity complicates leaderboard comparisons. A lower score may reflect weaker task performance, poor prompt compatibility, or both. Teams choosing models for production can therefore underrate a capable model before testing the prompt formats it handles best, especially on medical benchmarks where small ranking changes may affect validation and procurement decisions.

Inside the HELM-DSPy bridge

DSPy represents a prompting workflow as a program with declared inputs and outputs. Modules such as Predict and ChainOfThought turn that declaration into model-facing instructions, while optimizers can search over instruction wording and few-shot examples.

The integration starts with HELM’s baseline task instruction, generates structured variants through DSPy, and sends each variant back through HELM’s evaluation pipeline. This preserves HELM’s benchmark logic while varying the prompt strategy in a standardized, reproducible way.