This is exactly why I have become increasingly skeptical of choosing a model purely by reputation, benchmark position, or reasoning label.

Recently, I watched another developer run the same prompt through several models and reasoning levels. At first, the expected outcome seemed obvious: the most advanced model with the highest reasoning setting should produce the strongest result.

That was not consistently what happened.

The outputs differed in visible details—typography, icon choices, layout decisions, the amount of explanation—but the most expensive or longest-reasoning configuration was not always the one that solved the actual task best.

I have been seeing something similar in my own work, including tasks far beyond coding. I use models for biology, chemistry, food-product development, patent research, formulation decisions, and technical investigations where the system must combine knowledge from several disciplines, evaluate recent evidence, identify uncertainty, and avoid making confident but unsupported conclusions.

In those cases, longer reasoning does not automatically produce better reasoning.

Sometimes it produces:

• more elaborate justification for an incorrect assumption;

• unnecessary expansion of the task;

• additional speculative branches;

• excessive refactoring or redesign;

• higher token usage without a proportional increase in correctness;

• a polished answer that is weaker at the actual decision point.

Because of this, I now treat model selection as a small empirical evaluation rather than a permanent preference.

For the first meaningful stage of a project, I often give exactly the same task, context, constraints, and expected output format to several models. Then I compare them using criteria that matter for the real project:

1. Did it correctly understand the task?

2. Did it remain inside the requested scope?

3. Did it identify the highest-risk assumptions?

4. Did it use the provided evidence rather than inventing missing details?

5. Is the result directly usable?

6. What is the cost and latency?

7. How much verification or repair is still required?

8. Did additional reasoning materially improve the result?

This last question is especially important. A model that produces an 85% correct result in two minutes may be more useful than one that produces an apparently sophisticated 88% result after consuming several times more context, reasoning budget, and review time.

I also try not to evaluate models only on artificial benchmark prompts. A coding model should be tested on my repository, my architecture, my constraints, and my failure modes. A research model should be tested on a question where I already understand enough of the domain to detect shallow synthesis, outdated evidence, unsupported extrapolation, or citation laundering.

My current workflow is roughly:

• run a controlled first-stage comparison;

• select the strongest initial approach;

• preserve the best assumptions, constraints, and discoveries;

• continue the task in smaller verifiable stages;

• occasionally ask another model to challenge the current solution rather than independently recreating the entire project;

• measure not only answer quality, but total correction cost.

In practice, the best model may change between stages. One model may be better at repository exploration, another at implementation, another at finding hidden failure cases, and another at reviewing the final result. The useful unit may not be “the best model,” but “the best model for this specific transition in the workflow.”

This also changes how I think about harness engineering. The harness should not only preserve memory and context. It should help route tasks, compare candidate outputs, detect unsupported claims, enforce scope boundaries, and decide when additional reasoning is actually worth paying for.

I am continuing to collect results from these comparisons across OpenAI, Google, and other models.

Have any of you run controlled comparisons using the exact same real project task—not a benchmark—across different models and reasoning levels?

I would be particularly interested in cases where a lower reasoning setting or supposedly weaker model produced the more correct, usable, or economical result. What criteria are you using to decide which result actually won?