I ran into a problem with the responses API with 5.6. Our system uses function tools extensively. We have a large stable input context, and then we see a series of function tool calls, each appending tool results onto the end of that context until a final cycle where the response text is generated. In our system we require that at least one tool is called before the final generation happens.

The problem we’re running into is that we effectively need tool_choice=“required” on the first several calls while we assemble the evidence for the answer, and then we need tool_choice=“none” on the final call.

But, it appears, that tool_choice is rendered into the input context. And therefore if you change the tool_choice on the last cycle, you are effectively losing the entire cacheable prefix and we have pay write tokens for the entire size of the last cycle. That’s very expensive.

If we just leave tool_choice=“auto” (or not specify it at all), it normally works. But the model occasionally decides that it doesn’t really need to call a tool so it won’t.

Anyone have any clever ideas on how to change parameters like this without losing the entire cache? It’s easier to notice this problem now that OpenAI has made explicit caching effectively mandatory. You can still use implicit caching if you want – but it is never a better choice, just a lazy one.