Using the envelope model gpt-realtime-2.1, we juxtapose below timing stats over a large sample of relatively short utterances across ordinary final transcription received on the api session v.s. out-of-band transcriptions returned for the same VAD-supported auto-committed utterances:
+------------------------------+-----------------+-----------------+
| Stage after commit | In-band | Out-of-band |
+------------------------------+-----------------+-----------------+
| Request acknowledged | already running | 173 ms median |
| First text received | 240 ms | 556 ms |
| First text → full completion | 274 ms | 89 ms |
| Full completion | 527 ms | 646 ms |
+------------------------------+-----------------+-----------------+
This suggests that the incremental in-band transcription gets a head-start but its final transcription arrives only some 10% faster on average than the out-of-band transcription which is only requested once the auto-commit indication arrives from the server.
The texts spoken (versatile, only rare repeats of the same text) were 1 - 16 words in this aggregate data analysis, and usually only 1-10 words or less.
It’s somewhat intriguing that the incremental transcription’s final transcript hasn’t a more salient advantage in the time it takes for it to reach the client side over the expanse of the Internet; the current difference in final arrival makes the in-band incremental transcription almost useless for a scenario expecting only short utterances ― it arrives mildly faster while its context steering is far less capable than that of the out-of-band api channel.
The last partial preceding the final was usually identical to it and arrives 200ms-300ms earlier, but you don’t know it’s the last before the final unless you wait for the final, so the partials are really not helpful in that scenario.
The partials huddle in a short post-commit burst, as data shows the timeline is typically:
utterance ends → short wait → all partial tokens burst out rapidly → roughly 0.2-second pause → formal final event.
And hence in general partials would have no utility when dealing with a relatively short utterance of just a few words auto-committed, you’d wait for a final not use partials for that in most user interaction designs (or practically all of them).
Tentative Conclusion and Suggestion
- Maybe a handle for switching off the partials would be very useful for many applications ― if it makes the server faster to send its final.
- Maybe differently chunking the final transcript would help.
These can potentially help scenarios where relatively short sentences or instructions are expected.
Obviously, for scenarios where a very long dictation is taken, the above doesn’t probably matter, but the vector for improvement would be that partials would arrive every so often and not only bunch up near the end of the dictation’s duration:
When trying a very long dictation with auto-commit off, no partials show up at all up until commit is being requested from the client-side, not even after a sentence is clearly over and there’s even a pause after it, but this deserves more than a quick check in passing.
We’re approximately 4 ms round-trip from the public API edge, which is remarkably close.