Summary

I encountered a speaker-feedback problem while testing gpt-realtime-2.1-mini through WebRTC in a browser-based voice application.

When using external speakers, the assistant’s own voice was captured by the microphone and interpreted as new user speech. This caused repeated short responses, response loops, and assistant speech being interrupted before the audio finished.

The same setup worked normally with headphones. In my environment, the problem also did not occur with gpt-realtime-2.1 under the same basic conditions.

This post describes the mitigation that worked in my environment.

Environment

  • Browser-based WebRTC client

  • OpenAI Realtime API

  • Model: gpt-realtime-2.1-mini

  • External speakers and a built-in/external microphone

  • Server VAD

  • Function tools enabled

  • Assistant audio played through the browser

Observed problem

The failure pattern was:

  1. The assistant started speaking.

  2. Speaker output was captured by the microphone.

  3. The captured audio was transcribed as user speech, sometimes as very short fragments.

  4. A new response was created.

  5. The current assistant audio was interrupted.

  6. The new assistant response was captured again, creating a loop.

Typical unwanted transcripts were short fragments such as greetings, single words, or parts of the assistant’s previous response.

Sometimes the complete assistant text arrived through the data channel, but the audible speech stopped before the sentence finished.

Control test

Using headphones stopped the problem.

This strongly suggested an acoustic feedback path between speaker output and microphone input rather than a general WebRTC connection failure.

Stage 1: audio and VAD tuning

I first applied:

audio: {
  echoCancellation: true,
  noiseSuppression: true,
  autoGainControl: false
}

I also increased the server VAD threshold:

turn_detection: {
  type: "server_vad",
  threshold: 0.7
}

The same turn_detection configuration was included in later session.update events so that it would not be lost or replaced.

This reduced the feedback, but it did not fully solve the problem.

Stage 2: response-aware microphone gating

The effective mitigation was to disable the outgoing microphone track while the assistant was producing or playing audio.

The final state machine used these rules:

  • On response.created: immediately disable the microphone track.

  • On output_audio_buffer.started: keep it disabled.

  • On output_audio_buffer.stopped or output_audio_buffer.cleared: restore the microphone after an 800 ms delay.

  • If a new response starts during that delay: cancel the pending restore.

  • On response.cancelled or an error: use a controlled delayed restore.

  • On function-call-only responses with no audio: restore the microphone through a separate tool-only recovery path.

  • On disconnect or reconnect: clear all timers and reset the microphone-gate state.

  • Do not restore the microphone only because a fixed amount of time has passed during a long assistant response.

Simplified example:

function setMicrophoneEnabled(enabled) {
  const tracks = micStream?.getAudioTracks?.() || [];
  for (const track of tracks) {
    track.enabled = enabled;
  }
}

function muteForAssistantResponse() {
  clearTimeout(restoreTimer);
  restoreTimer = null;
  setMicrophoneEnabled(false);
}

function scheduleMicrophoneRestore(delay = 800) {
  clearTimeout(restoreTimer);

  restoreTimer = setTimeout(() => {
    setMicrophoneEnabled(true);
    restoreTimer = null;
  }, delay);
}

The session was also configured with:

interrupt_response: false

This prevented microphone feedback from interrupting assistant speech while the microphone gate was active.

Result

In my environment, the combination of:

  • autoGainControl: false

  • server VAD threshold 0.7

  • interrupt_response: false

  • microphone muting beginning at response.created

  • microphone restoration after actual audio-stop events

  • explicit recovery for cancelled, failed, and function-call-only responses

substantially reduced the feedback loop and allowed complete spoken responses and tool-call workflows to finish normally when using external speakers.

The strongest improvement came after adding response-aware microphone gating. AGC and VAD tuning alone were not enough.

Trade-off

This design disables normal barge-in while the assistant is speaking. The user cannot interrupt the assistant naturally during that period.

For this application, reliable speaker playback was more important than barge-in. Other applications may need a more advanced approach using acoustic echo cancellation, audio-level analysis, or a dedicated echo-reference signal.

Questions

  1. Is increased sensitivity to speaker feedback an expected difference between gpt-realtime-2.1-mini and gpt-realtime-2.1?

  2. Is there an official recommended event sequence for microphone gating with WebRTC?

  3. Is response.created the best event for starting the gate, or is there another event that better represents the start of assistant audio generation?

  4. Are there recommended settings for external-speaker use when server VAD is enabled?

I am sharing this as a practical field report and workaround, not as proof of a model-level defect. I hope it helps other developers testing Realtime applications without headphones.