Google Cloud API Gateway just shipped model routing in Public Preview, and it solves one of the most tedious parts of building multi-model AI applications: managing a different endpoint, SDK, and payload format for every model provider you want to use.

The proxy problem it replaces

When building AI applications, developers need the freedom to route traffic to the best model for the job without hardcoding endpoints or managing open-source proxies. The usual workaround is running something like LiteLLM as a self-hosted sidecar, which means you now have a proxy server to deploy, scale, monitor, and keep alive. Google Cloud API Gateway now offers model routing in Public Preview to solve this, providing a lightweight, serverless ingress layer that accepts OpenAI-compatible requests and dynamically routes them to Gemini, Claude, or OpenAI OSS-GPT.

The key word is serverless. There is no proxy infrastructure to manage. The routing logic lives in your OpenAPI spec, and the gateway handles the rest at the network edge.

How it works under the hood

The architecture is straightforward. When a request comes in, the gateway does four things in sequence:

1. Intercepts the incoming POST /chat/completions request
2. Inspects the model field in the JSON payload (e.g. {"model": "claude-opus-4-7"})
3. Matches that model name against routing rules you defined in your OpenAPI 3.x spec, falling back to a default model if no rule matches
4. Transcodes the OpenAI-compatible payload in-flight into the native Vertex AI schema, then dispatches it to the right backend

That last step is the real work. Each provider (Gemini, Claude, OpenAI OSS) has a different request/response schema. The gateway handles that translation transparently, so your client code never changes regardless of which model is actually serving the request.