Gemini Thinking Tokens Eat maxOutputTokens

The short version

Gemini’s thinking tokens consume the maxOutputTokens budget. Unless thinking is explicitly capped, thinking alone can reach the ceiling before the first token of structured output is generated, and the response ends empty with no error and no warning.

On the chat AI screen this showed up as a production hang: the “thinking” indicator never went away. The fix is not in one place but in three layers — client-side detection of an unterminated stream, a server-side stall abort, and an explicit thinking budget.

What it looks like

• Sending a message to the chat AI shows “thinking”
• In some cases no response ever comes back
• It never moves to an error screen. It just looks stuck on the loading state
• Until they reload, the user cannot tell whether it failed or whether waiting will help

In the logs, the Vertex call has not failed. The meta frame was sent. Only the response body after it never arrives.

Why

Three causes stacked.

1. Thinking tokens eat the output budget

The maxOutputTokens passed to streamObject is, on Gemini 2.5, the sum of thinking tokens and output tokens. If thinking runs long it exhausts the budget before the structured output — fields like reply or schedules — begins. The call itself is treated as completing normally, so the model layer reports nothing unusual. From the caller’s side, “ran out of budget while thinking” and “correctly generated nothing” are indistinguishable.

2. The server connection closes quietly

Cloud Run’s default request timeout is 300 seconds. With the model unresponsive and only the connection open, the socket sits pinned until the timeout with just the meta frame sent. With no mechanism to cut it actively, the server does nothing, treating it as “still generating”.

3. The client waits forever for the terminator

The client’s read loop only advances status past streaming when it receives a done or error frame. If the server connection closes on a timeout or proxy disconnect without sending a terminal frame, the client has no way to know, and status stays at streaming. “Thinking” never disappears because the UI never detects the abnormal end of the stream.

Fixing it

We built a layer for each of the three causes. Fixing one leaves the other two able to produce the same symptom independently, so all three layers are needed.

Layer 1: server-side stall abort

Re-arm a timer on every partial, and if there is no progress for a fixed period (60 seconds), actively abort the Vertex call itself with an AbortController. The same mechanism watches both the wait for the first token and a stall mid-generation.

const stallController = new AbortController();
let stallTimer: ReturnType<typeof setTimeout> | undefined;
const armStall = () => {
  if (stallTimer) clearTimeout(stallTimer);
  stallTimer = setTimeout(() => stallController.abort(), stallMs);
};

On abort, partialStream and done reject and the failure is folded to the client as stream_failed. We cut well before Cloud Run’s 300 seconds rather than waiting for it.

Layer 2: client-side detection of an unterminated stream

When the read loop ends, if status is still streaming, settle it as a retryable failure.

export function finalizeUnterminatedTurn(state: ChatState): ChatState {
  if (state.status !== "streaming") {
    return state;
  }
  return { ...state, status: "error", error: { reason: "stream_failed" } };
}

The point is that it returns unchanged when done or error has already arrived — it never overwrites a normal completion or a known refusal. This is a design that does not assume “the server always sends a terminal frame”, and puts the settling logic for the case where none arrives on the UI side too.

Layer 3: state the thinking budget explicitly

The first two layers are about handling what happened. The third mitigates the root cause. We wired GEMINI_THINKING_BUDGET=0 into the environment through Terraform, disabling thinking itself.

0        … thinking disabled
positive … cap on thinking tokens
unset    … the SDK's default dynamic allocation

The structured draft now emits its first token quickly, and the situation where thinking eats maxOutputTokens stops occurring at all.

Preventing a repeat

We split it into three layers because we knew any one of them alone leaves the same symptom reproducible through the other two paths.

• Setting the thinking budget to 0 alone still leaves the server holding the connection open when the model goes unresponsive for some other reason
• The server-side stall abort alone cannot be noticed by the client if the abort itself fails to send a terminal frame
• Client-side detection alone leaves the Vertex call alive indefinitely, holding server resources

Behind the single symptom of a stuck “thinking” indicator were paths that could break independently in the model layer, the server layer and the client layer, so rather than picking one layer, we put settling logic everywhere the symptom can originate.