Failover that actually fails over

Most gateways describe their failover story in terms of health checks: poll the upstream, mark it down, stop sending traffic. That works for a machine that is off. It does not work for the failure mode that actually shows up in production, which is an upstream that answers every probe in 40ms and then takes ninety seconds to finish a real completion.

Probe the thing you sell

Our health signal is not a ping. It is a continuous trickle of real completions against every model on every upstream, measured on time-to-first-token and tokens-per-second rather than on whether the socket opened. A pool whose p95 TTFT triples is degraded, even if its error rate is zero.

• Synthetic completions per model, per region, every few seconds
• Decisions driven by TTFT and throughput percentiles, not liveness
• Shift traffic gradually — a cliff-edge cutover moves the outage rather than fixing it

Fail over without breaking semantics

Rerouting mid-stream is the hard part. Once a client has received tokens you cannot silently restart on another pool without producing a response that contradicts itself. We only reroute before the first token; after that we ride the request out and let the retry budget handle it.

if attempt.tokens_emitted == 0 and pool.degraded():
    pool = router.next_healthy(model, exclude={pool})
    continue  # safe: client has seen nothing yet
else:
    stream_through(attempt)  # committed

The ten-hour test

In November 2023 a major provider was down, in the sense that matters, for the better part of a working day. Our routing layer moved traffic to alternate regional capacity inside of a minute, without anyone being paged. Customers saw a latency bump. Almost none of them noticed anything else.

The measure of a failover system is not that it triggers. It is that nobody has to know it did.