38ms and 180ms: reading keyless-inference overhead as a voice builder

Voice is the least forgiving surface you can ship an AI agent onto. Human conversation runs on turn-taking gaps averaging close to 200 milliseconds ie a figure that comes from real cross-linguistic research. A 2009 PNAS study by Stivers et al. measured turn-transition timing across ten languages and found gaps as tight as 7ms (Japanese) and as loose as ~470–490ms (Danish), with most languages clustering their most common gap between 0 and 200ms. Levinson and Torreira's follow-up work on the same data points out why this is remarkable: producing a spoken reply takes upwards of 600ms on its own, so speakers must be predicting where a turn will end well before it does. That's the bar a natural-sounding voice agent is measured against. It's not a UX nicety, a hardwired human expectation.

Most voice teams target a sub-second, and often sub-800ms, voice-to-voice budget (a caller stops speaking, agent's audio starts) already mostly consumed by the model's own response time and the network round-trip. Any layer sitting in that path has to justify every millisecond it adds.

Floe sits there: one key routing transcription, model tokens, speech, and telephony through a single metered endpoint, with spend controls enforced before a call goes out. So the fair question is exactly how much that costs.

The answer, on live traffic: p50 38ms, p99 ~180ms.

What "overhead" means here

The metric is floe_overhead_ms: the wall-clock time Floe adds on the caller's critical path, excluding the upstream vendor call (tracked separately as upstream_latency_ms) and any asynchronous post-response settlement. It's not a client-side round-trip, which would fold in network and model time we don't control. It's instrumented inside the gateway, across the path Floe actually owns: authenticate and gate the spend → resolve and route the model → stream the response → settle the debit.

Concretely: the model still takes however long the model takes. This number is only the routing, metering, and cap-enforcement that happen before and around the vendor call.

How it was measured

• Metric: floe_overhead_ms, persisted per call on the request ledger (one row per settled call), aggregated DB-side with percentile_disc (nearest-rank).

• Group: the keyless rail class — where Floe fronts the upstream provider from a pooled credential — reported separately from the proxy, BYOK, and x402-router paths, whose overhead profiles differ. Mixing them would blur the number.

• Window: a rolling 1-hour window over live production traffic.

• Sample: 2,141 keyless /v1/chat/completions calls (non-streaming). Two independent reads in the window: n=1,634 → p50 38ms / p99 166ms, and n=2,141 → p50 39ms / p99 181ms.

• Rail: measured on the direct-account (OpenAI-fronted) keyless rail. Because the metric excludes upstream latency, the choice of rail doesn't change what's being measured — the gate → route → settle path is the same regardless of which provider Floe fronts.

p50 is stable at 38–39ms; p99 sits in a 166–181ms band (±~15ms is normal tail jitter at this sample size), so we report it as ~180ms. This is a single-window snapshot on one rail — not a claim about every path through Floe, and we'll republish with a longer window as volume grows.

Why voice cares about p99, not just p50

• p50 (38ms) is the typical turn — half of calls see less.

• p99 (~180ms) is the worst case your users routinely hit — one call in a hundred.

For most apps, the median is all you read. For voice, the tail is the one that bites: the slow call is exactly the moment a caller hears an unnatural pause, and one awkward silence per hundred turns is enough to make an agent feel broken. A layer with a great median and an ugly tail still frustrates voice users. A bounded, measured tail is what lets you design around it instead of hoping it doesn't show up.

Where this sits next to other gateways

Published LLM-gateway overhead numbers span several orders of magnitude, mostly because they measure different things — a bare proxy against a mocked upstream is not the same test as a metered gateway on live production traffic with billing and spend enforcement in the path. With that caveat stated plainly:

Floe's median sits in the same band as Portkey and OpenRouter once routing and controls are switched on — not the bare-proxy numbers from LiteLLM or Helicone, which don't do spend enforcement at all. We think that's the honest comparison: Floe is doing more work per call than a pass-through proxy, and the number reflects that. What we haven't seen from any of the above is a published p99 on live traffic — most gateway benchmarks stop at a median against a fake upstream, which quietly deletes the tail that voice actually feels.

In context

Set 38ms and ~180ms against a 500–1,500ms voice-to-voice budget — the range cited across voice-infra vendors for a call that still feels natural, with production benchmarks from independent telemetry landing around p50 ~680ms / p95 ~1,180ms end-to-end. Floe's overhead is roughly 2.5–7.6% of that budget at p50, and even at the p99 tail it's a minority share of the tightest target. Set it against the components already inside that budget — Deepgram-class STT at roughly 150ms, an LLM leg commonly running 200–800ms+, TTS from Cartesia or ElevenLabs in the 75–85ms range — and 38ms of gateway overhead is closer to a rounding error than a line item.

That gap matters because performance is the thing actually slowing this market down. Deepgram's 2025 State of Voice AI survey found 72% of organizations name overall performance quality — clarity, natural flow, accuracy, and the latency underneath all three — as the top barrier to deploying voice agents in production; separately, AssemblyAI found 82.5% of builders feel confident building voice agents, but 75% still hit technical reliability walls once they ship. The market isn't stalling on ambition. It's stalling on exactly the milliseconds this piece is about, at a moment when Grand View Research puts the AI voice agent market at roughly $2.5B in 2025 growing past $35B by 2033.

The takeaway

Predictability is the point. When a layer's tail stays bounded and measured — not asserted — you can budget around it once and spend the rest of your latency where it actually matters: your model, your voice, your logic.

Numbers: p50 ≈ 38ms, p99 ≈ 180ms, n=2,141, rolling 1-hour window, live traffic, floe_overhead_ms (upstream-excluded, server-instrumented), keyless rail. Full methodology and sourcing in the docs.