A gateway in front of an LLM provider is only acceptable if nobody notices it. Model calls already take hundreds of milliseconds to seconds, so a few milliseconds sound negligible, but agents chain dozens of calls per task and streaming interfaces are judged on time to first token. Our target is under 5 ms of added latency per request, and this post explains how the work is arranged to stay inside it.
Do less before the first byte
The single most important design rule is that only decisions happen before the request is forwarded. Admission needs four answers: who is the tenant, is the tenant within its rate limit, is it within its budget, and is any breaker open for this session. Everything else, including pricing the response, updating the ledger and emitting billing events, happens after the response has started streaming back or after it completes.
The pre-forward path
| Step | Where the data lives | Why it is fast |
|---|---|---|
| Terminate TLS at the nearest PoP | Edge node | Short network round trip from your servers |
| Read X-PALM SOUQ headers | Request | No body parsing needed |
| Load tenant policy | In-memory cache, pushed on change | No database call per request |
| Rate limit check | Sliding window counters at the edge | Local increment |
| Budget check | Running total replicated to the edge | Local comparison |
| Breaker state | Per-session rolling state in memory | Hash and compare |
The budget check problem
Rate limits are easy to enforce locally. Budgets are harder, because a tenant's spend accumulates across every edge location that serves it. We replicate running totals to the edge continuously and enforce against the local copy. In the worst case a tenant can overshoot its budget by the requests in flight during one replication interval, which is cents for typical traffic. We decided that small, bounded overshoot was better than a cross-region lookup on every request.
Streaming without buffering
The gateway never buffers a full response before forwarding it. Server-sent event chunks are passed through as they arrive. Usage is read from the final chunk or from the usage block providers include when streaming usage is requested, and the SDK sets that option for you. If a provider omits usage, the gateway counts tokens with the provider's tokenizer after the stream closes, off the critical path.
Measuring it honestly
Added latency is measured as the time between receiving the request and sending it upstream, plus the time between receiving the first upstream byte and sending it downstream. We export both numbers per request to your OpenTelemetry pipeline, so you can see the gateway's share of every trace rather than trusting a headline figure. Provider time to first token is measured separately and reported per tenant and per model.
What we chose not to do in the path
Several features customers ask for would be easy to build in the request path and would cost latency on every call: payload inspection with a second model, prompt rewriting, and synchronous writes to a central database. We keep these out of the path or out of the product. Loop detection uses in-memory hashes and fingerprints instead of a classifier model, and anything that needs a durable write happens after the response, from a local queue that survives a node restart.
When the gateway is not reachable
Latency is one half of the reliability story. The other half is what happens if the gateway cannot be reached at all. The SDKs fail open by default and route directly to the provider, with metering replayed on recovery. That behavior, and when to choose fail-closed instead, has its own post.
Rohan Malhotra
VP Engineering, PALM SOUQ
NIT Trichy. Previously led streaming data infrastructure at Flipkart. Responsible for the token accounting pipeline and the 99.999% availability target.