Any control in the request path needs a defined behavior for when it cannot be reached. For an LLM gateway the two options are fail-open, where the client calls the provider directly and the controls are bypassed, and fail-closed, where the client returns an error and the feature is unavailable. Neither is correct for every feature. This guide gives you a way to decide.
What each mode protects
| Fail-open | Fail-closed | |
|---|---|---|
| Product availability | Preserved | Feature unavailable while unreachable |
| Per-tenant metering | Buffered and replayed by the SDK | Complete, nothing escapes |
| Quotas and budgets | Not enforced while open | Enforced, requests refused |
| Loop breakers | Local step ceiling only | Full protection, nothing runs |
| Model fallbacks | Not applied | Not applicable |
How long is the window?
PALM SOUQ's gateway SLA is 99.999%, which allows at most about 26 seconds of unavailability in a 30-day month. In practice, the more common cause of an unreachable gateway is a network problem on the client side: DNS, an egress proxy or a firewall change. Your own network history is a better guide to the window than our SLA.
Expected cost of fail-open
While the client is open, an agent loop is bounded only by the SDK's local step ceiling. Our reference coding-agent loop spends about $26.67 a minute on Claude Sonnet. The exposure is that rate multiplied by the length of the open window and by the probability that a loop starts during it. For a few seconds a month this is negligible. For an hour-long client-side network incident on a platform with many unattended agents, it is not.
Expected cost of fail-closed
While the client is closed, the feature returns errors. The cost is the value of the work that did not happen: an unhappy user in an interactive chat, a delayed batch job, a support conversation that falls back to a human. For background work that can retry later, this cost is close to zero. For an interactive product in front of paying customers, it is the cost of a visible outage.
A decision rule
- Interactive, human waiting, paid tier: fail-open. Availability matters most and a human is watching for odd behavior.
- Unattended agents and batch jobs: fail-closed. Nothing is lost by waiting, and a loop with no observer is the costliest case.
- Free and trial tiers: fail-closed. A short outage is acceptable and uncapped spend is pure loss.
- Regulated workloads that require every call to be logged centrally: fail-closed.
- Anything else: fail-open with a conservative local step ceiling.
Configuring both in one codebase
The fail mode is a client setting, so create separate clients for each class of work rather than choosing one mode for the whole service. Keep the local step ceiling low on fail-open clients used by agents; it is the only loop protection while open.
import { PALM SOUQ } from "@palmsouq/sdk";
export const chat = new PALM SOUQ({ provider: "openai", failMode: "open", localMaxSteps: 25 });
export const agents = new PALM SOUQ({ provider: "openai", failMode: "closed" });
export const freeTier = new PALM SOUQ({ provider: "openai", failMode: "closed" });Test the failure path
A fail mode you have never exercised is a guess. In staging, block the gateway hostname and run your normal test suite. In open mode, confirm requests succeed and metering replays when you unblock. In closed mode, confirm the product shows a clear message and agents stop rather than retrying in a tight loop. Repeat after every SDK upgrade.
Document the choice for each client in your security and reliability reviews. Auditors and enterprise customers increasingly ask how an AI feature behaves when a dependency is unavailable, and a table of clients, fail modes and the reasoning behind each one answers the question in a minute. It also prevents the most common regression: a new service copies an existing client configuration without anyone deciding whether that mode suits the new workload.
Revisit the decision when a workload changes character. A chat feature that gains an autonomous mode, or a batch job that becomes user-facing, should move to the other mode in the same release.
Finally, alert on fail-mode transitions. The SDK emits an event when it opens and when it recovers; route it to the same channel as your other availability alerts so an open client never goes unnoticed for long.
Rohan Malhotra
VP Engineering, PALM SOUQ
NIT Trichy. Previously led streaming data infrastructure at Flipkart. Responsible for the token accounting pipeline and the 99.999% availability target.