Any component that sits in the request path between your product and a model provider is a dependency your users can feel. Customers ask us the same question in the first technical call: what happens to my product if PALM SOUQ is down? The honest answer is that it depends on a setting you choose, and this post explains the default and the tradeoff.
The default: fail-open
Our Python and TypeScript SDKs wrap the provider clients you already use. In the normal path, requests go to gateway.palmsouqltd.com and the gateway forwards them to the provider. The SDK keeps a lightweight health view of the gateway. If the gateway cannot be reached within the probe timeout, the SDK sends the request directly to the provider using your own provider key, and records the usage locally.
When the gateway is reachable again, buffered usage records are replayed so per-tenant metering and billing stay complete. Your users see no error. The provider bill is the same either way.
What you give up while open
While the SDK is routing directly, the gateway's controls are not in the path. Per-tenant quotas cannot refuse a request in real time, model fallbacks do not apply, and the four loop heuristics are not evaluated centrally. The SDK keeps a local step ceiling per session as a backstop, but it cannot see traffic from other processes, so it is weaker than the edge breaker.
In other words, fail-open trades budget containment for availability during the rare windows when the gateway is unreachable from your servers. With a 99.999% gateway SLA, that window is about 26 seconds a month at the SLA limit. Most of the real-world fail-open events we see are caused by network problems inside the customer's own environment, not by the gateway.
When fail-closed is the right choice
Fail-closed means the SDK returns an error instead of calling the provider directly. Your product loses that feature while the gateway is unreachable, but no request ever escapes metering or limits. We recommend fail-closed for three situations:
- Free and trial tiers, where a short outage is acceptable and an uncapped loop is pure loss.
- Autonomous background agents with no human waiting, where a delay costs nothing and a runaway costs a lot.
- Regulated workloads where every model call must be logged by a central component for audit.
Choose per feature, not per company
The setting is a per-client option, so you can run the interactive assistant fail-open and the overnight research agent fail-closed in the same codebase. Teams that pick one mode for everything usually end up either over-exposed on background agents or over-cautious on the chat experience their customers see.
from palmsouq import PALM SOUQ
# Interactive assistant: keep the product up
chat = PALM SOUQ(provider="anthropic", fail_mode="open")
# Overnight research agent: never run unmetered
research = PALM SOUQ(provider="anthropic", fail_mode="closed", tenant_header="X-PalmSouq-Tenant")How we test it
Every SDK release runs a failover suite that blackholes the gateway mid-stream, returns slow responses, and returns malformed responses, and checks three things: the user request completes in open mode, it fails cleanly in closed mode, and the replayed usage records match the provider usage exactly. A metering gap after a failover would undermine the whole point of per-tenant billing, so we treat a mismatch as a release blocker.
If you want to see the behavior yourself, the demo includes a provider outage button and a gateway path toggle, and the SDK documentation includes the exact timeouts and buffer sizes.
Rohan Malhotra
VP Engineering, PALM SOUQ
NIT Trichy. Previously led streaming data infrastructure at Flipkart. Responsible for the token accounting pipeline and the 99.999% availability target.