Skip to content
PALM SOUQ

Dynamic model fallbacks

Graceful degradation instead of broken user experiences.

When a tenant nears its budget, or a provider starts failing, PALM SOUQ reroutes the request in the gateway: to a cheaper model, another provider, or a cached answer. Your users keep working and your margin stays intact.

Start free trial Explore the demo
Budget downgrades · cross-provider failover · cache at cap · no redeploys
routing chain · support-agent v23
{
  "chain": "support-agent",
  "version": 23,
  "primary": "claude-sonnet-4.5",
  "rules": [
    { "when": { "provider_error_rate_gt": 0.05, "for": "60s" },
      "route_to": "gpt-4.1" },
    { "when": { "tenant_budget_pct_gte": 90 },
      "route_to": "claude-haiku-4.5",
      "notify": "quota.downgraded" },
    { "when": { "tenant_budget_pct_gte": 100 },
      "serve": "response_cache",
      "on_miss": "claude-haiku-4.5" }
  ],
  "preserve": ["tools", "system", "max_tokens"]
}

Chain builder

Build a chain, then push the budget past the threshold.

Choose the models, move the budget slider, simulate a provider error spike. The serving node and the cost per call update from the PALM SOUQ price table.

routing / chain: support-agent

  1. Primary

    Claude Sonnet 4.5

    $0.0270/call

    serving

  2. Provider failover

    GPT-4.1

    $0.0168/call

  3. Budget ≥ 90%

    Claude Haiku 4.5

    $0.0090/call

  4. Budget ≥ 100%

    Response cache

    $0 on hit

72%
90%

Budget at 72%, under the 90% downgrade threshold. Primary provider healthy.

$0.0270 /call

primary price

Switching policies

Three triggers, evaluated on every request.

Rules are checked in order in the edge worker, inside the same latency budget as quotas and breakers.

  1. 01

    Budget downgrade

    A tenant crosses a share of its monthly budget, 90% by default. Move them from a frontier model such as GPT-4o to GPT-4o mini or DeepSeek V3.2, and send quota.downgraded so your app can tell the user.

  2. 02

    Provider failover

    A provider's error rate or latency passes a threshold for a sustained window. Route to a model from a different provider, for example OpenAI to Anthropic Claude Sonnet 4.5, and return when health checks recover.

  3. 03

    Cache at cap

    At 100% of budget, answer repeated read-only queries from the response cache at zero model cost, and send cache misses to the cheapest model in the chain instead of returning an error.

Downgrade economics

What a downgrade saves, per thousand calls.

Each call priced at 6,000 input and 600 output tokens, uncached, from current list prices in the PALM SOUQ price table.

PrimaryPer 1k callsFallbackPer 1k callsSaved
GPT-4.1$16.80GPT-4.1 mini$3.3680%
GPT-4o$21.00GPT-4o mini$1.2694%
Claude Sonnet 4.5$27.00Claude Haiku 4.5$9.0067%
GPT-5$13.50GPT-5 mini$2.7080%
Gemini 2.5 Pro$13.50Gemini 2.5 Flash$3.3076%
GPT-4.1$16.80DeepSeek V3.2 (chat)$1.9389%

Zero-downtime routing

Change routing in the console, not in a release.

Every routing chain is a versioned policy. Edit it in the dashboard or through the API, preview its effect on last week's traffic, publish to all edge PoPs in seconds, and roll back with one click.

support-agent / history
VerWhenByChangeState
v23Nov 6, 14:02ananya@Downgrade threshold 85% → 90% for pro tierlive
v22Nov 2, 10:41rohan@Added gpt-4.1 as cross-provider failoversuperseded
v21Oct 28, 18:15nisha@Response cache at 100% for read-only featuresrolled back
v20Oct 21, 09:30rohan@Fallback claude-3.5-haiku → claude-haiku-4.5superseded

Fallback questions

Does a fallback change the response format my code receives?

No. When the fallback is in the same provider family, the request and response shapes are identical. For cross-provider failover between OpenAI-style and Anthropic-style APIs, PALM SOUQ translates the request and the response so your client receives the format it sent, including tool calls.

How do I know a request was served by a fallback?

Every response carries X-PalmSouq-Served-Model and X-PalmSouq-Route headers, and the decision is written to the audit log with the rule that fired. Metering uses the model that actually served the call, so tenant COGS stays exact.

Can I exclude some features from downgrades?

Yes. Rules match on tier, tenant, feature and user, so a contract-review feature can stay on the frontier model while chat suggestions downgrade first. Exclusions and priorities are part of the same versioned policy.

How are routing changes deployed?

Policies are versioned in the console or through the API and pushed to every edge PoP within seconds. There is no SDK update and no application redeploy. Any version can be rolled back with one click, and the change log records who changed what.

More of the platform

Degrade gracefully instead of locking customers out.

Route your agent traffic through PALM SOUQ in 2 minutes. Change one base URL, pass a tenant header, and every token is attributed and governed.

14-day free production trial · 100k requests included · no card required

Start free trial Explore the demo