Procedure · 6 steps
- 1Classify features by quality sensitivityList every LLM feature and mark it as tolerant (classification, extraction, routing, short replies) or sensitive (the core output customers pay for).
- 2Pick fallback pairsFor each tolerant feature choose a cheaper model from the same provider family first, so prompts and tool schemas carry over with minimal changes.
- 3Evaluate the pair offlineReplay a few hundred recent requests through the fallback model and score the outputs against the primary before enabling anything in production.
- 4Set triggersConfigure the fallback to activate on a tenant budget threshold, on provider errors, and optionally on latency, each scoped to the tolerant features.
- 5Tell the userRead the served-model response header and show a short notice when a fallback model answered, with an upgrade path if the trigger was budget.
- 6Measure and adjustTrack quality signals and support tickets per served model for two weeks, then widen or narrow the feature scope.
A model downgrade policy decides which cheaper model answers when the preferred model should not. Done well, it turns a hard budget limit into a soft one, absorbs provider outages, and protects margin without a visible change in product quality. Done badly, it quietly degrades the feature customers pay for. This guide is the procedure we use with customers.
Why fallback beats refusal
A refusal stops the customer's work. A fallback keeps it going at lower cost. At 6,000 input and 500 output tokens per call, the cost difference between common pairs is large, which is why a fallback can carry a tenant through the end of the month on a small remainder of budget.
| Pair | Primary per call | Fallback per call | Ratio |
|---|---|---|---|
| Claude Sonnet 4.5 to Claude Haiku 4.5 | $0.0255 | $0.0085 | 3.0x |
| GPT-4.1 to GPT-4.1 mini | $0.0160 | $0.0032 | 5.0x |
| GPT-5 to GPT-5 mini | $0.0125 | $0.0025 | 5.0x |
| Gemini 2.5 Pro to Gemini 2.5 Flash | $0.0125 | $0.0031 | 4.1x |
Step 1: classify features
List every feature that calls a model and decide how sensitive it is to model quality. Classification, intent detection, extraction into a schema, routing and short conversational replies usually tolerate a smaller model well. Long-form drafting, complex reasoning over many documents and code generation usually do not. Tag every request with X-PalmSouq-Feature so the policy can tell them apart.
Step 2: pick pairs within a family first
Falling back within the same provider family, such as Sonnet to Haiku or GPT-4.1 to GPT-4.1 mini, keeps prompt format, tool calling conventions and safety behavior close. Cross-provider fallbacks are valuable for outages, but they need their own prompt tests, because system prompt conventions and tool schemas differ.
Step 3: evaluate offline
Replay a few hundred recent production requests per feature through the fallback model and compare outputs with the primary. Use whatever quality signal the feature already has: exact match for extraction, a rubric for summaries, test pass rate for code. If the fallback fails the bar for a feature, leave that feature off the fallback list rather than lowering the bar.
Step 4: set triggers
- Budget: activate at a share of the tenant's monthly budget, commonly 90%, after a soft-cap notice at 80%.
- Reliability: activate on provider 5xx responses or rate-limit errors for the primary, for every feature, not only tolerant ones.
- Latency: optionally activate when the primary's recent p95 exceeds a threshold for interactive features.
fallbacks:
- name: budget_degrade
when: { tenant_budget_percent_gte: 90 }
features: ["triage", "tagging", "short_reply"]
chain: ["claude-sonnet-4.5", "claude-haiku-4.5"]
- name: outage
when: { upstream_status: [500, 502, 503, 529] }
features: ["*"]
chain: ["claude-sonnet-4.5", "gpt-4.1"]Step 5: tell the user
PALM SOUQ returns the model that actually served each request in a response header. Use it. A one-line notice, such as answered with the fast model for the rest of this billing period, explains a change in tone or depth before the user files a ticket, and it is the natural place for an upgrade link when the trigger was budget.
Step 6: measure
For two weeks after enabling a fallback, compare quality signals and support tickets by served model for each feature. Attribution records the served model and the original requested model on every event, so cost savings and quality impact can be read from the same data. Widen the feature list where quality holds and remove features where it does not.
Review the pairs whenever a provider releases a new small model. The gap between frontier and small models changes quickly, and a feature that needed the primary last quarter may not need it now.
Nisha Pillai
Head of Customer Engineering, PALM SOUQ
BITS Pilani. Previously solutions architect at Wipro for generative AI rollouts. Runs onboarding, breaker tuning workshops and the 15-minute demo.