Runaway loop circuit breakers
Stop infinite loops before they cost thousands.
Four heuristics run in the request path on every call. When an agent starts thrashing, the next request gets an HTTP 429 in under 50 ms, before the provider bills another token.
HTTP/1.1 429 Too Many Requests
Content-Type: application/json
Retry-After: 0
X-PalmSouq-Reason: circuit_breaker_triggered
X-PalmSouq-Heuristic: cyclical_tool_thrashing
X-PalmSouq-Session: s_4c19
X-PalmSouq-Request: req_01JD8QB7K2PZ
{
"error": {
"type": "circuit_breaker_triggered",
"message": "Session s_4c19 halted: tool git_merge_resolve called 15 times with identical arguments.",
"tenant": "acme-legal",
"heuristic": "cyclical_tool_thrashing",
"observed": 15,
"threshold": 15,
"spent_usd": 7.20,
"review_url": "https://app.palmsouqltd.com/incidents/inc_8d2f"
}
}Trace player
Watch a loop meet the breaker.
Pick a heuristic and press play. Each line is one request as the breaker engine saw it, with its cost priced from the PALM SOUQ price table at Claude Sonnet 4.5 rates.
The tool thrashing trace is the coding-agent case: each call carries 150,000 input tokens of repository context and 2,000 output tokens, so every repeat costs $0.48.
Rule: same tool + argument hash repeated within one session
- Press play to replay the session through the breaker engine.
- Spent before the breaker
- $0.00
- Status
- idle
Detection
The four runaway heuristics.
Each heuristic has a default threshold you can tune per tier, feature or tenant, and an observe mode that logs without blocking.
| Heuristic | What it watches | Default trigger | Typical loop it catches |
|---|---|---|---|
| Cyclical tool thrashing | Hash of tool name plus normalised arguments, per session. Also catches A-B-A-B oscillation between two argument sets. | 15 identical or oscillating calls | Merge-conflict loops, retrying a failing search, re-reading the same file |
| Token velocity surge | Tokens per second per tenant, summed across every stream, over a rolling window. | > 50,000 tokens/s over 10 s | Fan-out storms, runaway batch jobs, a script hammering the API |
| Semantic echo loop | Short-lived similarity fingerprint of each completion compared with the previous ones in the session. Fingerprints expire with the session. | 3 consecutive turns above 0.92 similarity | Self-reflection spirals, repeated apologies, stuck chain-of-thought |
| Max step / recursion depth | Autonomous turns per session, counting sub-agents spawned under the same root session. | 40 steps (set per tier or feature) | Planner and executor ping-pong, unbounded sub-agent trees |
For a deeper catalog of loop types and how each one shows up in traffic, see the agent loop taxonomy.
The arithmetic
One stuck coding agent, two outcomes.
Without a breaker
$1,200
2,500 calls × (150k input × $3/M + 2k output × $15/M)
= 2,500 × $0.48 = $1,200 in 45 minutes
With PALM SOUQ (tool thrashing at 15)
$7.20
15 calls × $0.48 = $7.20
then HTTP 429, session paused for review
Intervention
Three ways to respond, chosen per policy.
- 01
Block
Immediate HTTP 429 with X-PalmSouq-Reason: circuit_breaker_triggered. Your SDK raises its usual rate-limit error; check the header and stop instead of retrying.
- 02
Reroute
Send the next call to a cheaper fallback model so the agent can finish with a bounded cost, for example Sonnet 4.5 to Haiku 4.5.
- 03
Escalate
Pause the session and page a human through Slack or PagerDuty. Approve resumption from the incident page in one click.
# palmsouq.yaml (breaker policy)
breakers:
default:
tool_thrashing: { repeats: 15, window: session }
token_velocity: { tokens_per_s: 50000, window: 10s }
semantic_echo: { similarity: 0.92, consecutive: 3 }
step_depth: { max_steps: 40, include_subagents: true }
on_trip: block # 429 + X-PalmSouq-Reason
overrides:
- match: { tier: free }
step_depth: { max_steps: 15 }
- match: { feature: research-agent }
on_trip: fallback # reroute to claude-haiku-4.5
fallback_model: claude-haiku-4.5
- match: { tier: enterprise }
on_trip: review # pause session, page on-call
notify: [slack:#agent-incidents, pagerduty:llm-oncall]from openai import RateLimitError
try:
step = agent.run_step(state)
except RateLimitError as err:
reason = err.response.headers.get("X-PalmSouq-Reason")
if reason == "circuit_breaker_triggered":
# Do not retry: the loop is the problem, not capacity.
return finish_with_message(
state,
"This task got stuck and was stopped. Our team has been notified.",
)
raise # genuine provider rate limit: your normal backoff applies“One-line drop-in, no latency we could measure, and the circuit breaker stopped tool thrashing completely.”
Breaker questions
How fast does a breaker trip?
Under 50 ms from the request that crosses the threshold. The heuristics run in the same edge worker that proxies the call, using in-memory counters, so the decision happens before the request is forwarded and the tripping call is never billed by the provider.
How do you detect semantic loops without storing prompts?
The gateway computes a compact similarity fingerprint of each completion while it streams through memory. Only the fingerprint is kept, only for the life of the session, and it cannot be reversed into text. Prompt and completion content is never written to disk.
Will breakers stop legitimate long-running agents?
Thresholds are per tier, feature or tenant, and you can start every heuristic in observe mode, which logs would-have-tripped events without blocking. Most teams run observe mode for a week, look at the distribution, and then enforce.
What happens after a breaker trips?
The breaker opens for that session (not the whole tenant, unless you choose tenant scope). Subsequent calls in the session get the same 429 until a cool-down passes, a human approves resumption in the console, or your code starts a new session.
Is this different from LangGraph's recursion_limit or a max_iterations setting?
Framework limits count steps inside one run, and they are useful. PALM SOUQ also sees across runs, sessions and services, measures money and tokens rather than steps, and works for agents written without a framework.
Put a breaker in front of your agents today.
Change one base URL, keep your provider keys, and every agent session gets four loop heuristics in the request path. Start in observe mode, enforce when you are ready.
14-day free production trial · 100k requests included · no card required