Designing the four loop heuristics: thresholds, false positives and defaults

A circuit breaker for agents has two jobs that pull against each other. It must stop a loop quickly, because every extra call costs money. It must also leave legitimate long-running work alone, because a breaker that trips on real work gets turned off within a week. This post explains how we approached that tradeoff for each of the four heuristics PALM SOUQ ships.

Constraints from the request path

Every heuristic runs inline, on the request, at the edge point of presence that received it. The gateway budget for all of PALM SOUQ's work is under 5 ms of added latency, so a heuristic cannot call an external model, cannot scan full transcripts, and cannot wait for a batch job. Each one keeps a compact rolling state keyed by session: counters, a few hashes, a short window of token counts.

Sessions are identified by the agent framework's own session or run ID when one is present, and otherwise by the tenant plus a conversation fingerprint. Zero Payload Retention applies here too: we store hashes and counts in memory, never prompt or completion text.

1. Cyclical tool thrashing

The signature of the most expensive loops we see is the same tool called with the same or near-same arguments, over and over, with the same failure. We hash the tool name plus a normalized form of the arguments (whitespace collapsed, volatile fields such as timestamps and request IDs removed) and count repeats inside a sliding window.

The default is 15 identical calls within 600 seconds. We picked 15 because legitimate polling patterns in our metered traffic, such as an agent checking a CI job status, rarely exceed 10 polls before the agent moves on, and because at frontier-model prices 15 calls is still a small bill. Teams with known polling tools add them to an allow list instead of raising the global limit.

2. Token velocity surge

Some loops do not repeat a tool at all. A planner that fans out to many sub-agents, each of which reads a large document, can burn through tokens without any single call looking unusual. Velocity catches these by measuring tokens per second per session over a short window.

The default threshold is 50k tokens per second sustained over a 10-second window. A human-driven chat session never approaches that. A batch job might, which is why velocity limits are set per tier and per feature: a nightly summarization feature tagged with X-PalmSouq-Feature can carry a higher limit than the interactive assistant.

3. Semantic echo loops

In a semantic echo the model keeps producing almost the same answer in slightly different words, often when it is stuck between two instructions. Exact hashing misses this. We compute a lightweight locality-sensitive fingerprint of each assistant turn in memory and compare it with the previous turns in the session. When consecutive turns stay above a similarity threshold for several turns, the heuristic trips.

This is the heuristic most likely to produce false positives on legitimate work, for example a form-filling agent that writes very similar records. It ships enabled with a conservative threshold, and every trip includes the similarity scores in the incident webhook so you can tune it with evidence.

4. Max step and recursion depth

The simplest heuristic is a ceiling on steps per session and on nesting depth for agents that spawn sub-agents. It has no cleverness and it never misses. Its weakness is that the right number depends entirely on the agent, so we do not ship a universal default. You set it per agent or per feature, ideally from the 99th percentile of step counts in your first two weeks of metering.

What happens on a trip

The gateway refuses the next request with HTTP 429 and X-PalmSouq-Reason: circuit_breaker_triggered, plus a header naming the heuristic. We chose 429 deliberately: every agent framework already treats it as a recoverable limit rather than a crash, which gives the application a clean place to show the user a message. Your SDK integration should treat this specific reason as terminal for the session, not as a signal to retry.

Each trip also emits an incident event with the tenant, session, heuristic, counters and cost so far, to Slack, a webhook, or your OpenTelemetry pipeline.

Tuning in practice

• Start in observe mode for a week: heuristics evaluate and log would-trip events without refusing requests.
• Review would-trip events per heuristic. Every false positive usually points to one tool or one feature that needs its own rule.
• Switch to enforce mode tenant tier by tier, starting with free and trial tenants where the cost of a false positive is lowest.