Procedure · 6 steps
- 1Choose the grainAggregate priced model cost per tenant per hour, and keep feature and model as dimensions for drill-down.
- 2Build hour-of-week baselinesFor each tenant and hour of the week, compute the median and median absolute deviation over the last four to eight weeks.
- 3Score each hourCompute a robust z-score for each new hour and flag hours above a threshold, with a minimum dollar amount to avoid noise.
- 4Handle new tenantsUntil a tenant has enough history, compare it with the baseline of its plan tier cohort.
- 5Route alerts by size and ownerSend small anomalies to a daily digest and large ones to the account owner and finance in real time.
- 6Feed findings back into limitsWhen an anomaly reveals a missing control, add a quota, a fallback or a breaker rule so the next one is stopped in the path.
Circuit breakers stop loops in seconds by watching individual sessions. Anomaly detection works on a slower clock: it compares a tenant's spend over hours or days with what is normal for that tenant and flags the difference. It catches what breakers are not designed to see, such as a customer who doubled their usage legitimately, a prompt change that broke caching, or a new integration that sends far more traffic than planned.
Step 1: choose the grain
Hourly cost per tenant is the right default. Finer grains are noisy and duplicate what breakers already do; coarser grains detect too late to matter. Keep feature and model as dimensions so an alert can say which part of the tenant's usage moved.
Step 2: baseline by hour of week
Usage follows the working week. A support tool is busy on Tuesday afternoon and quiet on Sunday night, so a single average per tenant produces false alarms every Monday morning. Build a baseline per tenant per hour of the week from the last four to eight weeks: 168 slots, each with a median and a median absolute deviation (MAD).
Median and MAD are robust: one past incident in the history does not inflate the baseline the way a mean and standard deviation would.
Step 3: score with a robust z-score
For each new hour, compute (observed - median) / (1.4826 x MAD). The constant makes MAD comparable with a standard deviation for normally distributed data. In the snapshot, the tenant's Tuesday 14:00 baseline has a median of $1.20 and a MAD of $0.30. Observed spend of $4.80 gives ($4.80 - $1.20) / (1.4826 x $0.30) = 8.1.
A threshold between 4 and 6 is a sensible start. Add a minimum dollar deviation as well, so a tenant moving from four cents to twenty cents does not page anyone.
Step 4: handle cold starts
New tenants have no history. Until they have two to four weeks of data, compare them with the baseline of their plan tier: the median and MAD across all tenants on the same plan for that hour of the week. This catches the most common new-tenant anomaly, an integration that sends every record through the model on day one, without waiting for a personal baseline to form.
Step 5: route by size and owner
- Below a dollar threshold: daily digest to the platform team.
- Above it: real-time alert to the account owner and to finance, with the tenant, feature, model, baseline and observed values.
- Above a much higher threshold: page on-call, because something may be wrong that breakers did not catch.
- Every alert should link to the tenant's cost view so the reader can see the shape of the change in one click.
Common causes and what to do
| Pattern | Likely cause | Action |
|---|---|---|
| Input cost up, requests flat | Prompt change broke the cache prefix | Check cache hit rate by prompt version |
| Requests up across all features | Legitimate growth or a new integration | Talk to the customer, review the plan |
| One feature up sharply | Feature bug or retry storm | Check error rates and retries |
| Output cost up, requests flat | Model change or reasoning increase | Check served model and reasoning tokens |
Step 6: close the loop
An anomaly is a signal that a control is missing. After each one, ask whether a quota, a fallback, a feature-level velocity limit or a breaker rule would have stopped it in the request path. If yes, add it. Over time the anomaly stream gets quieter, and what remains is genuine change that deserves a human conversation.
Kabir Sethi
Head of Security & Compliance, PALM SOUQ
IIT Delhi. Eight years in security engineering at Infosys and a cloud security startup. Leads the SOC 2 Type II program and Zero Payload Retention design reviews.