Glossary
LLM FinOps terms, defined plainly
45 terms for engineers and finance teams who share one model bill: tokens, caching, rate limits, breakers and margin.
{
"tenant": "northwind-ai",
"tier": "pro",
"feature": "support-agent",
"model": "claude-sonnet-4.5",
"usage": {
"input_tokens": 48210,
"cache_read_tokens": 41000,
"output_tokens": 1184
},
"cost_usd": 0.05169,
"breakers": {
"token_velocity": "ok",
"tool_thrashing": "ok",
"max_steps": "7/25"
},
"budget": {
"month_to_date_pct": 64,
"soft_cap_pct": 80
},
"payload_stored": false
}45 of 45 terms
A
- Agent loop
A run in which an autonomous agent repeats steps without making progress, such as calling the same tool with the same arguments. Each step is a billed model call, so loops turn directly into cost.
- AI gateway
A proxy between your application and model providers that handles routing, authentication, rate limits, metering and policy enforcement for every request.
- Anomaly detection
Flagging a tenant whose spend or token rate departs sharply from its own recent baseline, such as a rolling seven-day average.
B
- Batch API
An asynchronous endpoint where requests are processed within a window (often 24 hours) at a discount to real-time prices. Suited to evaluations, backfills and enrichment jobs.
- Blended price
A single per-token price that weights input and output by a traffic shape. At 3:1, blended = (3 x input price + output price) / 4.
- Budget cap
A limit on spend for a tenant, tier or feature over a period. A soft cap warns; a hard cap blocks or downgrades.
C
- Cache hit rate
The share of cacheable prompt tokens served from the provider's prompt cache rather than processed fresh. It decides whether caching saves money.
- Cache read
A prompt token served from the provider's cache, billed at a discounted cached-input rate.
- Cache write
The first processing of a prompt prefix that will be cached. Some providers, notably Anthropic, bill writes at a premium over the normal input rate.
- Circuit breaker
A control in the request path that stops traffic when a rule trips, returning an error instead of forwarding the call. In PALM SOUQ it answers with HTTP 429 and X-PalmSouq-Reason: circuit_breaker_triggered.
- COGS
Cost of goods sold. For an AI SaaS product, model API spend is usually the largest variable component of COGS per customer.
- Context window
The maximum number of tokens a model can consider in one request, counting prompt and output. Larger windows allow longer prompts, which also cost more per call.
- Cost attribution
Assigning the cost of each model call to a tenant, user, feature or plan so spend can be compared with revenue.
D
- Depth cap
A limit on how many levels of sub-agents an orchestrator may spawn. Stops recursive fan-out.
F
- Fail-open
A design where, if the gateway cannot be reached, the SDK sends traffic directly to the provider so the application keeps working.
- Fallback chain
An ordered list of models to route to when a tenant nears its budget or a provider fails, such as Opus to Sonnet to Haiku.
- FinOps
The practice of managing cloud and AI spend with shared accountability between engineering and finance, using unit costs and budgets.
G
- Gross margin
Revenue minus COGS, as a percentage of revenue. Unmetered LLM usage on flat-rate plans is the most common reason AI SaaS gross margins fall.
H
- Hard cap
A limit that blocks or downgrades requests once reached. Returns HTTP 429 or reroutes to a cheaper model.
I
- Idempotency key
A unique ID sent with a request so retries of the same logical operation can be recognized and not double counted or double executed.
- Input tokens
Tokens in the prompt sent to the model, including system instructions, tool definitions, retrieved documents and conversation history.
L
- Long-context pricing
A higher per-token rate some providers charge once a prompt exceeds a length threshold, such as 200K tokens.
M
- Max step cap
A ceiling on the number of model calls an agent may make in one session or task.
- Metered billing
Charging customers based on measured usage, such as tokens or requests, typically by syncing usage events to a billing system.
- Model routing
Choosing which model serves a request based on rules such as task type, tenant tier, budget remaining or provider health.
O
- Output tokens
Tokens the model generates, including hidden reasoning tokens on reasoning models. Usually priced several times higher than input tokens.
- Overage
Usage beyond what a plan includes, billed at a per-unit rate. Common for AI features with token-heavy power users.
P
- Per-tenant metering
Recording token usage and cost separately for each customer account, usually keyed by a tenant header on every request.
- Power user
A customer whose usage is far above the plan average. On flat pricing, a few power users can consume most of a plan's margin.
- Prompt caching
Provider-side reuse of a processed prompt prefix across requests, billed at a reduced rate for cache reads.
R
- Rate limit
A cap on requests or tokens per unit of time. Providers set them per organization; PALM SOUQ adds per-tenant limits in front.
- Reasoning tokens
Tokens a reasoning model generates internally before answering. They are not shown in the response but are billed at the output rate.
- Retry storm
A surge of retried requests during an outage or overload, often from several layers retrying independently, which multiplies cost and load.
- RPM
Requests per minute. A rate-limit dimension set by providers and enforceable per tenant at the gateway.
S
- Semantic echo
A loop in which a model repeats near-identical content across steps or within one output. Detected by comparing fingerprints of outputs.
- SOC 2 Type II
An independent audit report on how a service provider's security controls operated over a period of time.
- Soft cap
A threshold, often 80% of budget, that sends a warning webhook without blocking traffic.
T
- Tenant
A customer account in a multi-tenant SaaS product. In PALM SOUQ, identified on each request by the X-PalmSouq-Tenant header.
- Token
The unit models read and write, roughly a word fragment. Providers price input, output and cached tokens per million.
- Token velocity
The rate at which a tenant consumes tokens, measured over a rolling window. PALM SOUQ's default breaker trips above 50k tokens per second over 10 seconds.
- Tool thrashing
An agent repeatedly calling the same tool with identical or oscillating arguments without progress.
- TPM
Tokens per minute. The main throughput limit providers apply per model and organization.
U
- Unit economics
Revenue and cost per unit, such as per customer, per seat or per request. For AI products, model cost per active customer is the key input.
- Usage tier
A provider's account level, usually based on cumulative spend, that determines rate limits.
Z
- Zero Payload Retention
PALM SOUQ's architecture in which prompts and completions stream through memory and are never written to disk; only token counts, costs and metadata are stored.
Don't wait for a surprise $10,000 OpenAI invoice.
Route your agent traffic through PALM SOUQ in 2 minutes. Change one base URL, pass a tenant header, and every token is attributed and governed.
14-day free production trial · 100k requests included · no card required