Once traffic runs through more than one provider, someone will ask for a single view of usage. Adding up reported token counts looks like the answer and is not, for two reasons: providers report token classes in different shapes, and they count tokens with different tokenizers. This guide covers both and recommends normalizing on cost while keeping token classes for diagnosis.
Different shapes for the same classes
| Provider | Input field | Cached input | Output field | Reasoning |
|---|---|---|---|---|
| OpenAI | prompt_tokens, includes cached | prompt_tokens_details.cached_tokens | completion_tokens, includes reasoning | completion_tokens_details.reasoning_tokens |
| Anthropic | input_tokens, excludes cache | cache_read_input_tokens; writes in cache_creation_input_tokens | output_tokens | Counted within output_tokens |
| Gemini | promptTokenCount, includes cached | cachedContentTokenCount | candidatesTokenCount | thoughtsTokenCount |
| Bedrock Converse | inputTokens | cacheReadInputTokens; writes in cacheWriteInputTokens | outputTokens | Model dependent |
The traps
- Double counting: adding OpenAI cached_tokens to prompt_tokens counts the cached tokens twice, because prompt_tokens already includes them.
- Undercounting: treating Anthropic input_tokens as total input misses the cache read and write tokens, which are reported separately.
- Missing reasoning: summing only visible output on Gemini misses thought tokens, which are billed.
- Mixed tokenizers: the same text produces different token counts on different providers, so a token on one is not the same amount of text as a token on another.
A normalized schema
Map every response into five classes: uncached input, cache read, cache write, output (visible) and reasoning. For OpenAI, uncached input is prompt_tokens minus cached_tokens, and visible output is completion_tokens minus reasoning_tokens. For Anthropic, the classes map directly. For Gemini, uncached input is promptTokenCount minus cachedContentTokenCount, and reasoning comes from thoughtsTokenCount. Store the original usage block alongside, so a mapping bug can be fixed and replayed.
def normalize_openai(u):
cached = (u.get("prompt_tokens_details") or {}).get("cached_tokens", 0)
reasoning = (u.get("completion_tokens_details") or {}).get("reasoning_tokens", 0)
return {
"input_uncached": u["prompt_tokens"] - cached,
"cache_read": cached,
"cache_write": 0,
"output": u["completion_tokens"] - reasoning,
"reasoning": reasoning,
}
def normalize_anthropic(u):
return {
"input_uncached": u["input_tokens"],
"cache_read": u.get("cache_read_input_tokens", 0),
"cache_write": u.get("cache_creation_input_tokens", 0),
"output": u["output_tokens"],
"reasoning": 0, # thinking is billed inside output_tokens
}Normalize on cost, compare on cost
Because tokenizers differ, the comparable unit across providers is cost, not tokens. Price each class at its provider rate and sum. For a request with 10,000 input tokens (6,000 of them cached) and 800 output tokens, list prices give these costs:
| Model | No cache | 6,000 cached |
|---|---|---|
| GPT-4.1 | $0.0264 | $0.0174 |
| Claude Sonnet 4.5 | $0.0420 | $0.0258 |
| Gemini 2.5 Pro | $0.0205 | $0.0149 |
Non-token units
Not everything is billed in tokens. Embedding models bill input tokens only. Image inputs are converted to tokens by rules that differ by provider and by image size. Some hosted open-weight providers bill per second of compute for dedicated deployments rather than per token. Give these their own usage classes rather than forcing them into the five token classes, and price them the same way: at the provider rate, at the time of the request, per tenant. Cost remains the common denominator.
Reconcile per provider, report across providers
Normalization makes cross-provider reporting possible, but reconciliation still happens per provider, because each provider invoices separately and rounds in its own way. Each month, compare the normalized cost for each provider with that provider's invoice. If one provider drifts while the others tie out, the mapping for that provider is the first suspect, usually a new usage field after an API update or a token class that was not mapped.
Keep a short changelog of mapping changes with dates. When a reconciliation gap appears, the changelog usually explains it within minutes.
Limits across providers
The same reasoning applies to limits. A per-tenant token limit means different things on different providers; a per-tenant dollar budget means the same thing everywhere. When a fallback chain crosses providers, only a cost budget stays meaningful through the switch. Keep token rate limits for burst control per provider, and use cost for anything that represents money.
How PALM SOUQ does it
The gateway normalizes every provider response into the five classes, prices each at the serving model's rate, and stores both the normalized record and the original usage block. Reports, quotas and billing events all read the normalized cost, so a tenant that moves between providers or falls back across them keeps one consistent ledger.
When a provider changes its usage format, the mapping is updated in one place and historical records can be reprocessed from the stored originals.
Rohan Malhotra
VP Engineering, PALM SOUQ
NIT Trichy. Previously led streaming data infrastructure at Flipkart. Responsible for the token accounting pipeline and the 99.999% availability target.