Blog
Notes on governing LLM spend
Incident teardowns, pricing math and gateway engineering from the people who build PALM SOUQ. Every number is shown with its arithmetic.
All posts
12 of 12 posts
- What 25 power users do to a $49,500 MRR support SaaSA worked example: 500 customers at $99, 25 heavy users on GPT-4o, and how a per-tenant cap with a GPT-4o mini fallback recovers $9,125 a month.Ananya IyerFinOps3 min
- Under 5 ms: the latency budget of an in-path LLM gatewayWhere the time goes when a request passes through the PALM SOUQ gateway, which work happens before the first byte and which happens after, and how we keep it under 5 ms.Rohan MalhotraEngineering3 min
- Per-tenant rate limits are not per-tenant budgetsA tokens-per-minute limit caps speed, not spend. The same limit can cost fifteen times more on one model than another. How to combine rate limits with dollar budgets.Vineet SaresarEngineering3 min
- Introducing the PALM SOUQ LLM Price IndexA free, citable table of token and prompt cache prices across foundation model providers, with cache write and read rates side by side and a CC BY 4.0 license.Ananya IyerAnnouncements3 min
- Graceful degradation beats a hard 429: soft caps and model fallbacksCutting a customer off at their budget is the worst moment to say no. How soft caps, warnings and a Sonnet to Haiku fallback keep the product working and the margin intact.Nisha PillaiProduct3 min
- Zero Payload Retention: what we keep, what we hash, and what never touches diskA field-by-field account of how prompts and completions stream through the PALM SOUQ gateway without being written to disk, and what metadata we do store.Kabir SethiSecurity3 min
- Fail-open by default: what the PALM SOUQ SDK does when the gateway is unreachableWhy our SDKs default to fail-open, what you give up, when fail-closed is the right call, and how to choose per feature rather than per company.Rohan MalhotraEngineering3 min
- Designing the four loop heuristics: thresholds, false positives and defaultsHow PALM SOUQ detects tool thrashing, token velocity surges, semantic echo and runaway recursion in the request path, and how we chose the default thresholds.Meera KrishnamurthyEngineering4 min
- Cache hit rate is a tenant property, not a model propertyTwo tenants on the same model and prompt can see very different prompt cache economics. Why averaging cache savings across your account misprices both of them.Ananya IyerFinOps3 min
- Anatomy of a runaway coding agent: 2,500 calls on one merge conflictA step-by-step teardown of a coding agent stuck resolving a git merge conflict, what each call cost, and where each loop heuristic would have fired.Nisha PillaiIncidents3 min
- Why post-facto cost alerts arrive after the invoiceBilling exports and dashboard alerts measure spend that already happened. Here is the timing math on a runaway agent and why the control has to sit in the request path.Meera KrishnamurthyEngineering3 min
- Pricing AI features: seats, credits or overages, and what each does to marginA flat seat price hides a heavy tail of LLM cost. We compare seat, credit and allowance-plus-overage pricing with real token math on Claude Sonnet 4.5.Vineet SaresarPricing4 min
Who writes here
Written by the engineers and analysts who run the gateway
No ghostwriters. Posts are reviewed by engineering for accuracy and by FinOps research for every price, which comes from the LLM Price Index.
- Ananya IyerHead of FinOps Research3 posts
- Vineet SaresarFounder & CEO2 posts
- Meera KrishnamurthyCTO2 posts
- Rohan MalhotraVP Engineering2 posts
- Nisha PillaiHead of Customer Engineering2 posts
- Kabir SethiHead of Security & Compliance1 post
The Token Governor, weekly
New posts, price changes from the index and one worked margin example per issue. No filler.
Don't wait for a surprise $10,000 OpenAI invoice.
Route your agent traffic through PALM SOUQ in 2 minutes. Change one base URL, pass a tenant header, and every token is attributed and governed.
14-day free production trial · 100k requests included · no card required