Connecting Groq
Groq exposes an OpenAI-compatible endpoint, so the integration is a base URL change to gateway.palmsouqltd.com/groq/openai/v1 plus the PALM SOUQ headers. Model names such as llama-3.3-70b-versatile pass straight through, and so do tool calling and JSON mode.
The gateway connects to Groq over pooled connections from the same edge region as your application, which keeps the added latency inside the 5 ms budget. That matters more for Groq than for most providers, because teams choose Groq for time to first token.
Speed changes the loop math
A fast model makes a runaway loop faster too. An agent that would make 30 calls a minute against a slower frontier model can make several hundred against a high-throughput endpoint. The cost per call is low, but the number of calls in a ten-minute incident is high, and Groq's own organisation rate limits can be exhausted for every tenant at once.
PALM SOUQ's token velocity heuristic watches tokens per second per tenant over a rolling window (50,000 tokens per second over 10 seconds by default) and the cyclical tool heuristic counts repeated tool calls. Both trip on the call that crosses the threshold, not on a dashboard refresh minutes later.
Groq as a fallback target
Groq is often used as the second link in a fallback chain: a tenant on a frontier model is moved to Llama 3.3 70B on Groq when its budget runs low, so answers stay fast while cost per call drops. PALM SOUQ handles the switch in the gateway and records which model actually served each request, so per-tenant cost reports are correct after the downgrade.
Groq does not report cached input tokens, so PALM SOUQ prices all input at the standard rate and marks cache telemetry as not applicable for this provider instead of guessing.
Rolling it out safely
Because Groq is chosen for speed, measure it. Run a fixed set of prompts directly and through the gateway, from the same server, and compare time to first token. The difference should sit inside the gateway's 5 ms budget; if it does not, the console shows which PoP served the traffic so the route can be checked.
Then set a per-tenant tokens-per-second limit below your organisation's Groq limit, so one tenant's burst returns a PALM SOUQ 429 to that tenant only, instead of a provider 429 to everyone.
