About the role
A hard cap on a tenant's monthly LLM budget is only useful if every edge location agrees, within milliseconds, on how much of that budget is left. PALM SOUQ runs regional control planes in US East, US West, EU Central, India South and Singapore, and edge nodes that must make admission decisions locally without a round trip on the request path. Keeping those counters accurate enough to stop a runaway agent, while staying available when a region degrades, is the core distributed systems problem at PALM SOUQ.
As a Staff engineer you will own that problem across teams. You will decide where we accept bounded overshoot, where we need stronger guarantees, and how the system degrades. You will also be the person who explains those trade-offs plainly to customers' VPs of Engineering during architecture reviews.
What you will do
- Own the architecture of the quota plane: sliding-window counters, CRDT-style budget replication, lease-based local allowances and reconciliation with the metering store.
- Define correctness budgets, for example the maximum overshoot a hard cap may allow during a regional partition, and build the tests and chaos drills that prove them.
- Lead the metering pipeline that turns gateway events into per-tenant cost records, cache read and write accounting, and billing events for Stripe, Metronome, Orb and Lago.
- Mentor senior engineers, review designs across the Gateway and Platform teams, and set the bar for operational readiness.
- Partner with Security on data residency so tenant metadata stays in the region the customer chose.
What we are looking for
- 8+ years building and operating distributed systems in production, with clear ownership of at least one multi-region system.
- Deep working knowledge of consistency models, consensus, replication and failure detection, and the judgement to know when not to use them.
- Experience running high-throughput data pipelines (Kafka, Redpanda, Pulsar or similar) and columnar stores.
- A track record of raising the engineering bar of the people around you.
Nice to have
- Experience with rate limiting or admission control at a CDN, payments network or API platform.
- Formal methods exposure (TLA+, P, or property-based testing at scale).
- Experience with usage-based billing systems and the accounting rules behind them.
If you meet most of the first list and none of the second, please apply anyway.
Your first 90 days
Days 1 to 30
Map the quota plane end to end, run a regional failover drill with the on-call team, and publish a written list of the risks you see.
Days 31 to 60
Propose and get agreement on the overshoot budget for hard caps during partitions, with tests that measure it.
Days 61 to 90
Lead the first milestone of the next-generation counter replication design.
Interview process
| # | Stage | What happens |
|---|---|---|
| 1 | Intro call (30 min) | With Rohan Malhotra, VP Engineering. Scope of the role and what you are looking for. |
| 2 | Architecture deep dive (90 min) | Walk us through a distributed system you owned, including the incident you learned the most from. |
| 3 | Design session (90 min) | Design a global per-tenant budget counter with a 5 ms request-path budget and a regional partition. |
| 4 | Leadership and founders (60 min) | Mentorship, written communication and a conversation with the founder. |
Need an adjustment for any stage, such as extra time or a different format? Tell us in your application or at careers@palmsouqltd.com.