About the role
Every request that a PALM SOUQ customer sends to OpenAI, Anthropic, Gemini or Bedrock passes through a Rust process that you will help own. That process parses headers such as X-PalmSouq-Tenant, resolves the tenant's budget and breaker policy, forwards the request upstream, counts tokens on the streamed response as it flows back, and decides in under 50 ms whether an agent loop needs to be stopped with an HTTP 429. It does all of that while adding less than 5 ms of latency and without ever writing a prompt or completion to disk.
This is a role for an engineer who enjoys working against a stopwatch. The gateway runs across 280+ edge points of presence and our availability target is 99.999%, which leaves about five minutes of downtime a year. Small decisions, such as how a buffer is allocated for a server-sent events chunk or how a policy cache is invalidated, show up directly in customer latency graphs.
What you will do
- Design and ship features in the streaming proxy: SSE and chunked transfer handling for every supported provider, tool-call parsing, and incremental token counting on partial responses.
- Extend the breaker engine that evaluates the four loop heuristics (cyclical tool thrashing, token velocity surge, semantic echo loops, max step depth) without blocking the stream.
- Profile and remove latency: allocation audits, lock contention, tail latency at p99.9, and regression gates in CI that fail a build when p99 moves.
- Keep Zero Payload Retention true in code: review every new log line, metric label and crash handler so request bodies never reach disk or telemetry.
- Take part in a follow-the-sun on-call rotation with a documented runbook and a blameless review culture.
What we are looking for
- 5+ years building production systems, with at least 2 years of production Rust (async, tokio or similar).
- Hands-on experience with HTTP/1.1, HTTP/2, TLS and streaming protocols at the byte level.
- Comfort with performance tooling: flamegraphs, perf, heap profilers, and reading the numbers honestly.
- Clear written communication. Design docs here are short, specific and argued with data.
Nice to have
- Experience with proxies such as Envoy, Pingora, HAProxy or a CDN edge runtime.
- Familiarity with tokenizer internals (tiktoken, SentencePiece) or LLM provider streaming formats.
- Contributions to open-source Rust networking crates.
If you meet most of the first list and none of the second, please apply anyway.
Your first 90 days
Days 1 to 30
Ship a change to the streaming parser for one provider, shadow the on-call rotation, and read every breaker incident review from the last year.
Days 31 to 60
Own one latency improvement end to end, from flamegraph to production rollout behind a percentage flag.
Days 61 to 90
Write and land a design doc for a gateway capability on the current roadmap, such as per-tool budget limits.
Interview process
| # | Stage | What happens |
|---|---|---|
| 1 | Intro call (30 min) | With the hiring manager, Meera Krishnamurthy. Your background, what you want next, and our gateway architecture. |
| 2 | Systems conversation (75 min) | Design a streaming proxy that counts tokens without buffering the full response. No whiteboard tricks, just trade-offs. |
| 3 | Paid take-home or pairing (half day) | Your choice: a small paid Rust exercise on a streaming parser, or a pairing session on real gateway code. |
| 4 | Team and values (45 min) | Meet two engineers you would work with and talk through how you handle on-call, reviews and disagreement. |
Need an adjustment for any stage, such as extra time or a different format? Tell us in your application or at careers@palmsouqltd.com.