About the role
The core responsibilities for the job include the following:
Inference Optimization:
- Drive TTFT below 400ms for multi-step agent pipelines.
- Streaming optimization: the first token to the user while sub-agents are still running.
- KV cache strategy, prompt compression, dynamic context window management.
- Multi-provider routing: model selection by latency, cost, and task type across OpenAI, Anthropic, Gemini, and open-weight models.
Agent Architecture:
- Design and implement Plan-Execute-Synthesize pipelines that run sub-agents in parallel DAGs, not sequential chains.
- Build reliable orchestration on top of Temporal: retries, timeouts, partial failure recovery, idempotency.
- Structured output enforcement: JSON schema validation, retry loops on malformed LLM output, graceful degradation.
- Tool call design: schema design that LLMs actually follow reliably across providers.
Evaluation and Harness:
- Own the evaluation framework end-to-end: ground truth datasets, automated scoring pipelines, and regression detection on every PR.
- LLM-as-judge pipelines for qualitative output assessment.
- Latency regression testing - p50/p95/p99 tracked across every deployment.
- Adversarial test case design: ambiguous queries, missing data, conflicting sources, and malformed tool responses.
Infrastructure:
- Model serving and cold start optimization.
- Async worker architecture for parallel sub-agent execution.
- Observability: trace every token, every tool call, every synthesis step.
Requirements:
- You've built something that runs in production at a meaningful scale and you understand why it's fast (or why it isn't).
- Our stack: Go + Temporal for orchestration, a Plan-Execute-Synthesize agent architecture, and an evaluation harness we use to measure every regression. The problems are hard. The latency bar is aggressive. The accuracy requirements are unforgiving.
- Stack familiarity (we care more about depth than match): Go, Python, Temporal, Kafka, PostgreSQL, and Docker.
Strong signal:
- You've worked on inference pipelines where TTFT was the primary metric and you moved it meaningfully.
- You've built multi-step agent systems and you know where they break, not from reading papers but from watching them fail in production.
- You've written eval harnesses from scratch and you have opinions about what makes a ground truth dataset actually useful.
- You've debugged LLM non-determinism in production and built systems resilient to it.
- You've worked with streaming LLM responses and built infrastructure around partial output handling.
Weaker signal (but not disqualifying):
- You've fine-tuned models but haven't shipped inference systems.
- You've used LangChain/LlamaIndex but haven't built the layer underneath.
- Strong ML research background without systems exposure.