About the role
Responsibilities:
- Design, build and own production-grade LLM agents used in real healthcare workflows.
- Build the backend services, APIs and tools that agents interact with.
- Design agent workflows including tool calling, orchestration, context, memory, guardrails and failure handling.
- Build rigorous evaluation systems using golden datasets, deterministic checks, LLM-as-judge, human review and trajectory evaluation.
- Define and track measurable quality metrics such as precision/recall/F1, task success, tool-call accuracy and regression rates.
- Build eval-gated release processes including automated regression suites, shadow/canary deployments and rollback mechanisms.
- Instrument and monitor agents in production across quality, reliability, latency and cost.
- Debug failures across prompts, models, tools and underlying backend services.
- Make pragmatic decisions around model selection based on accuracy, cost and latency.
- Know when an agent is the right solution and when deterministic software is better
Requirements:
- 6+ years of software engineering experience, with strong ownership of production backend systems.
- 1+ year of experience building LLM agents in production with real users.
- Experience owning at least one agent end-to-end design, prompts, tools, evals, deployment and post-launch improvement.
- Strong backend engineering and distributed systems fundamentals.
- Hands-on experience evaluating AI systems against real datasets and clearly defined metrics.
- Strong understanding of APIs, production architecture, reliability and observability.
- Ability to reason about model choice, prompting, tool design, context and failure modes.
- Strong ownership and ability to work across engineering, product and domain teams.