About the role
Responsibilities:
- Design and own evaluation systems for agent domains.
- Take architectural responsibility for eval coverage across one or more capability domains: multi-turn conversation evals, tool-selection accuracy, goal completion, policy compliance, and safety boundary probing.
- Coverage spans behavioural equivalence, SLOs, idempotency, chaos-as-test, context and retrieval quality, and tool/dependency failure modes; each is a distinct failure surface that functional correctness evals alone do not cover.
- Define what "evaluated" means for the domain, not just execute a checklist.
Requirements:
- 5+ years of software engineering experience, with at least 2 years building evaluation, measurement, or quality systems for AI/ML products.
- Engineering-parallel fundamentals: you can design, build, and operate evaluation infrastructure as production-grade distributed systems (services, pipelines, storage, observability), not scripts and spreadsheets.
- Deep hands-on LLM evaluation experience: LLM-as-judge design and calibration, human annotation pipelines, golden dataset governance, and systematic error analysis from raw traces.
- Demonstrated ability to define acceptance criteria for nondeterministic behaviour, turning fuzzy notions of quality into thresholds an organisation can operate against.
- Experience diagnosing agentic systems across the full execution path: able to attribute a failure to prompt, context, tooling, or model capability, and defend the attribution with evidence.
- Experience wiring evaluation into deployment flows as a hard gate, blocking bad changes before they reach production rather than reporting on them after.
- Statistical fluency applied to evaluation: sample size design, confidence intervals, and significance testing, enough to defend a regression call when a team disputes it.
- Proficiency in Python and/or Java at the level needed to build and maintain a shared eval framework others depend on.
- Experience operating in environments where AI failures have real business or compliance consequences.