About the role
Responsibilities:
- Own day-to-day operations for multiple AI services (LLM apps, ML models, RAG services, agents), meeting SLO/SLA targets for availability, latency, cost, and quality.
- Maintain operational documentation (runbooks, escalation matrices) and participate in the on-call schedule.
- Coordinate incident response including triage, mitigation, and root cause analysis.
- Manage releases and deployments (including canary deployments and blue/green rollouts) and make sure test and production environments remain aligned.
- Follow the company's formal change management process and maintain required documentation for audits.
- Design and maintain full observability across AI systems covering both service health metrics and model quality indicators.
- Track and act on KPIs such as latency, throughput, error rates, safety filter triggers, factual accuracy issues, user acceptance rates, and operational cost per 1K tokens.
- Monitor data and model drift and take action when behaviour changes.
- Build dashboards and alerts; adjust thresholds based on trends.
- Run A/B tests and use controlled side-by-side evaluations to validate improvements.
- Operate and improve continuous integration and continuous delivery workflows for machine learning and large language model components, including pipelines, model registries, and version control.
- Trigger retraining or prompt updates when quality drops, new data becomes available, or policies change.
- Coordinate any required data labelling tasks.
- Manage rollbacks and controlled promotions between models to ensure reliability and auditability.
- Maintain full traceability for decisions, model versions, and deployments.
- Ensure compliance with data protection rules, regional data storage requirements, and secure secret handling.
- Integrate AI-safety measures such as content filtering and misuse detection.
- Maintain documentation describing how each model works, how it was trained, and any risks or limitations.
- Collaborate with Security, Risk, and Governance teams to provide required evidence for audits and reviews.
- Work closely with testing teams on release preparation, environment planning, and resilience testing.
- Contribute AI-specific checks such as prompt quality regression tests, bias/safety evaluations, and retrieval quality assessments.
- Collaborate with data engineering, application engineering, and AI-engineering teams to optimise operations and control costs.
- Help build shared components, templates, and best practices that other AI teams can reuse.
- Ensure all operational and risk-related documentation is kept up to date.
Requirements:
- Experience in Site Reliability Engineering, DevOps, Test Operations, Platform Engineering, or MLOps ideally with responsibility for production systems.
- Hands-on experience with container orchestration platforms such as OpenShift or Kubernetes; CI/CD tools; and observability solutions (metrics, logs, traces, dashboards).
- Practical experience operating machine learning or large language model workloads: serving models, running pipelines, maintaining model registries, and evaluating performance.
- Strong understanding of evaluating ML/LLM quality and safety, detecting drift, running A/B tests, and managing controlled deployments.
- Proficiency in Python and familiarity with one ML platform (MLflow, Vertex AI, Kubeflow, etc. ).
- Ability to communicate clearly and produce high-quality documentation (runbooks, change requests, incident reports).
- Experience with retrieval-based AI systems (vector databases, retrieval pipelines), prompt engineering, or multi-step AI agents.
- Background in AI security or risk assessments (safety testing, privacy, auditability).
- Knowledge of enterprise change management processes and business continuity testing.