ML Evaluation Engineer

We are looking for an ML Evaluation Engineer to own model quality, regression testing, release validation, and production impact analysis for clinical AI systems. This role sits between applied ML, clinical data, MLOps, and production operations. Your job is to ensure that every model or workflow release is measurable, stable, and does not degrade important clinical behaviour. You will maintain evaluation datasets, create hidden test sets, run regression checks, analyse production issues, and produce release-readiness reports.

Responsibilities:

• Build and maintain evaluation frameworks for clinical NLP, LLM, RAG, information extraction, and structured abstraction systems.
• Create and manage hidden test datasets that are not directly visible to model developers, reducing overfitting and evaluation leakage risk.
• Define release metrics, regression thresholds, slice-based evaluation, failure-mode tracking, and release/blocker criteria.
• Compare model versions and identify performance degradation across clinical segments, document types, clients, data sources, labels, and edge cases.
• Work with Clinical AI Data Specialists to design gold sets, hidden test sets, adjudication workflows, and label quality checks.
• Work with Research Engineers to understand model changes, expected behaviour, and evaluation risks without compromising test-set independence.
• Work with MLOps/Data Engineering to monitor production behaviour, triage bugs, analyse incident impact, and prioritise fixes.
• Create release-readiness reports before production deployment.
• Build dashboards, scripts, and automated checks for evaluation, monitoring, regression testing, and model comparison.
• Prioritise model bugs based on clinical severity, user impact, frequency, regression risk, and operational urgency.

Requirements:

• 3-6+ years of experience in ML engineering, data science, model evaluation, ML QA, applied NLP evaluation, or data-heavy quality engineering.
• Strong Python and data analysis skills.
• Strong understanding of precision, recall, F1 calibration, confidence thresholds, dataset splits, data leakage, overfitting, statistical testing, and error analysis.
• Experience building evaluation pipelines, benchmark suites, test harnesses, dashboards, or regression frameworks.
• Ability to work with imperfect labels, annotation disagreement, clinical ambiguity, and hidden evaluation sets.
• Strong independence and judgment, with the ability to challenge releases when evaluation evidence is weak.
• Clear written communication skills for release reports, incident analysis, and quality decisions.
• Experience with LLM evaluation, RAG evaluation, extraction evaluation, clinical NLP, or healthcare ML.
• Experience with model monitoring, production incident analysis, data drift, or observability.
• Experience with MLflow, Weights and Biases, Evidently, Great Expectations, DeepEval, Ragas, pytest, Airflow, Prefect, or similar tools.
• Clinical or biomedical NLP exposure.