QA Automation Manager

The candidate will have responsibilities across the following functions:

QA Strategy and Leadership:

• Define and drive the overall QA and Quality Engineering strategy for complex products and programs.
• Establish quality goals, automation strategy, test coverage targets, release criteria, and quality metrics.
• Drive shift-left testing and ensure quality is embedded throughout the software development lifecycle.
• Own quality for critical projects and ensure timely delivery without compromising product quality.
• Identify quality risks early and establish mitigation plans.
• Drive continuous improvement in testing processes, tools, and engineering practices.

Test Automation - Playwright:

• Lead the adoption and implementation of Playwright-based automation for UI, API, and end-to-end testing.
• Design scalable, maintainable, and reusable automation frameworks.
• Define automation architecture, coding standards, test design patterns, and best practices.
• Drive migration from legacy automation frameworks such as Selenium to Playwright where applicable.
• Establish automated regression suites and integrate them into CI/CD pipelines.
• Improve automation reliability, execution speed, coverage, and maintainability.
• Review automation code and provide technical guidance to QA engineers.
• Drive automation toward meaningful coverage rather than simply increasing test-case counts.

LLM / AI Testing:

• Lead QA strategy for applications and features powered by Large Language Models (LLMs).
• Define testing approaches for prompt quality, response accuracy, relevance, hallucination, consistency, robustness, safety, guardrails, context handling, regression, and latency.
• Develop automated testing approaches for LLM-based applications.
• Work closely with AI/ML and engineering teams to establish measurable quality standards.
• Evaluate AI features across different models, prompts, configurations, and datasets.

Evals / AI Evaluation:

• Demonstrate strong practical knowledge of LLM Evals and AI evaluation methodologies.
• Define evaluation criteria, datasets, test scenarios, and quality thresholds for AI applications.
• Design and implement automated evaluation pipelines for LLM outputs.
• Work with metrics such as accuracy, relevance, faithfulness, toxicity, safety, and consistency as applicable.
• Establish golden datasets and benchmark test suites for continuous AI evaluation.
• Analyse evaluation results and convert findings into actionable engineering improvements.
• Integrate Evals into CI/CD so AI quality is continuously monitored across releases.
• Establish repeatable processes for evaluating prompt, model, RAG, and application changes.

Critical Project Ownership:

• Take end-to-end ownership of quality for business-critical and high-risk projects.
• Lead QA planning from requirements through production release.
• Build risk-based test strategies for complex projects with multiple dependencies.
• Identify critical business flows and ensure comprehensive E2E validation.
• Lead release readiness reviews and provide quality recommendations to leadership.
• Establish war-room processes for critical releases when required.
• Coordinate cross-functional teams during critical production issues.
• Drive RCA, corrective actions, and preventive actions for major defects.
• Ensure critical projects are delivered with predictable quality and minimal production risk.

QA Process and Engineering Excellence:

• Establish standardised QA processes across teams and projects.
• Define and monitor test strategy, automation coverage, defect leakage, regression effectiveness, production incidents, release quality, automation stability, and test execution efficiency.
• Improve defect prevention and early defect detection.
• Establish quality gates for development and release processes.
• Drive continuous improvement using metrics and data.
• Ensure QA processes scale effectively as products and teams grow.

Team Leadership and Mentoring:

• Lead, mentor, and develop QA engineers and automation engineers.
• Build a high-performing Quality Engineering organisation.
• Establish technical and career development plans for team members.
• Conduct technical reviews and provide feedback on automation and testing practices.
• Encourage adoption of AI-assisted testing and modern QA engineering practices.
• Promote engineering ownership, accountability, and quality culture.

Cross-Functional Collaboration:

• Partner closely with Engineering, Product, DevOps, Data Science, AI/ML, and Business teams.
• Collaborate with architects and developers to ensure testability and observability are built into the product.
• Communicate quality risks and release readiness clearly to senior leadership.
• Represent QA in architecture, design, planning, and release discussions.
• Influence engineering teams to adopt quality-first development practices.

The core requirements for the job include the following:

Automation:

• Strong hands-on experience with Playwright.
• Strong understanding of browser automation and E2E testing.
• Strong programming skills in JavaScript/TypeScript or another suitable language.
• Experience designing automation frameworks from the ground up.
• Strong API automation and integration-testing knowledge.
• Experience with CI/CD tools such as Jenkins, GitHub Actions, or GitLab CI.
• Strong knowledge of Git and software engineering practices.

AI / LLM Testing:

• Strong understanding of LLM applications and AI-based systems.
• Strong practical knowledge of LLM testing and evaluation.
• Understanding of prompts, context windows, embeddings, RAG, agents, and model behaviour.
• Experience designing test datasets and evaluation scenarios.
• Knowledge of automated Evals and continuous AI quality monitoring.
• Understanding of AI-specific failure modes such as hallucination, inconsistency, irrelevant responses, and prompt sensitivity.

Testing Quality:

• Strong understanding of Functional, Integration, API, E2E, Regression, Performance, Risk-based, and Production Validation testing.
• Strong understanding of SDLC, STLC, Agile, Scrum, and CI/CD practices.

Leadership Competencies:

• Strong ownership and accountability.
• Ability to drive projects independently.
• Excellent analytical and problem-solving skills.
• Ability to work effectively under tight deadlines.
• Strong stakeholder management skills.
• Ability to challenge decisions constructively.
• Strong communication and presentation skills.
• Ability to manage ambiguity and rapidly changing requirements.
• Strong decision-making capability during critical releases.
• Ability to balance speed, risk, quality, and business priorities.

Experience:

• 15+ years of experience in Software Quality Assurance / Quality Engineering, with significant experience in automation.
• 5+ years of experience in QA leadership/management or technical leadership.
• Strong experience with modern automation frameworks, preferably Playwright.
• Experience owning quality for complex, enterprise-scale applications.
• Experience working with AI/ML or LLM-based applications is highly preferred.
• Practical knowledge of LLM Evals / AI evaluation is strongly preferred.
• Experience managing multiple teams/projects and driving critical releases.

Preferred Qualifications:

• Experience migrating legacy automation frameworks to Playwright.
• Experience building automation frameworks and QA platforms.
• Experience with AI-assisted testing and test-case generation.
• Experience with RAG, AI agents, or conversational AI testing.
• Experience implementing automated Evals in CI/CD.
• Experience with cloud platforms such as AWS, Azure, or GCP.
• Experience with observability, quality metrics, and production monitoring.