Senior Staff QA

Role Overview 

We are looking for a QA Lead who can architect the overall quality strategy for JazzX AI’s platforms and AI-powered products. This role requires a blend of traditional QA expertise, AI/ML awareness, and strong leadership capabilities. 
You will lead the quality roadmap for agent workflows, LLM outputs, APIs, and cloud services while managing a high-performing QA team. 

 Key Responsibilities 

1. Quality Strategy & Leadership

• Build and own JazzX AI’s end-to-end QA strategy across AI copilots, workflows, and enterprise deployments. 

• Lead, mentor, and scale a team of QA engineers and automation specialists. 

• Partner closely with Product, ML Engineering, and Platform Engineering to embed quality in early design phases. 

• Define and enforce quality gates and release-readiness standards. 

1. AI/LLM-Specific Quality Assurance

• Develop evaluation systems for LLM accuracy, hallucination detection, factuality, safety, and bias mitigation. 

• Design scalable tests for agent reasoning, tool-use correctness, and multi-step workflow validation. 

• Create synthetic datasets and scenario-based tests for edge-case prediction and regression testing. 

• Validate model performance across fine-tuning, RAG, knowledge-grounding, and inference scenarios. 

1. Automation Engineering

• Architect and manage automation frameworks using Python/Java/JS, Playwright, Selenium, PyTest, Postman, etc. 

• Ensure seamless integration of automated suites into CI/CD pipelines (GitHub Actions/GitLab/Jenkins). 

• Automated API, microservices, and backend testing for JazzX AI’s cloud architecture running on AWS/GCP. 

1. Performance Testing

• Load Testing, Stress Testing, Spike Testing, Endurance/Soak Testing using industry-standard tools like JMeter, LoadRunner, Gatling, k6, or Locust. 

1. Release, Monitoring & Reliability

• Own the release sign-off process for all AI software components and platform features. 

• Conduct RCA, maintain issues of dashboards, and drive continuous improvement in QA lifecycle. 

• Collaborate with DevOps/ML Ops to monitor model drift, performance anomalies, and post-production behaviors. 

1. Documentation & Communication

• Prepare detailed QA documents: strategies, test plans, evaluation reports, and execution metrics. 

• Provide clear communication of risks, quality insights, and impact analysis to leadership. 

 Required Qualifications 

• 12+ years of experience in Software QA. 

• Strong hands-on experience with test automation frameworks and scripting (Python/Java/JavaScript). 

• Experienced in working with microservice-based architecture 

• Working knowledge of AI/ML systems, including datasets, model evaluation, vector stores, RAG, and LLM operations. 

• Architect and manage automation frameworks using Python/Java/JS, Playwright, Selenium, PyTest, Postman, etc. 

• Automated API, microservices, and backend testing for JazzX AI’s cloud architecture running on Azure /AWS. 

• Experience testing distributed cloud systems, APIs, and high-performance backend services. 

• Strong analytical and problem-solving skills with the ability to drive organization-wide quality initiatives. 

 Preferred Skills 

• Experience testing LLMs, RAG pipelines, or autonomous AI agents. 

• Familiarity with AI observability tools, logging/monitoring stacks (ELK, Prometheus, Datadog). 

• Exposure security testing, performance/load testing, and compliance (SOC2, GDPR). 

• Ability to operate in a fast-paced, zero-to-one startup environment.