Senior Software Development Test Engineer

The core responsibilities for the job include the following:

Generative AI and LLM Evaluation:

• Build automated testing suites to detect hallucinations, bias, toxicity, and prompt injection vulnerabilities across LLM-powered products
• Implement automated evaluations for RAG systems measuring context relevance, groundedness, and answer faithfulness using frameworks like RAGAS or DeepEval
• Design test beds to validate multi-agent workflows: tool-calling accuracy, multi-step reasoning, memory, and autonomous decision loops
• Build and run automated conversation simulations: scripted and synthetic user journeys to stress-test agent behavior across intents, edge cases, and multi-turn dialog flows
• Create prompt regression frameworks to assess how changes in system prompts, temperature, and sampling parameters impact output consistency

Data Quality Assurance:

• Statistically validate AI data outputs: distributions, precision/recall, and error pattern analysis to catch silent data quality failures before production.
• Programmatically audit data ingestion, transformation, and feature store pipelines for schema drift and data corruption.
• Validate vector DB indexing, embedding semantic similarity accuracy, and retrieval latency.
• Verify quality, diversity, and privacy compliance of synthetic datasets used for model training and evaluation.

Classical ML and Deep Learning Validation:

• Maintain automated suites tracking ML metrics: precision, recall, F1, ROC-AUC, and deep learning loss curves across model versions.
• Implement continuous monitoring scripts to detect data and concept drift on live inference endpoints.

Automation Engineering and CI/CD:

• Build and maintain scalable test automation frameworks for APIs, backend services, and model endpoints.
• Embed AI evaluation and data QA suites into MLOps and CI/CD pipelines so quality failures block releases automatically.
• Define and track AI quality KPIs and communicate release readiness to engineering and product teams.

Requirements:

• Experience of 5+ years.

Core Programming:

• Python: expert level; test automation, eval pipelines, data analysis (Pandas, NumPy, Pytest).
• SQL: data output validation, ground truth querying, and pipeline data quality checks.

GenAI and Evaluation:

• RAGAS / TruLens / DeepEval / Promptflow, etc. LLM evaluation frameworks for measuring faithfulness, hallucination rate, and task success.
• LangChain / LangSmith / LlamaIndex: agent workflow testing, prompt tracing, and LLM response debugging.
• OpenAI / Anthropic / Hugging Face APIs: direct LLM endpoint testing and output consistency validation.
• Vector DBs: retrieval quality testing, embedding validation, and latency benchmarking.
• Pandas / NumPy, etc.statistical analysis for output validation and error pattern investigation, data profiling, schema validation, and pipeline integrity checks.

API and Automation:

• Pytest: modular, reusable test framework for AI eval and automation suites.
• Postman / REST Assured / Requests: API contract validation and service-level integration testing.

MLOps and CI/CD:

• MLflow: tracking model versions and eval runs to detect regressions across updates.
• Docker / GitHub Actions / Jenkins: containerized test environments and deployment pipeline automation.

Observability:

• Grafana / Kibana / OpenTelemetry: monitoring AI system health, output drift, and distributed tracing across agent pipelines.

Preferred Qualifications:

• Cloud AI Services: AWS Bedrock, Azure OpenAI, or GCP Vertex AI for testing managed model endpoints and cloud-deployed agents.
• MLOps Platforms: MLflow, Kubeflow, Weights and Biases, or Feast feature stores for experiment tracking and model governance.
• ML Frameworks: Scikit-learn, TensorFlow, or PyTorch familiarity for understanding model internals and validating training pipelines.
• Infrastructure as Code: Docker, Kubernetes, and Terraform for managing containerized test environments at scale.
• UI Automation: Playwright or Cypress for end-to-end conversational AI application testing.
• Performance Engineering: Locust or JMeter for load testing heavy AI inference endpoints under peak traffic.
• Statistical Hypothesis Testing: t-tests, confidence intervals, and significance testing to distinguish real quality signals from noise.
• Synthetic Data Generation: using LLMs to generate diverse test cases and evaluation datasets at scale.