AI Engineer - Agent Validator

Description

As organizations shift from traditional software to autonomous, LLM-driven agents, quality assurance is evolving from "does the code work" to "does the agent behave, reason, and act correctly — reliably, safely, and at scale." We're looking for an Agent Validator to own quality across the Agentic Development Lifecycle (ADLC) — designing evaluation harnesses, validating agent behavior against expected outcomes, and building the guardrails that let autonomous systems ship with confidence.

This role sits at the intersection of classical QA rigor and emerging agentic AI evaluation practice — ideal for someone who wants to shape how quality is defined for the next generation of software.

• 6–9 years of experience in software quality engineering, with deep expertise in test design methodology and validation processes, including exposure to manufacturing/industrial domains — a strong foundation for reasoning about agent behavior in complex, rules-bound environments.
• Hands-on experience with Jira, GitHub, Azure DevOps (ADO), or similar tools — repurposed here for defect and drift tracking, issue triage, and agent behavior regression logs.
• Proven ability to create, execute, and maintain test artifacts — evolving into eval suites, golden datasets, and behavioral fingerprints used to benchmark agent outputs over time.
• Solid grounding in SDLC, Agile methods, and ceremonies — translated into ADLC practices: eval-driven development, shadow-mode releases, and CI/CD-style evaluation pipelines for agentic systems.
• Active awareness of GenAI trends in QA, with hands-on use or advocacy of AI-powered tools (GitHub Copilot, OpenAI, or equivalents) — extending naturally into LLM-as-judge frameworks and AI-assisted test generation.
• Direct experience in agent testing and evaluation — validating multi-step reasoning, tool use, and task completion against defined success criteria.
• Breadth across testing types — functional, regression, exploratory, API, smoke, sanity, risk-based, and end-to-end — reframed as coverage dimensions for agent validation: task success, tool-call accuracy, hallucination/drift detection, and edge-case robustness.
• Proactive leadership in identifying automation opportunities, with working knowledge of (or willingness to upskill in) Playwright, Selenium, or Python — applied to automated eval pipelines and continuous agent regression testing.
• Hands-on experience building/maintaining automation frameworks — extended to eval harness design and observability tooling for agentic pipelines.
• Hands-on API testing experience (Postman, Swagger, REST-assured) — critical for validating agent-to-tool and agent-to-system integrations, including tool-call schemas and function-calling correctness.
• Strong analytical, leadership, and communication skills, with a passion for innovation and mentoring — essential for establishing quality gates and best practices in a domain still being defined.

• Prior experience in Agent or Agentic AI testing — a strong advantage given the evolving nature of this discipline.
• Exposure to performance, security, or accessibility testing tools — extendable to agent safety testing, including blast-radius assessment and adversarial/red-team style evaluation.
• Track record championing new tooling or methods within project teams — valuable for driving eval framework adoption and cross-team quality standards.
• Experience leveraging AI/ML tools for test case generation, automation, or code review — directly transferable to synthetic eval dataset generation and AI-assisted regression detection.

Enable Skills-Based Hiring