Senior QA / ML Tester

We are looking for a Senior QA / ML Tester to join the AI Platform team and take ownership of quality assurance for the Agent Evaluation Framework built on AWS Agent

Core. This role involves designing, implementing, and maintaining a functional test suite that validates the correctness of AI agent evaluation pipelines — covering on-demand integration testing, online sampling accuracy, and multi-evaluator execution — and delivering a feasibility assessment for non-Agent

Core runtime evaluation scenarios. This is a hands-on, production-focused role at the intersection of software quality engineering and AI/ML system testing, operating within an Agile delivery team and contributing to the reliability of enterprise-grade agentic AI infrastructure.

Design and implement a functional test suite for the AWS Agent

Core Evaluation API using pytest, covering known-good / known-bad session pair validation, multi-evaluator execution correctness, and edge case handling Develop and maintain integration tests for on-demand evaluation mode, integrated into the CI/CD pipeline with automated execution on each build Validate online mode sampling accuracy, design test scenarios, define acceptance criteria, and report deviations with reproducible evidence Conduct and document a feasibility assessment for non-Agent

Core runtime evaluation: analyze alternative runtimes, define evaluation methodology, and deliver a structured findings report Test OpenTelemetry trace-based evaluation inpu