Senior QA / ML Tester

Market insights

Stack

Apply

Upload your CV — we will connect you with the employer directly through our pool.

Description

We are looking for a Senior QA / ML Tester to join the AI Platform team and take ownership of quality assurance for the Agent Evaluation Framework built on AWS AgentCore. This role involves designing, implementing, and maintaining a functional test suite that validates the correctness of AI agent evaluation pipelines — covering on-demand integration testing, online sampling accuracy, and multi-evaluator execution — and delivering a feasibility assessment for non-AgentCore runtime evaluation scenarios. This is a hands-on, production-focused role at the intersection of software quality engineering and AI/ML system testing, operating within an Agile delivery team and contributing to the reliability of enterprise-grade agentic AI infrastructure. Responsibilities Design and implement a functional test suite for the AWS AgentCore Evaluation API using pytest, covering known-good / known-bad session pair validation, multi-evaluator execution correctness, and edge case handling Develop and maintain integration tests for on-demand evaluation mode, integrated into the CI/CD pipeline with automated execution on each build Validate online mode sampling accuracy, design test scenarios, define acceptance criteria, and report deviations with reproducible evidence Conduct and document a feasibility assessment for non-AgentCore runtime evaluation: analyze alternative runtimes, define evaluation methodology, and deliver a structured findings report Test OpenTelemetry trace-based evaluation inpu

Employer contacts (email/phone/telegram) are hidden from the public preview — send your CV, and we will connect you directly.