Software Development Engineer in Test – AI

This role goes beyond traditional QA; the ideal candidate should be capable of validating, evaluating, and improving AI-powered systems, agents, copilots, and LLM-driven workflows while building scalable automation frameworks and intelligent quality solutions. You will work closely with Engineering, AI/ML, Platform, Product, and DevOps teams to ensure reliability, safety, observability, scalability, and quality of modern AI-native applications and Agentic systems.

What you’ll do & how you’ll make your mark.

• AI/LLM Quality Engineering

• Design, validate, and optimize testing strategies for AI-powered applications, copilots, chatbots, RAG pipelines, and autonomous AI agents.
• Develop AI evaluation (Evals) frameworks for: Hallucination detection Groundedness Toxicity & bias validation Response quality Context relevance Prompt regression testing Determinism & consistency Safety and guardrail validation
• Build automated validation pipelines for LLM outputs using structured assertions, semantic scoring, embeddings, and evaluation metrics.
• Create synthetic datasets and benchmark datasets for AI testing scenarios.
• Test multi-agent orchestration workflows and agent memory/context retention.
• Validate MCP (Model Context Protocol) integrations and tool interoperability.
• Work on Context Engineering, Prompt Engineering, and Prompt Optimization strategies.
• Evaluate latency, token usage, model cost optimization, throughput, and scalability.

• Automation Engineer