Senior AI Quality & Evaluation Engineer

[…] is an international engineering firm helping businesses operate with AI at the core. With 6,000+ experts worldwide, we combine engineering expertise with AI-native thinking to turn ambitious ideas into real-world impact across industries, technologies, and markets. At the core of the role In Bulgaria, are looking for a highly skilled Senior AQA Engineer to design and build automated testing frameworks for AI-powered applications, with a strong focus on LLM evaluation, AI safety, RAG and agent testing, and integrating quality checks into CI/CD pipelines.

Core skills you’ll bring 5+ years in software quality or test engineering, with strong Python (C#/.NET is a plus). You write test frameworks, not only test cases. Hands-on experience evaluating LLM applications: evaluation frameworks (for example Promptfoo, DeepEval, Ragas, Langfuse or LangSmith), building datasets, designing judges and metrics.

Working knowledge of RAG and agent architectures and how they fail: retrieval quality, tool-call errors, loops, hallucination, context loss. Experience with adversarial or security testing, of AI systems or in general. CI/CD and infrastructure-as-code literacy; comfortable running tests on at least one of Azure, AWS or GCP.

Clear written reporting: you turn findings into decisions. Full working proficiency in English. Nice-to-have Familiarity with the EU AI Act, ISO/IEC 42001 or the NIST AI RMF.

Performance and load testing of LLM-backed services. Benchmarking open-weight models