SDET/QA Engineer, AI Evaluation

• Evaluate, validate, and improve quality across teams
• Participate in developing and executing exploratory and automated tests
• Review system requirements and track quality assurance metrics such as pass rate, defect densities, and open defect counts
• Liaise with technical and business stakeholders from Codurance and client teams
• Take ownership of a Python LLM evaluation tool, diagnostic suite, and golden datasets to raise pass rates on production AI agents
• Refine scoring criteria and design multi-turn conversational evaluations for AI agents
• Set up automated regression testing for core product flows using Playwright, covering integrated workflows and third-party channels
• Implement SonarQube quality gate monitoring and daily deviation reporting
• Partner with senior stakeholders on QA strategy, standards, and continuous improvement
• Estimate, prioritize, plan, and coordinate testing activities
• Stay up to date with new testing tools and test strategies

• At least 4 years of experience in a similar role
• Fluent in Spanish and English (speaking and writing)
• Consultancy experience and an open mind around what is quality (adapting to clients)
• Hands-on experience with Playwright, SonarQube and other testing tools (Cucumber, Selenium, Appium, Jenkins)
• Knowledge of CI/CD tools and processes
• Continuous learning and curiosity
• Strong programming skills with Python
• Practical experience in AI evaluation, including LLM evals, agent testing, multi-turn chat scoring, and golden dataset management
• QA strategy knowledge (holistic testing, Agile testing Manifesto, ...) (bonus)
• Anthropic certification (bonus)
• Experience with performance/load testing for AI agents/call center agents and multi-channel integration (bonus)
• Familiarity with BDD and TDD (bonus)
• ISTQB certificate (bonus)
• Eligible to work in the EU; visa sponsorship and relocation assistance are not provided

• Transparency - all of our salary bandings and company finances are available to everyone from day one.
• Personal Learning Budget - Access to Learnerbly, our online learning & development tool, to support your personal development and career progression here.
• Training - No fixed training budget. So long as the course is relevant for the company and you'll do a lightning talk on the subject, we’re happy to pay for it.
• Working 100% remote (or hybrid if you want to drop by the office from time to time)
• Private medical and dental insurance
• Restaurant, transport and nursery tickets (retribución flexible/ tax deductible)
• Gym discounts
• Language classes (Spanish, English, French, German)
• You choose between Linux, Mac or Windows!
• Our People - You’ll be working alongside Craftspeople whether that’s on a client project or contributing to our internal projects.
• Equal Opportunities Employer and inclusive workplace