QA / ML Tester – Evaluation Framework

About the Position

We are looking for a QA / ML Tester – Evaluation Framework to ensure the quality, reliability, and correctness of evaluation systems used across enterprise AI and agent-based platforms. In this role, you will design and execute validation strategies for evaluation frameworks, test evaluator behavior across multiple scenarios, and verify the accuracy of automated quality assessment pipelines. You will work closely with AI engineers, platform teams, and quality specialists to establish confidence in evaluation results and support enterprise-grade AI governance.

Responsibilities

• Design, implement, and maintain automated test suites for AI evaluation frameworks and evaluation pipelines.
• Develop Python-based test automation using pytest to validate evaluator behavior, quality scoring, and framework reliability.
• Create and maintain known-good and known-bad test datasets, sessions, and workflows for evaluator correctness validation.
• Validate the accuracy and consistency of evaluation results across different agent workflows, prompts, tools, and execution scenarios.
• Design and execute integration tests for on-demand evaluation workflows integrated into CI/CD pipelines.
• Verify online evaluation behavior, sampling accuracy, and evaluation result consistency in production-like environments.
• Conduct functional testing of evaluation components, including evaluator execution flows, scoring logic, and result aggregation.
• Collaborate with AI and platform engineering teams to identify edge cases, failure scenarios, and evaluation blind spots.
• Validate workflow compliance, tool execution assessment, and end-to-end quality evaluation processes.
• Support feasibility assessments for applying evaluation frameworks to non-AgentCore runtimes and alternative AI execution environments.
• Analyze defects, inconsistencies, and quality regressions within evaluation systems and provide actionable recommendations.
• Contribute to quality assurance standards, testing methodologies, and best practices for AI evaluation platforms.

Requirements

Skills:

• Python test automation (pytest)
• Evaluator correctness testing (known-good / known-bad session pairs)
• On-demand mode integration testing with CI/CD
• Online mode sampling accuracy validation
• Non-AgentCore runtime feasibility assessment methodology

Experience:

• 4+ years QA or ML testing engineering
• AI/LLM system quality testing
• Integration test design for evaluation pipelines
• Experience with AWS environment

Nice to Have:

• AWS AgentCore Evaluation API testing
• OpenTelemetry trace-based evaluation input testing
• Multi-evaluator execution correctness testing

We Offer

At Intellias, where technology takes center stage, people always come before processes. By creating a comfortable atmosphere in our team, we empower individuals to unlock their true potential and achieve extraordinary results. That’s why we offer a range of benefits that support your well-being and charge your professional growth.

We are committed to fostering equity, diversity, and inclusion as an equal opportunity employer. All applicants will be considered for employment without discrimination based on race, color, religion, age, gender, nationality, disability, sexual orientation, gender identity or expression, veteran status, or any other characteristic protected by applicable law.

We welcome and celebrate the uniqueness of every individual. Join Intellias for a career where your perspectives and contributions are vital to our shared success.

About the Company

Intellias is a global technology company that specializes in delivering innovative software solutions. Our mission is to help clients engineer comprehensive software ecosystems for cutting-edge products and services. With offices in over 180 countries, we are dedicated to providing best-in-class eCommerce, Digital Marketing, and IoT solutions. As part of our Core Architecture Team, you will be responsible for the architecture and implementation of best practices in our Digital Engineering Enterprise Platform.