Platform Engineer – CI/CD Gate & Online Evaluation

About the Position

We are looking for a Platform Engineer – CI/CD Gate & Online Evaluation to help establish automated quality controls for enterprise AI platforms and agent-based systems. In this role, you will integrate evaluation workflows into deployment pipelines, configure online quality monitoring, and build operational visibility for AI performance and reliability. You will work closely with platform, DevOps, AI engineering, and governance teams to ensure that AI agents meet quality standards before and after production deployment.

Responsibilities

• Design and implement CI/CD quality gates using AWS AgentCore Evaluation to validate AI agents and workflows before deployment.
• Integrate evaluation execution into GitHub Actions and deployment pipelines to support automated release decisions.
• Configure and maintain on-demand evaluation workflows for pre-release quality validation and regression detection.
• Implement and manage online evaluation capabilities, including production sampling strategies and evaluation scheduling.
• Build CloudWatch dashboards that provide visibility into evaluation scores, quality trends, and agent performance over time.
• Configure CloudWatch alarms and notification mechanisms to detect quality degradation, performance regressions, and reliability issues.
• Collaborate with AI engineers to define evaluation thresholds, release criteria, and quality acceptance standards.
• Support platform-wide observability by integrating evaluation results into monitoring and operational reporting workflows.
• Analyze evaluation outcomes and identify quality regressions across agent versions, prompts, tools, and workflows.
• Develop and maintain automation that enables continuous validation of AI systems throughout the software delivery lifecycle.
• Partner with platform and DevOps teams to improve deployment safety, operational readiness, and governance controls for AI workloads.
• Contribute to the evolution of enterprise AI quality engineering practices, deployment standards, and monitoring frameworks.

Requirements

Skills:

• AWS AgentCore Evaluation API (on-demand mode - CreateEvaluation in CI/CD pipelines)
• GitHub Actions integration for evaluation deployment gate
• Online evaluation configuration (AgentCore online mode, configurable sampling rate)
• CloudWatch dashboards for per-agent per-evaluator evaluation scores over time
• CloudWatch alarm configuration for quality degradation thresholds

Experience:

• 4+ years platform or DevOps engineering
• CI/CD pipeline integration for quality gates
• CloudWatch dashboard and alarm design

Nice-to-have

• AWS AgentCore Evaluation online mode sampling configuration
• AWS AgentCore Runtime version comparison using evaluation (regression detection)
• CloudWatch EMF for structured evaluation score metrics

We Offer

At Intellias, where technology takes center stage, people always come before processes. By creating a comfortable atmosphere in our team, we empower individuals to unlock their true potential and achieve extraordinary results. That’s why we offer a range of benefits that support your well-being and charge your professional growth.

We are committed to fostering equity, diversity, and inclusion as an equal opportunity employer. All applicants will be considered for employment without discrimination based on race, color, religion, age, gender, nationality, disability, sexual orientation, gender identity or expression, veteran status, or any other characteristic protected by applicable law.

We welcome and celebrate the uniqueness of every individual. Join Intellias for a career where your perspectives and contributions are vital to our shared success.

About the Company

Intellias is a global technology company with a focus on delivering innovative solutions in the fields of AI, IoT, and software engineering. Our mission is to help clients transform their businesses through cutting-edge technology and expert engineering. With offices in over 180 countries, we are dedicated to providing top-tier services and fostering a culture of excellence and inclusivity.