Python Engineer – Evaluator Library

About the Position

We are looking for a Python Engineer – Evaluator Library to design and implement reusable evaluation components that ensure the quality, safety, and compliance of enterprise AI agents and LLM-powered workflows. You will build custom evaluation capabilities used across AI platforms, focusing on automated quality validation, workflow compliance, PII protection, and structured output verification. Working closely with AI Platform Engineers, ML Engineers, and DevOps teams, you will help establish reliable evaluation standards and scalable quality assurance mechanisms for agent-based systems.

Responsibilities

• Design, develop, and maintain reusable Python-based evaluator libraries for AI agents and LLM-powered workflows.
• Implement AWS Lambda-based custom evaluators to perform deterministic quality, compliance, and validation checks.
• Develop LLM-as-a-judge evaluation logic to assess subjective dimensions such as relevance, helpfulness, consistency, and response quality.
• Build automated PII detection evaluators using regex-based techniques and AWS Bedrock Guardrails integrations.
• Implement TOOL_CALL-level validation mechanisms to verify structured outputs, JSON schema compliance, and tool response correctness.
• Develop SESSION-level evaluators to validate workflow contract compliance, execution integrity, and cross-step behavioral expectations.
• Create TRACE-level evaluators for numerical accuracy verification, calculation consistency, and deterministic result validation.
• Integrate evaluator components with AWS AgentCore Evaluation workflows and enterprise AI quality pipelines.
• Collaborate with AI platform teams to define evaluation standards, scoring methodologies, and quality acceptance criteria.
• Design and maintain unit, integration, and validation tests for evaluator libraries and quality frameworks.
• Support observability and troubleshooting by integrating evaluation outputs with CloudWatch logging and monitoring capabilities.
• Contribute to enterprise AI governance initiatives by improving evaluation coverage, auditability, and compliance controls.

Requirements

Skills:

• Python (Lambda functions as AWS AgentCore custom code-based evaluators)
• LLM-as-judge prompt engineering for subjective evaluation dimensions
• PII detection (regex-based + AWS Bedrock Guardrails)
• Tool response schema validation (JSON Schema - TOOL_CALL level evaluator)
• Workflow contract compliance checking (SESSION level evaluator)
• Numerical accuracy validation logic (TRACE level evaluator)

Experience:

• 4+ years Python engineering
• LLM evaluation or quality assurance for AI/ML systems
• AWS Lambda function development and deployment

Nice-to-have

• AWS AgentCore Evaluation custom evaluator Lambda registration
• AWS Bedrock Guardrails for PII detection integration
• CloudWatch Logs as evaluator output sink

We Offer

At Intellias, where technology takes center stage, people always come before processes. By creating a comfortable atmosphere in our team, we empower individuals to unlock their true potential and achieve extraordinary results. That’s why we offer a range of benefits that support your well-being and charge your professional growth.

We are committed to fostering equity, diversity, and inclusion as an equal opportunity employer. All applicants will be considered for employment without discrimination based on race, color, religion, age, gender, nationality, disability, sexual orientation, gender identity or expression, veteran status, or any other characteristic protected by applicable law.

We welcome and celebrate the uniqueness of every individual. Join Intellias for a career where your perspectives and contributions are vital to our shared success.

About the Company

Intellias is a global technology company that specializes in delivering innovative software solutions. Our mission is to help clients engineer comprehensive software ecosystems for cutting-edge products and services. With offices in over 180 countries, we are dedicated to providing best-in-class eCommerce, Digital Marketing, and IoT solutions. Our teams are involved in the engineering of core platform components for a range of industries, ensuring compliance and best practices in every project.