About the Position

Your vision. Our solutions.

We are seeking a Senior AI QA Engineer to join our AI Incubator program, focusing on advanced AI prototypes and ensuring reliability of AI systems.

Role Summary

The AI QA Engineer is the quality, safety, and reliability backbone of delivery. You ensure that agentic workflows, RAG systems, models, data pipelines, APIs, and UX layers behave reliably, safely, and consistently under real-world conditions.

This is not traditional QA. You design and own continuous evaluation strategies for non-deterministic AI systems and act as the final line of defense before solutions reach production environments.

Responsibilities

Own the end-to-end QA strategy across UI, backend, data, retrieval, and AI layers
Design and maintain LLM, RAG, and agent evaluation frameworks
Build automated test harnesses for Python services, APIs, agents, and pipelines
Integrate testing into CI/CD pipelines and prevent regressions
Validate data quality, embeddings, retrieval accuracy, and ranking performance
Identify hallucinations, reasoning failures, bias, and model drift
Design red-team and edge-case scenarios for safety and robustness
Define observability metrics for behavior, latency, cost, and failures
Run defect triage and deliver clear, actionable defect reporting
Collaborate closely with Tech Leads, engineers, and AI Ops

Requirements

Technical

Strong Python scripting for automation and evaluation
Experience with LLM / RAG testing, ML evaluation, or model benchmarking
Familiarity with vector databases, retrieval systems, and agent workflows
CI/CD, DevOps tooling, and observability platforms
Ability to validate embeddings, precision/recall, and ranking metrics

QA & Risk

5–6+ years in QA, SDET, testing, or ML evaluation roles
Experience testing non-deterministic or probabilistic systems (preferred)
Strong instincts for edge cases, failure modes, and adversarial risks

Mindset

Curious, skeptical, and systematic
High ownership and strong communication skills
Comfortable defining what “quality” means for AI systems

We Offer

Opportunity to work within an AI Incubator program
Focus on building advanced AI prototypes and new AI-powered products and services
Continuous exploration of frontier AI capabilities

About the Company

The Workly s.r.o.

4D CENTER, Kodaňská 1441/46,

101 00, Praha 10.

Contact: [email protected]

Phone: + 420 775 371 991

Senior AI QA Engineer

Job description