Senior AI QA Engineer

About the Position

We are looking for a Senior AI QA Engineer to join our team at Waverley Software. This is a remote position with a flexible schedule, based in Cochabamba, Cochabamba, BO.

About the Company

Waverley Software is a global software engineering powerhouse dedicated to solving complex digital challenges. We partner with innovation-driven clients to build production-ready enterprise applications using cutting-edge technologies. Our culture thrives on engineering excellence, transparent communication, and a passion for pushing the boundaries of what is possible in the AI landscape.

Role Summary

As a Senior AI Quality Assurance Engineer, you will ensure the reliability, accuracy, and security of our AI-driven products. During project initiation, you will consult with clients to define quality metrics and testing strategies for probabilistic systems. Once in the execution phase, you will build and run the automated testing pipelines, evaluating model outputs, mitigating hallucinations, and assuring overall software quality.

Responsibilities

• Pre-Sales Support: Support the pre-sales team when needing QA insights/inputs.
• Test Automation: Architect and maintain automated CI/CD testing pipelines for both traditional software features and AI-specific workflows.
• Model Evaluation: Systematically evaluate LLM and RAG pipeline outputs for contextual accuracy, retrieval relevance, and latency.
• Quality Assurance: Execute comprehensive API, UI, and integration testing to ensure enterprise-grade stability.

Requirements

• 6+ years of QA automation engineering experience using tools like Selenium, Cypress, or Playwright.
• Senior-level QA capabilities, including generating comprehensive Test Plans, defining QA technical standards, and owning the end-to-end Defect Management Lifecycle.
• Strong scripting skills in Python and/or JavaScript.
• Experience testing RESTful APIs and modern web applications.
• Hands-on experience with LLM evaluation frameworks and observability tools (e.g., Ragas, TruLens, LangSmith, or Promptfoo).
• Data expertise, including data management, preparation, generation, and quality assessment.
• Knowledge of CI/CD pipelines and automated deployment processes.
• Strong communication skills, with the ability to communicate clearly to both technical and non-technical stakeholders.

Nice to Have

• Experience with AI security testing, including prompt injection vulnerabilities.
• Familiarity with load and performance testing for AI endpoints (JMeter, Locust).
• Background in DevOps and CI/CD configuration (GitHub Actions, Jenkins, GitLab CI).

We Offer

• Remote Work: Flexible schedule and remote work options.
• Professional Growth: Opportunities for continuous learning and career advancement.
• Collaborative Environment: Work with a team of passionate and innovative professionals.

Contact

Flavia Taborga

Senior Recruiter

Email: [email protected]