Senior Automation Tester in Python (for Gen AI powered applications)

About the Position

We are seeking a Senior Automation Tester in Python for Gen AI-powered applications to join our team. This role focuses on validating AI outputs and ensuring compliance within our clinical applications.

Responsibilities

• Build automated test frameworks from scratch for AI-powered clinical applications using Playwright, Selenium, and pytest
• Design and execute test strategies for non-deterministic AI/GenAI outputs, including accuracy, RAG relevance, hallucination checks, and prompt regression
• Integrate automated tests into CI/CD pipelines with quality gates such as block-on-failure and coverage thresholds
• Execute GxP-compliant IQ/OQ/PQ scripts and generate qualification evidence packages
• Manage AI-specific test data, including edge cases, adversarial inputs, and golden datasets
• Monitor model drift, regression, and performance degradation over time
• Produce test strategy documents, test plans, and compliance-ready test evidence
• Collaborate with Data Scientists and Backend Developers to define testable acceptance criteria for AI features

Requirements

• 5+ years of hands-on QA/test automation experience
• Expertise in building and maintaining automated test suites from scratch using Playwright, Selenium, Cypress, and pytest
• Skills in API and backend test automation, including REST APIs, microservices, and serverless architectures with Postman/Newman and pytest + requests
• Proficiency in Python as the primary automation language, with the ability to write harnesses, fixtures, and utilities independently
• Knowledge of CI/CD test integration, embedding automated tests into pipelines such as Jenkins and GitHub Actions with quality gates
• Background in testing AI/GenAI applications, validating non-deterministic AI/LLM outputs for accuracy, RAG relevance, hallucination, and prompt regression
• Familiarity with AI/ML model testing, including accuracy, bias, and performance validation using RAGAS, LLM-as-judge, and golden datasets
• English proficiency at B2 level or higher

Nice to Have

• Background in pharmaceutical / life sciences domain, with GxP awareness and IQ/OQ/PQ qualification
• Experience with Jira/Xray, TestRail, or Zephyr test management
• Skills in performance/load testing using JMeter and Locust
• Understanding of security testing basics such as OWASP
• Familiarity with AWS services including Lambda, API Gateway, S3, and CloudWatch

We Offer

We gather like-minded people: Top tech minds driving innovation in AI, cloud and digital platform modernization. Our supportive team and agile, startup-like culture offer a hybrid work mode with the opportunity to work remotely within Poland. There is also a chance to work abroad for up to 60 days annually and business-driven relocation opportunities.

We provide growth opportunities: Career development programs, thought leadership, mentoring, soft skills and well-being programs, and certifications (Anthropic, Gemini, GCP, Azure, AWS). We also offer English classes.

We cover it all: Stable pay, participation in the Employee Stock Purchase Plan with a 15% discount, benefits package (health insurance, multisport, shopping vouchers), and referral bonuses up to $2,000. Our offices feature entertainment and relaxation zones, table tennis and football, free snacks, coffee, and more. We also organize corporate, social, and well-being events.

Please, note: Benefits listed above are available to employees only. We are open for working with Contractors. Terms of B2B cooperation agreements are agreed individually. We will reach out to selected candidates exclusively.