Lead AI Engineer with Spark, AWS Services

About the Position

We're looking for a Lead AI Engineer skilled in Spark and AWS Services to become part of the RBQM Production Pod within the program. In this position, you'll construct and sustain data pipelines that drive AI/GenAI applications supporting Risk-Based Quality Management for clinical trials. The primary focus of this role includes RAG document ingestion, vector indexing, and developing data APIs for AI applications.

Responsibilities

• Architect and construct RAG document ingestion pipelines (chunking, embedding, vector indexing) to support clinical trial quality data
• Establish and oversee vector databases (AWS OpenSearch) to support RAG-driven AI workflows
• Create batch and streaming ETL/ELT pipelines from the ground up for unstructured clinical data (PDF, DOCX, clinical reports)
• Construct and expose data APIs that AI applications can consume
• Enhance chunking strategies, embedding generation, and retrieval performance within RAG architectures
• Oversee data quality, lineage, and governance across AI/ML data pipelines
• Set up and sustain AWS data infrastructure (S3, Lambda, Glue, Athena, Step Functions, DynamoDB)
• Partner with Data Scientists and Backend Developers as part of a unified pod team

Requirements

• Minimum 7 years of practical, large-scale data engineering experience
• Strong background in RAG document ingestion pipelines (chunking, embedding, vector indexing)
• Skilled in using AWS OpenSearch as a vector database for RAG workflows
• High-level command of Python, along with SQL and Spark SQL
• Experience transforming unstructured data (PDF, DOCX) for use in RAG/LLM applications
• Working knowledge of AWS Services: S3, Lambda, Glue, Athena, Bedrock, Step Functions, API Gateway, CloudWatch, DynamoDB
• Understanding of Docker-based containerization
• Ability to develop custom pipelines from the ground up, going beyond simple configuration of pre-built services
• Proficiency in English at a B2+ level

Nice to Have

• Experience within the pharmaceutical or life sciences sector
• Exposure to Snowflake and Pinecone (as an alternative vector database)
• Understanding of SageMaker processing jobs
• Proficiency with CI/CD tools (Jenkins, Git/Bitbucket) and infrastructure-as-code tools (CDK or Terraform)
• Familiarity with clinical data standards (CDISC, ADaM, SDTM)

We Offer

• Enjoyable working environment, whether it is the vibrant office or the comfort of your own home
• Opportunity to work abroad for up to two months per year
• Relocation opportunities within our offices in 55+ countries
• Corporate and social events
• Leadership development, career advising, soft skills and well-being programs
• Certifications, including GCP, Azure and AWS
• Unlimited access to EPAM's internal learning database
• Free English classes with certified teachers
• Participation in the Employee Stock Purchase Plan
• Monetary bonuses for engaging in the referral program
• Comprehensive medical & family care package
• Five trust days per year (sick leave without a medical certificate)
• Benefits package (sports activities, a variety of stores and services)

About the Company

We connect like-minded people: Delivering innovative solutions to industry leaders, making a global impact