Lead Databricks Engineer

About the Position

Lead Databricks Engineer

Lead · Hybrid · London, UK

We’re looking for a Lead Databricks Engineer to join our team in London, UK in a hybrid working mode. This role involves designing and leading enterprise-scale Azure and Databricks data platforms to enable advanced analytics and real-time processing capabilities. You will be responsible for platform architecture, governance, cost optimization, and ensuring delivery of high-quality solutions that meet critical business needs in a complex, data-driven environment.

Responsibilities

• Lead architecture design and implementation of scalable solutions on Azure Databricks
• Define and enforce platform standards, ensuring compliance with data governance and security policies
• Build and optimize Databricks streaming workloads, including Structured Streaming and Delta Live Tables
• Collaborate with cross-functional teams to translate requirements into robust technical solutions
• Drive platform performance tuning, cost optimization, and monitoring strategies for cloud workloads
• Provide leadership across the full engineering lifecycle, including solution delivery and production support
• Mentor and guide developers, establishing best practices for data engineering on Azure and Databricks
• Participate in technical decision-making and review designs for scalability and maintainability
• Implement observability for critical pipelines and maintain quality assurance standards
• Engage directly with stakeholders to ensure alignment on strategic platform initiatives

Requirements

• Minimum 8+ years of experience in data engineering, including 3+ years in a tech lead capacity
• Strong hands-on expertise in Databricks for enterprise-scale workloads
• Proficiency in PySpark, Spark Structured Streaming, and Delta Lake
• Solid programming skills in Python and demonstrated SQL performance optimization experience
• Deep understanding of Azure Data Platform services and cloud-native engineering patterns
• Proven experience implementing data governance, quality management, and observability frameworks
• Ability to manage complex data pipelines for batch and streaming use cases in mission-critical environments
• Strong communication and leadership skills to work with distributed teams and senior stakeholders

Nice to Have

• Knowledge of Delta Live Tables (DLT) and advanced workflows in Databricks
• Familiarity with CI/CD for data engineering using GitHub or Azure DevOps
• Experience designing infrastructure-as-code solutions for Azure environments
• Understanding of data mesh or lakehouse principles for large-scale architectures
• Background in financial trading or capital markets data domains

We Offer

EPAM Employee Stock Purchase Plan (ESPP)

Protection benefits including life assurance, income protection, and critical illness cover

Private medical insurance and dental care

Employee Assistance Program

Competitive group pension plan

Cyclescheme, Techscheme, and season ticket loans

Various perks such as free Wednesday lunch in-office, on-site massages, and regular social events

Learning and development opportunities including in-house training and coaching, professional certifications, and courses

If otherwise eligible, participation in the discretionary annual bonus program

If otherwise eligible and hired into a qualifying level, participation in the discretionary Long-Term Incentive (LTI) Program

All benefits and perks are subject to certain eligibility requirements

About the Company

[Company description if present]