Senior/Lead Databricks Engineer

About the Position

Senior/Lead Databricks Engineer

We are looking for a Senior/Lead Databricks Engineer with expertise in cloud infrastructure, big data processing, data integration, and analytics to join our team at EPAM Systems in Malta. This role is integral to one of our projects in the client’s IT area, focusing on the integration component of their landscape. You will play a crucial part in advancing our client’s Data-Driven Company initiative by designing and building robust data architecture to support various aspects of the client’s business. Join us at our Malta office, which offers a flexible hybrid work setup.

Would you like to be a part of this ambitious and innovative project? We look forward to receiving your application.

Responsibilities

• Data Integration, Transformation, and Modelling: Serve large amounts of structured & semi-structured financial data.
• Analyze Existing Process Logic: Implement best practices in data architecture, including security, governance, compliance, and performance optimization.
• Optimize Data Storage: Optimize the data storage across Delta Lake for data management.
• Develop and Maintain Scalable Data Pipelines: Build out new API integrations to support continuing increases in data volume and complexity.
• Collaborate with Teams: Work with analytics and business teams to improve data models that feed business intelligence tools, increasing data accessibility and fostering data-driven decision-making across the organization.

Requirements

• At least 5+ years of professional experience in the data engineering field.
• Proficiency in Azure Databricks, Delta Lake, Python, and PySpark for data manipulation.
• Proficiency in designing and optimizing pipelines using Azure Data Factory, including performance tuning and optimizing data pipelines with Databricks Notebooks.
• Experience with Azure data platforms and related services.
• Good understanding of SQL and experience with relational database systems (e.g., MSSQL, PostgreSQL, Oracle DB).
• Understanding of data structures, data modelling, and data architecture patterns (Data Lake, DWH, Data mesh, etc.).
• Ability to work in an Agile environment and eager to learn and develop new skills.
• Good problem-solving skills and attention to detail.
• Strong written and spoken English (B2+).

Nice to Have

• Certifications in Azure data or Cloud Services (e.g., Azure Data Engineer Associate).
• Business understanding of the insurance or financial domain.
• Experience with BI Tools (Power BI).

We Offer

• Opportunity to work on an innovative project with a leading company.
• Flexible hybrid work setup from our Malta office.
• Professional growth and development opportunities.

About the Company

EPAM Systems is a global software engineering and product development company that partners with the world’s leading brands to deliver technology solutions that drive digital transformation.