Senior Data Engineer (Databricks)

About the Position

We are seeking a Senior Data Engineer (Databricks) to build cloud-based data products that power insurance analytics and reporting. You will design reliable pipelines on Microsoft Azure using Azure Databricks and Python, improve data quality and governance, and partner with data and business teams to turn complex datasets into trusted insights.

Responsibilities

• Design, build and maintain scalable ETL and ELT pipelines in Azure Databricks
• Develop batch and streaming data solutions using Azure Data Factory, Azure Data Lake Storage Gen2, Azure Event Hubs, and Kafka
• Partner with data architects, analysts, and business stakeholders to define requirements and deliver fit-for-purpose data assets
• Implement data quality checks, validation, cleansing, and governance including Informatica lineage, metadata, and cataloging
• Apply security and compliance controls including role-based access control (RBAC), managed identities, and encryption aligned to Hong Kong Personal Data (Privacy) Ordinance (PDPO) and General Data Protection Regulation (GDPR)
• Improve deployment reliability through GitHub Actions automation and solid operational practices
• Troubleshoot pipeline issues, perform root cause analysis, and drive performance tuning
• Maintain clear documentation for transformations, testing, and runbooks

Requirements

• Hands-on experience building data pipelines with Azure Databricks and Apache Spark (PySpark or Scala)
• Strong background with Microsoft Azure data services including Azure Data Factory and Azure Data Lake Storage Gen2
• Proficiency in Python for data transformation and automation
• Experience working with streaming platforms such as Apache Kafka or Azure Event Hubs
• Knowledge of data governance and data quality practices including metadata and lineage tools such as Informatica
• Understanding of security and compliance controls including RBAC, encryption, and secrets management such as Azure Key Vault
• Practical experience with continuous integration and continuous delivery (CI/CD) automation such as GitHub Actions
• Clear communication in Cantonese, Chinese, and English (conversational)

Locations

• Guangzhou, China
• Shenzhen, China

We Offer

• Opportunity to work with cutting-edge technologies
• Collaborative and innovative work environment
• Competitive compensation package
• Professional growth and development opportunities