Senior Data Engineer, Databricks/Kafka

About the Position

We are seeking a Senior Data Engineer with expertise in Databricks and Kafka to optimize scalable data pipelines and ensure data reliability for our analytics and machine learning applications.

Responsibilities

• Architect, build and optimize scalable data pipelines for batch and real-time data processing
• Develop and implement ETL/ELT workflows, ensuring efficient data ingestion, transformation and storage
• Leverage modeling (Party model, Datavault) methodologies to enable scalable and flexible data modeling
• Ensure data consistency, reliability and governance across data lakes, warehouses and operational data stores
• Optimize performance and cost efficiency of data infrastructure on Azure
• Implement and manage big data processing frameworks, such as Databricks, Kafka
• Enhance data security and compliance, integrating RBAC & ABAC, encryption and regulatory frameworks (GDPR) into data infrastructure
• Develop automation tools for data pipeline orchestration, using Airflow, Azure Data Factory or Prefect
• Monitor, troubleshoot and optimize data pipelines, ensuring minimal downtime and quick recovery from failures
• Collaborate with federated engineering teams to align data architecture with business and engineering goals
• Provide clean, reliable and scalable datasets for advanced analytics and machine learning in partnership with data scientists and business analysts
• Evaluate and adopt emerging technologies, ensuring continuous improvement in data engineering best practices
• Mentor and guide junior engineers, fostering technical excellence and knowledge sharing within the team

Requirements

• 5+ years of experience in data engineering, ETL development or big data technologies
• Expertise in designing and optimizing ETL/ELT workflows, using tools such as dbt, Airflow, Azure Data Factory or Apache NiFi
• Hands-on experience with cloud-native data platforms, including Azure Synapse, Databricks, Snowflake or BigQuery
• Knowledge of data modeling techniques, including Data Vault 2.0, star schema and normalization strategies
• Experience with large-scale distributed computing frameworks (Apache Spark, Hadoop, Kafka, Event Hub)
• Advanced proficiency in SQL and programming languages, such as Python, Scala or Java
• Understanding of Infrastructure as Code (Terraform, Pulumi, ARM templates) for managing cloud-based data infrastructure
• Skills in data security and governance best practices, including RBAC, encryption and data lineage
• Experience working in a federated engineering environment, ensuring seamless integration across teams
• Proficiency in observability and monitoring tools for data pipelines, such as Monte Carlo, DataDog and Great Expectations
• Familiarity with Agile and DevSecOps methodologies, ensuring continuous integration, deployment and monitoring of data solutions
• Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems or a related field
• Upper-Intermediate English language proficiency (B2+)

Nice to Have

• Apache Spark
• Hadoop
• Event Hub
• Terraform
• Pulumi
• ARM templates
• Data governance
• Observability
• Agile
• DevSecOps

We Offer

• Opportunity to work remotely from Türkiye
• Engaging and challenging projects in a leading tech company

About the Company

EPAM Systems is a global software engineering and product development company, delivering digital transformation and technology innovation to the world’s leading companies.