Lead Data Software Engineer with Databricks, Apache Kafka, Apache Spark, Kubernetes

About the Position

We are seeking a Lead Data Software Engineer to drive the migration of an existing data analytics platform, originally built on Azure resources such as DataFactory, Databricks, EventHub, and Cassandra, to a cloud-agnostic platform capable of on-premises deployment.

Responsibilities

• Implement streaming and batch Spark pipelines running on Kubernetes (K8S)
• Develop objects of the data generator framework
• Deploy and test solutions on local and development environments
• Enhance and refine the current actively evolving solution
• Investigate and resolve bugs
• Conduct unit testing to ensure code quality
• Participate in refinement, planning, and demo sessions
• Guide and mentor team members while syncing with the team lead

Requirements

• 5+ years of experience with Apache Spark, Databricks, and Kubernetes (K8S)
• Proficiency in Python
• Experience with Confluent/Apache Kafka
• Completed EPAM Data Engineering course or equivalent qualification
• Capability to quickly adapt and dive into the active implementation phase of a project
• Ability to work independently while coordinating with a team lead
• Strong communication skills and a proactive, hands-on, team-player attitude
• English proficiency at B2 level or higher

Nice to Have

• Knowledge of Java
• Familiarity with Kafka Connect, KSQL, and Kafka Streams
• Background in Ansible and Argo CD
• Understanding of TDD

We Offer

• Continuous Upskilling, Learning & Development

  • Diversity of tasks and projects
  • Assessment center for objective review of competency level
  • Personal development plan
  • Mentoring programs and leadership development
  • Certification and professional development support
  • Access to learning platforms including more than 2,500 internal courses
  • English courses taught by certified teachers

• Corporate Benefits

  • Extra leave days
  • Referral bonuses

• Compensation Package

  • Competitive compensation paid in USD
  • Regular salary and performance reviews

• Medical & Healthcare

  • Private health insurance
  • Well-being events

• Working Environment

  • Recreation areas and kitchens
  • Tea, coffee, and snacks
  • Sports equipment and game consoles
  • IT Equipment
  • Microsoft’s Software Assurance Home Use Program (HUP)

About the Company

[Company description if present]