Senior Data Engineer (Scala/Spark, Databricks)
Primary stack
Nice to have's
Job description
About the Position
We are seeking a skilled Senior Data Engineer with expertise in Scala, Spark, and Databricks to join our team. In this role, you will lead the design, implementation, and optimization of data pipelines, ensuring scalable and efficient data processing. Your work will enable the aggregation of commerce purchase and catalog data into a centralized data lake while contributing to innovative future projects.
Please note that for positions in Ukraine, we only consider candidates who are currently based in Ukraine or plan to return in the near future. Remote work is available only from Ukraine.
Responsibilities
- Develop and maintain code bases for ETL and ELT pipelines, large batch/micro-batch processing, and streaming systems
- Build infrastructure for optimal extraction, transformation, and loading of data using Spark, ADF, or similar technologies
- Identify and implement internal process improvements, including automation, data optimizations, and infrastructure redesign for scalability
- Design and implement data services solutions leveraging technologies such as Spring Boot, ReactJS, or NoSQL
- Ensure governance of processes in delivery management and production adhering to the selected delivery model
- Serve as a single point of accountability for delivery-related matters, including escalations, upsells, and ramp-downs
- Lead technical delivery to ensure sound architecture and compliance with quality standards
- Define and document stories with associated acceptance criteria for agile workflows
- Coordinate with stakeholders and teams across various disciplines to ensure seamless project execution
- Deliver projects following customer processes, methodologies, and agile approaches
- Establish continuous delivery risk management strategies for proactive decision-making across the lifecycle
- Enhance delivery productivity through measurement and process improvements
- Provide expert consulting and guidance to Data Engineers to ensure quality deliverables
- Manage production support and deployment activities
Requirements
- At least 3 years of working experience in data engineering roles
- Proficiency in Scala, Spark, and Databricks
- Expertise in SQL, including writing and optimizing complex queries
- Familiarity with Spark Streaming and Apache Kafka
- Background in designing data pipelines using Databricks or similar tools
- Strong soft skills with excellent English proficiency (B2+) for direct client communication
Nice to Have
- Skills in Apache Kafka, Apache Kafka Streams, PySpark, and Spark Streaming
- Understanding of Python and its use in data engineering contexts
- Knowledge of Azure cloud platforms for data solutions
We Offer
- Opportunity to work with cutting-edge technologies
- Collaborative and innovative work environment
- Competitive compensation package
About the Company
[Company description if present]
© EPAM. This job description was sourced from the employer's public career page. TheJob is not the employer — we index the posting and route candidates to the source. All content rights and hiring decisions belong to the employer.
EPAM helps organizations innovate their business processes and rethink the way they manage their businesses so they can remain competitive in this new digital age.