Lead Data Software Engineer (Databricks)
Primary stack
Nice to have's
Job description
About the Position
We are searching for a Lead Data Software Engineer to join our team and oversee the development of our cutting-edge supply-chain data analytics platform.
This platform delivers a comprehensive view of suppliers, products, material categories, shipments, and compliance activities while employing artificial intelligence to generate insights and recommendations tailored to our clients’ needs.
If this role excites you – apply!
Responsibilities
- Design and develop data architectures and ETL pipelines utilizing Databricks and external orchestrators such as Airflow
- Collaborate with machine learning teams to incorporate AI-driven insights into the platform
- Develop system-level improvements and components for data engineering workflows
- Optimize the performance and scalability of data-intensive applications
- Adhere to software engineering standards, including containerization, unit testing, linting, and code style reviews
- Maintain and improve data workflow, Delta Lake, and Delta Live Tables within Databricks
- Provide technical mentorship and leadership to the data engineering team
- Engage with clients and stakeholders to understand their requirements and offer customized solutions
- Tackle challenges independently and ensure successful project delivery
Requirements
- Experience leading data engineering efforts on projects utilizing Databricks
- Solid understanding of data architectures and skills in data modeling
- Background in designing and building ETL pipelines in Databricks with external orchestrators such as Airflow
- Hands-on expertise in Databricks (Delta Lake, workflows, Delta Live Tables, deployment, and versioning)
- Proficiency in Python and cloud-native technologies
- Competency in Spark/PySpark
- Engineering experience with either AWS or Azure
- Knowledge of big data and techniques for optimizing data-intensive applications
- Strong communication skills with a proactive attitude and client-facing experience
- Ability to navigate ambiguity and complete projects independently
- Comfort working in a dynamic and transparent startup environment
- Fluent English communication skills at a C1 level
Nice to Have
- Proficiency in configuring and managing CI/CD pipelines in Azure DevOps
- Familiarity with Data Observability and Data Quality Monitoring practices
- Capability to integrate data quality checks into data pipelines
About the Company
[Company description if present]
Location
Hybrid in Greece
© EPAM. This job description was sourced from the employer's public career page. TheJob is not the employer — we index the posting and route candidates to the source. All content rights and hiring decisions belong to the employer.
EPAM helps organizations innovate their business processes and rethink the way they manage their businesses so they can remain competitive in this new digital age.