Lead Data Engineer
Primary stack
Nice to have's
Job description
About the Position
We are seeking a visionary and experienced Lead Data Engineer to join our Data Engineering team in Bulgaria.
In this role, you will lead a Platform Team, driving the design, development, and scaling of data pipelines in Databricks with PySpark on Microsoft Azure for our AI Factory team. This is a unique opportunity to shape the future of our data platforms, influence technical strategy, and mentor a talented team, all while working at the forefront of big data and cloud engineering.
Responsibilities
- Lead the design and architecture of cloud-native analytical solutions using Big Data and NoSQL technologies
- Oversee the development and optimization of scalable data pipelines in Databricks with PySpark on Azure
- Own the strategy for building and maintaining data lakes and warehouses, ensuring reliability, performance, and scalability
- Define and implement ETL/ELT workflows and best practices for data collection, cleaning, and structuring
- Establish and enforce data quality, lineage, and monitoring frameworks across the team
- Collaborate closely with ML, analytics, and business teams to deliver production-ready datasets and solutions
- Conduct and lead code reviews, setting technical standards and fostering best practices
- Mentor and coach data engineers, cultivating a high-performance and collaborative team culture
- Champion CI/CD methodologies in data engineering workflows using tools like Jenkins or GitLab CI/CD
- Drive requirements gathering and solution alignment with architects, technical leads, and cross-functional teams
- Engage with stakeholders at all levels to understand business processes, model input data, and ensure deliverable alignment with strategic goals
Requirements
- 6+ years of experience in Data Engineering or a related field, with at least 2 years in a technical leadership role
- Deep proficiency in Python and PySpark
- Extensive hands-on experience with Databricks and Microsoft Azure cloud services
- Strong background in software version control tools (e.g., GitHub, Git)
- Proven track record with CI/CD frameworks such as Jenkins, Concourse, or GitLab CI/CD
- Demonstrated expertise in architecting and building scalable, robust, and highly available data solutions
- Excellent problem-solving, analytical, and stakeholder management skills
- Experience in mentoring and leading engineering teams
Nice to Have
- Experience with additional programming languages such as Java, SQL, or Scala
- Knowledge of SAP BTP or similar enterprise data platforms
- Familiarity with agile development methodologies
- Experience in strategic planning and technical roadmap development
Location
- City: Sofia
- Country: Bulgaria
We Offer
- Opportunity to work at the forefront of big data and cloud engineering
- Chance to shape the future of our data platforms
- Influence technical strategy and mentor a talented team
- Collaborative and high-performance team culture
© EPAM. This job description was sourced from the employer's public career page. TheJob is not the employer — we index the posting and route candidates to the source. All content rights and hiring decisions belong to the employer.
EPAM helps organizations innovate their business processes and rethink the way they manage their businesses so they can remain competitive in this new digital age.