Senior Data Engineer (Azure, Databricks)
Primary stack
Nice to have's
Job description
About the Position
This initiative centers on a mature, high-availability data hub architected within Azure Databricks, utilizing Unity Catalog, Spark Structured Streaming, and Asset Bundles to ensure robust performance and governance. The solution is built upon the Medallion Architecture framework, providing a scalable foundation that is now being expanded to ingest, transform, and seamlessly integrate new data sources into the existing ecosystem.
Responsibilities
- Developing Databricks pipelines in PySpark / SQL
- Ingesting data from APIs
- Handling Spark Structured Streaming pipelines
- Designing and managing objects in Unity Catalog according to Medallion Architecture
Requirements
- At least 6 years of commercial experience in Data Engineering with focus on large scale data pipelines.
- Proven track record using advanced solutions in Azure cloud and Databricks
- Knowledge of Spark Structured Streaming
- Proficiency in Python and SQL
- Ability to build pipelines using Github or similar tools
We Offer
- 100% remote work with the possibility to use an office in Warsaw
- Flexible working hours and engaging company events
- Access to Azure and GCP training packages available to official partners
- Possibility to use an in-house incubator to develop your own ventures
- A supportive community of 50+ senior data consultants with extensive experience in Data Warehouse and Big Data
- and much more!
About the Company
Senior Data Engineer (Azure, Databricks)
Senior
Can’t find your ideal role?
No worries! You’re welcome to send us your resume, and we’ll reach out if a suitable position comes up. We believe we’ll find the right spot for you that matches your skills and aspirations.
CONTACT US
© DS STREAM. This job description was sourced from the employer's public career page. TheJob is not the employer — we index the posting and route candidates to the source. All content rights and hiring decisions belong to the employer.
FMCG,Retail,E-commerce,Healthcare,Telco,Finance,Logistics
About the Company DS Stream is a fast-growing AI & Data consulting company focused on analytics and data management for global brands. The company employs a team of over 200 senior experts who specialize in Data Science, Machine Learning, and MLOps. Their mission is to transform know-how into effective architectures for their clients. Products, Services & Tech Stack Core Solutions Data Engineering: Transforming disparate data into valuable insights for informed business decisions. AI Engineering: Providing modern AI technologies and custom AI tools to support AI-driven business decisions. MLOps: Integrating ML technologies with DevOps practices to automate Machine Learning workflows and improve efficiency. Technologies & Frameworks Specific platforms, languages, or tools are not mentioned in the provided text. Project Scope & Target Clients Client Base Industries Served: FMCG, retail, e-commerce, healthcare, telco, finance, logistics, and more. Engineering Scale The company delivers tailored and complex data analytics for global corporations. Projects include real-time intelligence for the longevity ecosystem, category management for FMCG, and sales automation for resort packages.