Senior Data Engineer
Primary stack
Nice to have's
Job description
About the Position
We are looking for an experienced Senior Data Engineer to drive the expansion of a high-availability data hub on Azure Databricks, working in the US time zone. In this role, you will design and implement robust data pipelines using PySpark and Structured Streaming, ensuring seamless integration of new data sources within a Medallion Architecture framework. You will work closely with the team to manage Unity Catalog objects and optimize performance, building scalable solutions that power critical data ecosystems. This position is ideal for an expert in Azure and Databricks who thrives on handling large-scale data challenges.
This initiative centers on a mature, high-availability data hub architected within Azure Databricks, utilizing Unity Catalog, Spark Structured Streaming, and Asset Bundles to ensure robust performance and governance. The solution is built upon the Medallion Architecture framework, providing a scalable foundation that is now being expanded to ingest, transform, and seamlessly integrate new data sources into the existing ecosystem.
Responsibilities
- Developing Databricks pipelines in PySpark / SQL.
- Ingesting data from APIs.
- Handling Spark Structured Streaming pipelines.
- Designing and managing objects in Unity Catalog according to Medallion Architecture.
Requirements
- Commercial experience of 5–8 years in Data Engineering with focus on large scale data pipelines.
- Proven track record using advanced solutions in Azure cloud and Databricks.
- Knowledge of Spark Structured Streaming.
- Proficiency in Python and SQL.
- Ability to build pipelines using GitHub or similar tools.
Nice to Have
- Cloud or Data Engineering certifications or specialization certifications (e.g. Google Professional Machine Learning Engineer, Microsoft Certified: Azure AI Engineer Associate Exam AI-102, AWS Certified Machine Learning Specialty (MLS-C01), Databricks Certified Machine Learning).
- Contributions to open-source AI projects.
- Experience with Agile/Scrum methodologies.
We Offer
- Work with a senior team of 10+ years experienced ML/AI experts.
- Hands-on experience with latest Generative AI and agentic technologies.
- Global exposure working on projects for leading brands.
- Technology-agnostic environment fostering innovation.
- Competitive compensation package.
- Opportunity to shape AI strategy in our growing India operations.
Location & Work Mode
- Location: Bangalore, Karnataka, India
- Work Mode: On-site
- Time Zone Preference: Candidates willing to work in the US time zone are preferred.
About the Company
CAREERS
Senior Data Engineer - Bangalore, India
Senior Data Engineer - Bangalore, India
Senior
Can’t find your ideal role?
No worries! You’re welcome to send us your resume, and we’ll reach out if a suitable position comes up. We believe we’ll find the right spot for you that matches your skills and aspirations.
CONTACT US
© DS STREAM. This job description was sourced from the employer's public career page. TheJob is not the employer — we index the posting and route candidates to the source. All content rights and hiring decisions belong to the employer.
FMCG,Retail,E-commerce,Healthcare,Telco,Finance,Logistics
About the Company DS Stream is a fast-growing AI & Data consulting company focused on analytics and data management for global brands. The company employs a team of over 200 senior experts who specialize in Data Science, Machine Learning, and MLOps. Their mission is to transform know-how into effective architectures for their clients. Products, Services & Tech Stack Core Solutions Data Engineering: Transforming disparate data into valuable insights for informed business decisions. AI Engineering: Providing modern AI technologies and custom AI tools to support AI-driven business decisions. MLOps: Integrating ML technologies with DevOps practices to automate Machine Learning workflows and improve efficiency. Technologies & Frameworks Specific platforms, languages, or tools are not mentioned in the provided text. Project Scope & Target Clients Client Base Industries Served: FMCG, retail, e-commerce, healthcare, telco, finance, logistics, and more. Engineering Scale The company delivers tailored and complex data analytics for global corporations. Projects include real-time intelligence for the longevity ecosystem, category management for FMCG, and sales automation for resort packages.
More at DS STREAM
All 15 rolesSenior DevOps/Platform Engineer
DS STREAM · India
Senior Data Engineer (Azure, Databricks)
DS STREAM · Warsaw
AI Engineer (AWS)
DS STREAM · Warsaw
Python Developer
DS STREAM · Warsaw