Senior Data Integration Engineer (Azure)
Primary stack
Nice to have's
Job description
About the Position
We are seeking a Senior Data Integration Engineer (Azure) to build and run reliable data pipelines on Microsoft Azure. You will use Azure Databricks, PySpark, Spark SQL, Delta Lake, and Azure Data Factory to support reporting and data products using a Medallion Architecture approach. You own delivery, quality, and secure operations.
Responsibilities
- Build and maintain data pipelines using Azure Databricks, PySpark, and Spark SQL
- Implement Medallion Architecture patterns using Delta Lake tables and workloads
- Develop and schedule orchestration workflows in Azure Data Factory
- Integrate data from Azure Data Lake Storage (ADLS) Gen2, Azure SQL Database, and REST application programming interfaces (APIs)
- Implement secret management using Azure Key Vault and Databricks secret scopes
- Maintain continuous integration and continuous delivery (CI/CD) using GitHub Actions and GitHub
- Support reporting enablement and dataset operations in Power BI Service
- Apply security best practices across Azure and Databricks using service principals and role-based access control (RBAC)
Requirements
- Hands-on data engineering experience in production environments
- Strong knowledge of Apache Spark, Spark SQL, and Delta Lake
- Experience building pipelines with Medallion Architecture concepts
- Advanced Structured Query Language (SQL) skills, including common table expressions (CTEs), MERGE, window functions, and complex transformations
- Python scripting capability for data integration, including REST API calls
- Experience with Azure services such as Azure Data Factory, ADLS Gen2, Azure SQL Database, Azure Synapse Analytics, Azure Key Vault, and Azure Identity
- Knowledge of dimensional modeling, including fact tables, dimension tables, and star schemas
- Experience implementing data quality checks, logging, and monitoring
Nice to Have
- Experience with Microsoft Fabric
About the Company
EPAM Systems is a global software engineering and product development company. We partner with the world’s leading brands to deliver innovative digital solutions and drive their digital transformation.
We Offer
- Opportunity to work in a hybrid model
- Competitive compensation package
- Professional growth and development opportunities
- Collaborative and inclusive work environment
© EPAM. This job description was sourced from the employer's public career page. TheJob is not the employer — we index the posting and route candidates to the source. All content rights and hiring decisions belong to the employer.
EPAM helps organizations innovate their business processes and rethink the way they manage their businesses so they can remain competitive in this new digital age.