Senior/Lead Data Quality Engineer
Primary stack
Azure DevOpsScalaDatabricksAzure Event HubsGitTerraformSQL
Nice to have's
Power BIAzure Data LakeSpark StreamingJiraInfrastructure as Code (IaC)
Job description
Description language:
About the Position
Our team is looking to hire a talented Senior/Lead Data Quality Engineer. In this role, you will be instrumental in safeguarding the integrity, reliability, and quality of our data engineering platforms and pipelines.
Responsibilities
- Build and architect a strong automated testing strategy and framework for data engineering pipelines, emphasizing integration with Databricks, PySpark, Scala, Spark SQL, and other pertinent technologies
- Deploy automated tests that guarantee data quality, integrity, and reliability across every data pipeline and platform
- Embed automated testing into Azure DevOps CI/CD pipelines to enable smooth testing, build, and deployment workflows
- Carry out performance testing to evaluate the scalability and efficiency of data processing systems
- Collaborate closely with data engineers and additional stakeholders to gather requirements and align testing strategies with business goals
- Set up monitoring for automated tests and deliver routine reports covering test coverage, defects, and data quality concerns
Requirements
- At least 3 years of experience working in Data Quality Engineering or an adjacent field
- Strong grasp of data engineering principles and automated testing frameworks
- Hands-on expertise with Databricks, PySpark, Scala, Spark SQL, Terraform, and Azure Event Hubs
- Background in connecting testing frameworks to Azure DevOps and handling source control through Git
- Understanding of pipeline configuration and deployment using Terraform
- Working knowledge of Jira
- Excellent analytical skills with a demonstrated history of diagnosing and resolving complex testing and data issues
- English proficiency at a B2+ level
Nice to Have
- Exposure to Power BI, Azure Data Lake, Spark Streaming, and additional data visualization and processing tools
- Previous work in a data-heavy industry or within large-scale data environments
- Skill in delivering solutions via Infrastructure as Code (IaC)
Location
- Remote in Ukraine: Lviv
© EPAM. This job description was sourced from the employer's public career page. TheJob is not the employer — we index the posting and route candidates to the source. All content rights and hiring decisions belong to the employer.
EPAM helps organizations innovate their business processes and rethink the way they manage their businesses so they can remain competitive in this new digital age.