Senior/Lead Data Quality Engineer
Primary stack
Nice to have's
Job description
About the Position
We are seeking a skilled Senior/Lead Data Quality Engineer to join our team. The successful candidate will play a crucial role in ensuring the integrity, reliability, and quality of our data engineering pipelines and platforms.
Responsibilities
- Design and develop a robust automated testing framework and strategy for data engineering pipelines, with a focus on integration with Databricks, PySpark, Scala, Spark SQL, and other relevant technologies
- Implement automated tests to ensure data integrity, quality, and reliability across all data pipelines and platforms
- Integrate automated testing within Azure DevOps CI/CD pipelines to ensure seamless testing, build, and deployment processes
- Conduct and execute performance tests to assess the efficiency and scalability of data processing systems
- Work closely with data engineers and other stakeholders to understand requirements and ensure testing strategies are aligned with business objectives
- Establish monitoring for automated tests, providing regular reports on test coverage, defects, and data quality issues
Requirements
- Minimum of 3 years of experience in Data Quality Engineering or a related field
- Solid understanding of automated testing frameworks and data engineering principles
- Proficiency with Databricks, PySpark, Scala, Spark SQL, Terraform, and Azure Event Hubs
- Experience in integrating testing frameworks with Azure DevOps and managing source control with Git
- Knowledge of pipeline configuration and deployments utilizing Terraform
- Familiarity with Jira
- Strong analytical abilities with a proven track record in troubleshooting and resolving complex data and testing challenges
- Fluency in English with a B2+ level
Nice to Have
- Experience with Power BI, Azure Data Lake, Spark Streaming, and other data processing and visualization tools
- Prior experience in a data-intensive industry or with large-scale data environments
- Proficiency in implementing solutions through Infrastructure as Code (IaC)
We Offer
- Opportunity to work remotely from Lviv, Ukraine
- Competitive contractor position
About the Company
EPAM Systems is a global software engineering and product development company, providing end-to-end software product engineering and technology consulting services.
© EPAM. This job description was sourced from the employer's public career page. TheJob is not the employer — we index the posting and route candidates to the source. All content rights and hiring decisions belong to the employer.
EPAM helps organizations innovate their business processes and rethink the way they manage their businesses so they can remain competitive in this new digital age.