Lead Data Engineer
Primary stack
Amazon AthenaAirflowDevOpsPandasPythonAgileAI
Nice to have's
GitHubCI/CDAIAgile
Job description
About the Position
We're seeking a hands-on Lead Data Engineer to construct and maintain scalable data pipelines along with cloud-based data solutions. The right person for this role brings substantial background in Python, PySpark, AWS, Airflow, and DevOps, with particular attention to production-grade engineering and automation practices.
Skills
- Data Software Engineering
- Amazon Athena
- Amazon Managed Workflows for Apache Airflow
- DevOps
- Pandas
- PySpark
- Python for Data Integration
- Agile
- Continuous Integration Development & Maintenance
- Gen AI Application Development
- Gen AI in SDLC
Responsibilities
- Create, construct, and sustain scalable data pipelines running on AWS infrastructure
- Build dependable data processing solutions leveraging Python, Pandas, and PySpark
- Establish and oversee workflows via AWS Managed Workflows for Apache Airflow (MWAA)
- Craft and fine-tune data queries through AWS Athena to enable efficient analytical work
- Streamline infrastructure setup and deployment workflows through automation using Terraform and GitHub Actions
- Put in place testing, monitoring, troubleshooting practices, and data-quality safeguards to keep pipelines dependable
- Take part in reviewing code and help promote sound engineering practices throughout the team
- Work alongside engineering, product, and business groups within an Agile setting
Requirements
- At least 5 years of applicable professional background
- A minimum of one year spent leading and managing development teams
- Solid foundation in core Python programming applied to data engineering work
- Practical experience applying Python Pandas for manipulating and analyzing data
- Strong capability with PySpark for handling large-scale distributed data processing
- Background working with AWS MWAA or Apache Airflow to coordinate and manage data workflows
- Familiarity with AWS Athena for running queries and analyzing data housed within AWS
- Hands-on background using Terraform to automate infrastructure setup and management
- Working knowledge of GitHub Actions for constructing automated CI/CD pipelines
- Good grasp of DevOps principles as applied within data engineering contexts
- Strong capabilities in data engineering, troubleshooting, and resolving complex problems
- Upper-intermediate English language skills or higher (B1)
Nice to Have
- Background applying GitHub Copilot to assist with development, code reviews, testing, and release processes
- Knowledge of CI/CD methodologies and associated tools for smoother software delivery
- Background working within Agile/Scrum team structures
- Experience gained within financial services or investment management sectors
- Background in Gen AI application development and incorporating Gen AI practices into the software development lifecycle (SDLC)
We Offer
- International projects with top brands
- Work with global teams of highly skilled, diverse peers
- Healthcare benefits
- Employee financial programs
- Paid time off and sick leave
- Upskilling, reskilling, and certification courses
- Unlimited access to the LinkedIn Learning library and 22,000+ courses
- Global career opportunities
- Volunteer and community involvement opportunities
- EPAM Employee Groups
- Award-winning culture recognized by Glassdoor, Newsweek, and LinkedIn
Locations
- Argentina
- Brazil
- Colombia
- Mexico
Work Location Type
Remote
© EPAM. This job description was sourced from the employer's public career page. TheJob is not the employer — we index the posting and route candidates to the source. All content rights and hiring decisions belong to the employer.
EPAM helps organizations innovate their business processes and rethink the way they manage their businesses so they can remain competitive in this new digital age.