Job Description, Responsibilities & Requirements
About the Position
We are seeking a skilled Lead Python Data Engineer to join our team and help us build and maintain our data pipelines.
As a Python Data Engineer, you will be responsible for designing, developing, and testing Python-based data pipelines and performing data cleansing, transformation, and quality assurance.
Responsibilities
- Design, develop, test, and maintain Python-based data pipelines
- Write high-quality and maintainable Python code to migrate data between different storage systems with modifications
- Create ETL jobs, airflow operators, and Spark jobs to enhance data processing capabilities
- Implement Data Intake processes for Data Sources (ETL/ELT, DWH Modelling and Optimization)
- Perform data cleansing, transformation, and quality assurance to maintain data integrity
- Optimize data processing, storage, and retrieval to improve performance
- Collaborate closely with cross-functional teams to ensure data accuracy and consistency
Requirements
- In-depth knowledge of Python programming with the ability to write maintainable and scalable code
- Strong understanding of distributed computation and multiprocessing
- Knowledge of data warehouse concepts and ETL processes
- Proficiency in working with state machines, web back-end development, networking, and protocols
- Experience with cloud computing platforms, such as AWS or GCP, and tools like Step Functions, Batch
- Knowledge of Spark, Hive, Databricks, and other data processing frameworks
- Familiarity with data orchestration tools like Airflow
- Expertise in SQL, NoSQL, and cloud-based data storage solutions
Location
- Office in Kazakhstan
About the Company
[Company description if present]