About the Position

We are seeking a Senior Specialist - Data Engineering to manage and optimize our data pipelines and infrastructure on leading cloud platforms.

Responsibilities

Data Pipeline Development & Maintenance: Design, build, and maintain highly scalable and efficient ETL/ELT data pipelines utilizing PySpark and Spark SQL for complex data transformations.
Cloud Data Infrastructure Management: Deploy, manage, and scale critical data infrastructure components on leading cloud platforms such as Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP).
Data Warehousing & Storage Optimization: Strategically manage data layout, partitioning, and indexing within Apache Hive and various cloud data lake solutions to optimize performance and accessibility.
Performance Tuning & Optimization: Proactively identify and resolve performance bottlenecks in Spark jobs leveraging Spark UI for in-depth analysis, effectively managing data skewness, and optimizing memory utilization.
Diverse Data Integration: Develop robust solutions for ingesting high-volume and diverse datasets from both structured relational databases and unstructured flat files into our data ecosystem.
Automated Workflow Orchestration: Implement and manage automated data workflows using industry-standard scheduling tools like Apache Airflow or platform-native schedulers, ensuring timely and reliable data delivery.
Strategic Collaboration: Partner closely with data scientists, business analysts, and cross-functional enterprise teams to translate complex business requirements into technically sound and efficient data solutions.

Mandatory Certification: Databricks Certified Associate Developer for Apache Spark
Skills:
- Proficiency in PySpark and Spark SQL
- Experience with ETL/ELT processes
- Knowledge of Apache Airflow or similar scheduling tools

[Company description if present]