Job Description, Responsibilities & Requirements
About the Position
We are seeking a skilled Big Data DevOps Engineer with profound knowledge of cloud technologies and experience with a variety of data tools and platforms. The ideal candidate will have proven experience in managing and optimizing big data operations across cloud environments.
Responsibilities
- Cloud Solutions Development: Build, deploy, and maintain scalable and secure big data solutions across major cloud platforms (AWS, Azure, Google Cloud) to support advanced data processing and analytics.
- Data Tools Management: Configure and manage data processing tools such as Apache Spark, Kafka, Airflow, and Databricks to efficiently handle large-scale data workflows.
- Automation and CI/CD: Utilize Terraform for infrastructure as code, automate workflows with Jenkins, and maintain code with GitLab for continuous integration and continuous delivery processes.
- Performance Optimization: Monitor and optimize the performance of big data tools and cloud platforms, ensuring cost-effectiveness and operational efficiency.
- Security and Compliance: Implement and enforce data security measures, ensuring compliance with data protection regulations and best practices in cloud environments.
- Collaborative Development: Work collaboratively with data scientists, data engineers, and other stakeholders to understand requirements and deliver high-quality data solutions.
Requirements
- Strong experience with big data technologies, particularly Apache Spark or Databricks.
- Extensive background in cloud services administration with platforms like AWS, Azure, or Google Cloud.
- In-depth knowledge of infrastructure as code using Terraform.
- Proficiency in implementing CI/CD pipelines with tools such as Jenkins, GitLab or GitHub Actions.
- Solid scripting skills in Python, Scala, or similar languages.
- Experience with SQL and relational/non-relational databases.
- Excellent problem-solving abilities and a team-oriented attitude.
Nice to Have
- Certifications in cloud technology (e.g., AWS Certified Solutions Architect, Google Cloud Certified Professional Cloud Architect).
- Additional experience with real-time data processing tools like Apache Kafka.
About the Company
[Company description if present]