About the Position

IT Teams is a Romanian company specialized in software outsourcing and remote staffing. We aim to assemble passionate experts to help companies and startups achieve success. We deliver the perfect combination of technical skills, methodology, and high-speed delivery techniques to help startups and established companies extend their development teams.

We are looking for a Senior Data Engineer to develop, implement, and optimize complex DataWarehouse (DWH) and Data Lakehouse solutions using the Databricks platform (including Delta Lake, Unity Catalog, and Spark) to ensure a scalable, high-performance, and governed data foundation for analytics, reporting, and machine learning.

Apply now »

Responsibilities

Databricks Development and Architecture

Advanced Design and Implementation: Design and implement robust, scalable, and high-performance ETL/ELT data pipelines using PySpark/Scala and Databricks SQL on the Databricks platform.
Delta Lake: Expertise in implementing and optimizing the Medallion architecture (Bronze, Silver, Gold) using Delta Lake to ensure data quality, consistency, and historical integrity.
Lakehouse Platform: Efficient implementation of the Lakehouse architecture on Databricks, combining best practices from DWH and Data Lake.
Performance Optimization: Optimizing Databricks clusters, Spark operations, and Delta tables (e.g., Zordering, compaction, query tuning) to reduce latency and compute costs.
Streaming: Design and implementation of real-time/near-real-time data processing solutions using Spark Structured Streaming and Delta Live Tables (DLT).

Governance and Security

Unity Catalog: Implementing and managing Unity Catalog for centralized data governance, fine-grained security (row/column-level security), and data lineage.
Data Quality: Defining and implementing data quality standards and rules (e.g., using DLT or Great Expectations) to maintain data integrity.

Operations and Collaboration

Orchestration: Developing and managing complex workflows using Databricks Workflows (Jobs) or external tools (e.g., Azure Data Factory, Airflow) to automate pipelines.
DevOps/CI/CD: Integrating Databricks pipelines into CI/CD processes using tools such as Git, Databricks Repos, and Bundles.
Collaboration: Working closely with Data Scientists, Analysts, and Architects to understand business requirements and deliver optimal technical solutions.
Mentoring: Providing technical guidance (mentorship) to junior developers and promoting best practices.

Requirements

Required Knowledge (Senior Level)

Databricks Platform: Demonstrated, expert-level experience with the entire Databricks ecosystem (Workspace, Cluster Management, Notebooks, Databricks SQL).
Apache Spark: In-depth knowledge of Spark architecture (RDD, DataFrames, Spark SQL) and advanced optimization techniques.
Delta Lake: Expertise in implementing and administering Delta Lake (ACID properties, Time Travel, Merge, Optimize, Vacuum).
Programming Languages: Python (with PySpark) and/or Scala (with Spark) at an advanced/expert level.
SQL: Advanced/expert skills in SQL and Data Modeling (Dimensional, 3NF, Data Vault).
Cloud: Solid experience with a major cloud platform (AWS, Azure, or GCP), particularly with storage services (S3, ADLS Gen2, GCS) and networking.

Additional Knowledge (Major Advantage)

Unity Catalog: Hands-on experience with implementing and administering Unity Catalog.
Lakeflow: Experience with Delta Live Tables (DLT) and Databricks Workflows.
ML/AI Fundamentals: Understanding of basic MLOps concepts and experience with MLflow to facilitate integration with Data Science teams.
DevOps: Experience with Terraform or equivalent for Infrastructure as Code (IaC).
Certifications: Databricks certifications (e.g., Databricks Certified Data Engineer Professional) are a significant advantage.

Education and Experience

Education: Bachelor’s degree in Computer Science, Engineering, Mathematics, or a relevant technical field.
Professional Experience: At least 5+ years of experience in Data Engineering, including at least 3+ years working with Databricks and Spark at scale.

We Offer

Hybrid Work Environment: Flexible remote and on-site options.
Full-time Position: Commitment to a full-time role.
Competitive Salary: Attractive compensation package.
Professional Growth: Opportunities for career advancement and skill development.

About the Company

We are driven by curiosity, ensuring that the technology we deliver stays in tune with our customers' business goals. We encourage our team members to speak up, share their advice, and voice any concerns, enabling us to handle software development project risks effectively. Honesty is a key ingredient of our collaboration with customers and colleagues. Our commitment to customer goals is reflected in our regular progress reporting, constant revision of project goals and deadlines, and solid quality control.