Big Data Engineer
Primary stack
Nice to have's
Job description
About the Position
DataLabs is seeking a Big Data Engineer with expertise in SQL, Java, Python, and Scala to join our team in Ukraine. We craft beautiful work using the latest technology and solve business problems that lead to satisfaction not just for our customers - but for the design team as well. From developing a BI solution that provides impactful business value, to producing a critical sales presentation on a tight timeline, we bring together gifted engineers to deliver solutions that actually matter.
Responsibilities
- Research, design, and development of appropriate algorithms for collecting, processing, and analyzing extensive data;
- Tools and platforms integration for working with Big Data;
- Identification of external and internal Big Data sources for analytical work;
- Obtaining, processing large data amounts from heterogeneous data sources, developing methods and regulations for the necessary ETL processes;
- Development of a scalable and reliable data processing infrastructure to meet the needs;
- Identifying and developing selective data pipelines to reconcile disparate data sources;
- Data processing structure development for receiving, cleaning, reconciling, and storing processed data in a data lake or data warehouse;
- Data quality problems identification and elimination;
- Platform development to automate code deployment and integration;
- High-quality code and libraries development;
- CI/CD pipelines and version control environment development;
- Necessary analytical Big Data reports, cuts, and aggregates development;
- Data lifecycle management (processes of obtaining, placing, storing, distributing, migrating, archiving, and deleting extensive data);
- High-level goals transformation into achievable, testable, and measurable functional specifications;
- Development of solutions that meet security and data management policies;
- Providing access to data;
- Anomaly detection and auditing, performance monitoring.
Requirements
- Expertise in Big Data;
- Expertise in programming languages (SQL, Java, Python, Scala, Shell, etc.);
- Expert knowledge and understanding of technologies (Spark, Databricks, etc.);
- Expertise in building data warehouses and data mining, processing, and transformation processes;
- Expertise in data modeling and data analysis;
- Expertise in integrating data of different formats and from several sources;
- Expertise in databases (Postgres, MongoDB, etc.);
- Data management tools (Apache, etc.);
- English - Intermediate and above;
- Expert knowledge of Agile/Scrum and project management principles (will be beneficial);
- High motivation;
- Strong communication and teamwork skills.
We Offer
- A range of benefits for full-time employees;
- A professional development program.
About the Company
We craft beautiful work using the latest technology and solve business problems that lead to satisfaction not just for our customers - but for the design team as well. From developing a BI solution that provides impactful business value, to producing a critical sales presentation on a tight timeline, we bring together gifted engineers to deliver solutions that actually matter.
© DataLabs. This job description was sourced from the employer's public career page. TheJob is not the employer — we index the posting and route candidates to the source. All content rights and hiring decisions belong to the employer.
DataLabs — консалтингова компанія, яка надає рішення на основі BI, орієнтовані на успішне управління бізнесом.Ми розробляємо комплексні рішення для вилучення, аналізу, очищення та візуалізації великих обсягів даних із різних джерел. Розкриваючи весь потенціал клієнтських даних, ми робимо їх гнучкими та динамічними, таким чином перетворюючи на цінну інформацію.
