Senior Big Data Software Engineer
Primary stack
KafkaPythonAWSSQL
Nice to have's
Database AdministrationAWS RedshiftPySparkScalability
Job description
Description language:
About the Position
EPAM is seeking a Senior Big Data Software Engineer to develop and support internal data storage systems for our client's products and applications.
Responsibilities
- Development and support of complex data storage systems for processing large volumes of data
- Designing and optimizing database architecture and data schemas
- Developing and implementing efficient SQL queries for data analysis and processing
- Developing and implementing software solutions using the Hadoop ecosystem (Yarn, MapReduce, Hive, HDFS)
- Using Spark and Kafka for real-time data processing and transmission
- Configuring and managing AWS cloud services (EMR, S3, Lambda)
- Optimizing existing solutions to reduce time and resource consumption
- Collaborating with other engineers and teams to create scalable and efficient solutions
- Performing data analysis, creating reports, and providing recommendations for system improvements
- Identifying and implementing best practices to ensure data security and integrity
- Participating in the preparation of technical documentation and maintaining user documentation
Requirements
- More than 4 years of experience in database development and database design
- SQL expert with at least 2+ years of experience
- Good Python development skills
- Software development experience using components of the Hadoop ecosystem (Yarn, MapReduce, Hive, HDFS)
- Practical experience with Spark and Kafka
- Experience with AWS cloud services (EMR, S3, Lambda)
- Excellent analytical and problem-solving skills
- Good communication and presentation skills
- Attention to detail
- Ability to meet tight deadlines and prioritize tasks
Nice to Have
- Experience in database administration
- Experience with AWS Redshift
- Practical experience with PySpark
- Understanding of service/application scaling approaches
About the Company
[Company description if present]
© EPAM. This job description was sourced from the employer's public career page. TheJob is not the employer — we index the posting and route candidates to the source. All content rights and hiring decisions belong to the employer.
EPAM helps organizations innovate their business processes and rethink the way they manage their businesses so they can remain competitive in this new digital age.