Position Overview:
We are seeking a highly skilled Senior Data Engineer to join our dynamic team. The ideal candidate will have extensive experience in building and optimizing data pipelines, data architecture, and data sets for optimal performance. The Senior Data Engineer will collaborate closely with cross-functional teams to design and implement scalable solutions for data ingestion, processing, and analysis.
Responsibilities:
- Design, build, and maintain robust data pipelines and ETL processes to support data integration, transformation, and storage.
- Develop and implement data architecture solutions using technologies such as Kafka, Hadoop, Hive, PySpark, and Cloudera.
- Optimize and tune data pipelines for performance, reliability, and scalability.
- Collaborate with data scientists, analysts, and software engineers to understand data requirements and deliver actionable insights.
- Work closely with infrastructure teams to manage on-premises data infrastructure and ensure high availability and security.
- Implement best practices for data governance, data quality, and data security.
- Evaluate and recommend new technologies, tools, and frameworks to enhance data engineering capabilities.
- Provide technical leadership and mentorship to junior members of the data engineering team.
Qualifications:
- Bachelors or masters degree in computer science, Engineering, or related field.
- 3+ years of experience in data engineering roles with a focus on building data pipelines and data infrastructure.
- Extensive hands-on experience with Kafka, Hadoop, Hive, PySpark, and Cloudera.
- Proficiency in Python programming language for data manipulation, scripting, and automation.
- Strong understanding of distributed systems, data modeling, and database concepts.
- Experience working with on-premises data infrastructure and cloud platforms (e.g., AWS, Azure, GCP).
- Streaming experience Spark streaming, Flink.
- Extensive hands-on experience in coding Java/Scala/Python
- Need to have experience in Linux/Scripting
- Strong understanding of Data Modelling
- Excellent problem-solving skills and ability to troubleshoot complex data issues.
- Strong communication and collaboration skills with the ability to work effectively in cross-functional teams.
- Proven track record of delivering high-quality solutions on time and within budget.
