Senior Data Engineer Lead - DataOps/Data Pipeline
About the Role:
We are seeking a highly skilled and experienced Senior Data Engineer Lead to join our growing team and play a pivotal role in building and maintaining our robust DataOps/Data Pipeline infrastructure. You will lead a team of engineers to design, develop, and implement scalable and reliable data pipelines that power our business intelligence, machine learning, and other data-driven initiatives.
Responsibilities:
- Lead and mentor a team of Data Engineers: Guide, coach, and empower team members to deliver high-quality data solutions.
- Architect and build robust data pipelines: Design, develop, and maintain high-performance, scalable data pipelines using cloud-based technologies (e.g., AWS/ Azure/ GCP) and tools like Apache Spark, Apache Kafka.
- Master data ingestion, transformation, and loading (ETL): Leverage your expertise in ETL processes to design efficient data pipelines that meet the specific needs of the business.
- Embrace data streaming technologies: Work with tools like Apache Kafka to handle real-time data and build dynamic, responsive pipelines.
- Champion data quality: Implement and maintain data quality standards to ensure accuracy, consistency, and completeness across all pipelines.
- Collaborate with stakeholders: Work closely with business users, data scientists, and other stakeholders to understand their data requirements and translate them into technical solutions.
- Contribute to the development of the DataOps strategy: Participate in strategic discussions and help define and implement best practices for data management and pipeline development.
- Stay at the forefront of data technologies: Continuously learn and adopt new technologies and methodologies to enhance efficiency and effectiveness.
Qualifications:
- Proven Data Engineering Expertise: 8+ years of experience as a Data Engineer or related role with a strong track record of building and maintaining DataOps/Data Pipeline infrastructure.
- Programming Prowess: Sound experience in SQL and at least one programming language like Python, Java, or Scala.
- Data Pipeline Mastery: Experience in implementing data pipelines, handling data transformations, and writing code for data processing and analysis.
- In-depth Data Understanding: Solid grasp of large-scale data sets, including both structured and unstructured data.
- Open-Source Tool Experience: Knowledge of and experience leveraging open-source tools like Airbyte and Mage AI for data integration and pipeline development.
- Cloud Platform Knowledge: Strong understanding of cloud-based technologies (AWS, Azure, GCP) and associated data services.
- Containerization and Orchestration: Knowledge of containerization and orchestration tools like Docker and Kubernetes.
- Communication and Collaboration: Excellent communication and interpersonal skills, with the ability to work effectively within a team.
- Problem-Solving Mindset: Strong analytical and problem-solving skills, with a passion for solving real-world problems using data.
