Lead Data Engineer

Lead Data Engineer

The Lead data engineer role at our company presents an exciting opportunity to work on large-scale data infrastructure projects and contribute to the development and maintenance of our data platform. The ideal candidate will have a strong background in data engineering, data architecture, and cloud technologies. As a data engineer, you will collaborate with cross-functional teams including data scientists, analysts, and software engineers to build and optimize data pipelines, data integration processes, and data storage solutions.

Experience Level:

Candidates should have at least 5 years of experience in data engineering or a related field, with a solid understanding of cloud platforms and big data technologies. Experience leading a team of data engineers.

Core Skills:

1. Strong proficiency in Python or another programming language commonly used in data engineering, such as Java.
2. Excellent SQL skills with ability to work with data across different SQL databases including Postgres, Databricks, SQL Server etc. NoSQL databases is a plus

3.Knowledge of elastic Search/log stash implementation skills very desirable
4. Experience in building and optimizing ETL/ELT processes, data pipelines, and workflows, using tools like Apache Nifi, or Apache Kafka.
5. Familiarity with data modeling, data warehousing concepts, and data governance best practices.
6. Strong problem-solving and troubleshooting skills, with the ability to identify and resolve data quality and performance issues.
7. Experience with version control systems, such as Git, and knowledge of CI/CD practices in a data engineering context.
8. Knowledge of data security and privacy principles, as well as experience implementing data access controls and managing sensitive data.
9. Understanding of distributed computing principles and experience with distributed data processing frameworks like Apache Spark or Hadoop.
10. Familiarity with containerization technologies like Docker and orchestration tools like Kubernetes.
11. Excellent communication and collaboration skills, with the ability to work effectively in cross-functional teams and translate business requirements into technical solutions.

12. Mentoring more junior resources to keep our projects on track.

Core Responsibilities:

1. Design, develop, and maintain data pipelines, ETL/ELT processes, and data integrations to support efficient and reliable data ingestion, transformation, and loading.
2. Collaborate with API developers, and other stakeholders to understand data requirements and ensure the availability, reliability, and accuracy of the data.
3. Optimize and tune performance of data processes and workflows to ensure efficient data processing and analysis at scale.
4. Implement data governance practices, including data quality monitoring, data lineage tracking, and metadata management.
5. Work closely with infrastructure and DevOps teams to ensure the scalability, security, and availability of the data platform and data storage systems.
6. Continuously evaluate and recommend new technologies, tools, and frameworks to improve the efficiency and effectiveness of data engineering processes.
7. Collaborate with software engineers to integrate data engineering solutions with other systems and applications.
8. Document and maintain data engineering processes, including data pipeline configurations, job schedules, and monitoring and alerting mechanisms.
9. Stay up-to-date with industry trends and advancements in data engineering, cloud technologies, and data processing frameworks.
10. Provide mentorship and guidance to junior data engineers, promoting best practices in data engineering and ensuring the growth and development of the team.

11.Able to implement and troubleshoot Rest services in Python.