Job Title: Lead Data Engineer
Location: Bengaluru, Karnataka, India
Experience: 7-10 years
Key Responsibilities:
Technical Leadership:
- Lead and drive the delivery of scalable data and analytics solutions.
- Lead a team of junior and senior data engineers.
- Participate in solution design and architecture discussions.
- Design, implement, and integrate new technologies.
- Evolve data and analytics products.
- Contribute to all aspects of data engineering from ingestion, and transformation, to consumption.
- Build reusable frameworks, automated workflows, and libraries at scale to support analytics products.
Collaboration:
- Work closely with internal partners, product owners, engineering leaders, data analysts, big data leads, and engineers.
- Build positive relationships across product and engineering.
Solution Development:
- Design and build reusable components, frameworks, and libraries at scale to support analytics products.
- Implement complex automated workflows and routines using workflow scheduling tools.
- Build continuous integration, test-driven development, and production deployment frameworks.
- Clean, prepare, and optimize data at scale for ingestion and consumption.
Data Management:
- Anticipate, identify, and solve issues concerning data management to improve data quality.
- Drive the implementation of new data management projects and restructure the current data architecture.
- Analyze and profile data for the purpose of designing scalable solutions.
- Troubleshoot complex data issues and perform root cause analysis.
Mentorship:
- Mentor and develop other data engineers in adopting best practices.
- Drive collaborative reviews of design, code, test plans, and dataset implementation.
Required Skills and Experience:
Technical Expertise:
- Minimum 8 years of experience developing scalable Big Data applications or solutions on distributed platforms.
- Strong skills in SQL, Spark, Python, Airflow, and Scala.
- Experience with cloud computing platforms (AWS, GCP, Azure).
- Experience with Jenkins pipelines for CI/CD processes and building Docker images.
- Working knowledge of data warehousing, data modeling, governance, and data architecture.
- Experience with data platforms such as EMR, Airflow, and Databricks (Data Engineering Delta Lake components, and Lakehouse Medallion architecture).
- Experience with data warehousing tools like SQL database, Presto, and Snowflake.
- Experience with streaming, serverless, and microservices architecture.
Soft Skills:
- Excellent problem-solving and analytical skills.
- Strong communication and collaboration skills.
- Ability to influence and communicate effectively with team members and business stakeholders.
- Ability to work independently and as part of a global team.
- Self-motivated and able to work in a fast-paced environment.
- Detail-oriented and committed to delivering high-quality work.
- Strong skills in building positive relationships across product and engineering.
