Lead Data Engineer

Responsibilities:

Collaborate with cross-functional teams to ensure quality and timely delivery of products.

Identify areas of improvement in data quality processes and propose solutions to enhance data accuracy and reliability and create test strategies.

Collaborate with the Data Delivery technology teams and architects to define and develop solutions.

Collaborate with data engineering and development teams to implement data quality best practices and optimize data workflows.

Develop and execute Data Quality frameworks, solutions and automated test cases for software applications to ensure the reliability of data pipelines and data transformations.

End to end integration testing between multiple independent systems and interfaces (flat files, APIs (ETL)) etc.

Utilize AWS services (EC2, Glue, Athena etc. to test data integration and migration processes.

Conduct functional, integration, regression, and performance testing of database systems using industry standard technologies (SQL, NOSQL, Python, etc.)

Knowledge, Skills and Abilities:

Educational Qualification : Bachelor's/ Master's Degree in Computer Science.

7-8 years of experience in Data Engineering, Data Quality and Data Validation.

Hands-on experience in Python and libraries like Pandas, Spark, NumPy.

Working experience with Relational SQL and NoSQL databases.

Strong understanding of Data Warehouse/Data Lake Architecture and Development and data pipelines.

Experience with data processing concepts like mapping documents and complex data relationships.

Experience with requirement analysis, defect tracking, coordinating with team members in different locations, and test reporting and signoff.

Strong analytical and technical skills to address issues.

Detailed and organized approach to solving problems.

Location:

Work from office (Koregaon Park, Pune)