Data Engineer

Job Title: Data Engineer

• Location: Gurugram, Delhi

Experience: 3-5 Years

• Budget

: 24 LPAWork Mode: Work From Office (WFO)

• Interview Mode: Virtual

Must-Have Skills: PySpark, Databricks, ML Flow, Azure DevOps, SQL

• Job Description

:
We are seeking a skilled Data Engineer to design, build, and maintain data pipelines, data architectures, and ETL processes using advanced tools and technologies. The ideal candidate will have hands-on experience with PySpark, Databricks, and Azure DevOps, and will be proficient in building scalable solutions for data processing and analytics.

As a Data Engineer, you will collaborate with global teams, including data science and business intelligence, to ensure efficient data handling and gain insights that drive business decisions.

• Key Responsibilities:

Data Pipeline Development: Design and build data pipelines using Spark-SQL and PySpark within Azure Databricks to handle large datasets efficiently.

• ETL Pipeline Design: Develop ETL pipelines using Azure Data Factory (ADF) for robust data transformation and integration.

Lakehouse Architecture: Build and maintain a Lakehouse architecture in ADLS/Databricks, ensuring scalable and efficient data storage and processing.

Data Preparation: Perform data preparation tasks including cleaning, normalization, deduplication, and type conversion for optimized data usability.Collaboration with DevOps: Work closely with the DevOps team to deploy data solutions in production environments using Azure DevOps.

• Error Handling: Monitor data processes, identify errors, and execute corrective actions, ensuring data integrity. This includes troubleshooting and resolving root causes of data errors.

Global Team Collaboration: Participate as a member of the global analytics team, collaborating on projects, sharing insights, and adopting best practices from data science and business intelligence colleagues.

Project Leadership: Lead and participate in projects, applying change management tools to manage upgrades, changes, and data migrations. This includes training, documentation, and communication efforts.

• Qualifications

:3-5 years of experience in data engineering, with a strong focus on PySpark, Databricks, and Azure DevOps.

• Expertise in designing and building

ETL pipelines using ADF. Strong knowledge of Lakehouse architecture and ADLS/Databricks.

• Solid experience with SQL for database queries and data management.

* Proven ability to collaborate with global teams and lead data projects.

• Strong problem-solving skills and attention to detail in managing data processes and errors.

Job Type: Full-time

Pay: 2,000,

• 00 - 2,400,000.00 per year
  
  Schedule:
• Day shift

Work Location: In person