Big Data Developer

Job Role: Big Data Engineer

Position: Big Data Engineer

Relevant Experience 3+ years

Location: Navi Mumbai /Bengaluru/Pune

Notice Period: Immediate or serving Notice Period

Primary Skills: Big Data, PySpark, Cloud- Azure or AWS, Databricks, SQL, Python

Overview: We are seeking a talented Big Data Engineer proficient in PySpark to join our dynamic team. The ideal candidate will play a pivotal role in designing, implementing, and maintaining scalable data solutions leveraging the Big Data technologies like PySpark. This role requires a strong understanding of big data technologies, data engineering best practices, and the ability to collaborate effectively with cross-functional teams

Responsibilities:

1. Design, develop, and implement end-to-end data pipelines using PySpark to process, cleanse, and transform large-scale datasets.

2. Collaborate with data scientists, analysts, and stakeholders to understand data requirements and translate them into technical solutions.

3. Optimize and tune Spark jobs for performance and scalability, ensuring efficient resource utilization and processing speed.

4. Develop and maintain data ingestion processes from various sources such as databases, streams, and file systems into the Databricks environment.

5. Implement data quality checks and validation processes to ensure the accuracy, completeness, and reliability of the data.

6. Have worked on Data Orchestration Tools like Airflow.

7. Working on storytask using Agile Methodology.

8. Must have hand-on on GitHub and GitLab

9. Work closely with DevOps teams to automate deployment processes and ensure the reliability and availability of data pipelines.

10. Monitor, troubleshoot, and resolve issues related to data pipeline failures, performance degradation, and system errors.

11. Stay updated on emerging technologies and industry trends in big data, distributed computing, and cloud platforms to drive continuous improvement and innovation.

12. Implementing Data Solutions by using Azure Services like Azure Data Factory.

13. Creating Data Pipelines using Notebooks using different Clusters in Databricks.

Qualifications:

1. Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

2. Proven experience as a Big Data Engineer with expertise in building and optimizing data pipelines using PySpark.

3. Strong programming skills in Python and proficiency with SQL for data manipulation and analysis.

4. Familiarity with distributed computing frameworks like Hadoop, Scala.

5. Hands-on experience with cloud platforms such as Azure or AWS.

6. Working experience on Databricks

7. Excellent problem-solving skills and the ability to work independently as well as collaboratively in a fast-paced environment.

8. Effective communication skills with the ability to explain complex technical concepts to non-technical stakeholders.

Benefits You'll Get

Unlimited opportunities to learn onourmultiple Training Platforms

Certifications Reimbursement

Flexibility

Opportunity towork onmultiple technologies

Medical Coverage &Life Insurance

Company Events and Outings

Tech Thursdays and Fun Fridays

5 days working

Work-Fun Environment