Big data engineer- Spark Scala- Visa

Job Title: Big Data Engineer - Spark/Scala - Visa

Location: Bengaluru, Karnataka, India

Job Type: Full-time

Experience: 6+ years

---

Job Description:

Key Responsibilities:

- Design, develop, and maintain big data processing pipelines using Apache Spark and Scala.

- Optimize and enhance the performance of existing big data solutions.

- Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver efficient solutions.

- Develop and maintain scalable data architectures and frameworks.

- Implement data processing workflows for batch and real-time data processing.

- Ensure data quality, integrity, and security across all data processing pipelines.

- Troubleshoot and resolve any issues related to data processing and analytics.

- Document technical solutions and maintain comprehensive documentation.

Required Qualifications:

- Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.

- 6+ years of experience in big data engineering.

- Extensive experience with Apache Spark and Scala.

- Strong understanding of big data technologies such as Hadoop, Hive, HBase, and Kafka.

- Proficient in SQL and experience with relational databases.

- Experience with cloud platforms like AWS, Azure, or Google Cloud.

- Strong problem-solving skills and the ability to troubleshoot complex issues.

- Excellent communication and collaboration skills.

- Experience with version control systems like Git.

Preferred Qualifications:

- Experience with data visualization tools such as Tableau, Power BI, or similar.

- Knowledge of machine learning frameworks and libraries.

- Familiarity with containerization and orchestration tools like Docker and Kubernetes.

- Experience with Agile methodologies and DevOps practices.