This is a remote position.
We are seeking a highly skilled Senior Spark Engineer with expertise in Python, Spark, Databricks, and AWS to join our dynamic team. The ideal candidate will possess a deep understanding of Spark architecture and be proficient in fine-tuning Spark jobs for optimal performance. Additionally, strong knowledge of software engineering best practices, DevOps principles including CI/CD pipelines, Kubernetes, and either Jenkins, Airflow, or SageMaker, is essential for this role. Experience in MLOps is desirable but not mandatory.
Requirements
- Proficiency in Spark, Python, Databricks, and AWS services.
- 7-10 years of experience
- Lead the design, development, and implementation of Spark-based solutions for complex data processing tasks.
- Collaborate with cross-functional teams to gather requirements and translate them into technical specifications.
- Implement best practices in software engineering, DevOps, and MLOps methodologies.
- Design and maintain data pipelines for efficient data processing and analysis.
- Perform data modeling and schema design to support business requirements.
- Mentor junior team members and provide technical guidance as needed.
- Stay updated with the latest advancements in Spark, Python, and related technologies, and advocate for their adoption when appropriate.
- Bachelor's degree in Computer Science, Engineering, or a related field. Master's degree preferred.
- Strong understanding of software engineering principles and best practices.
- Hands-on experience with DevOps tools and practices including CI/CD pipelines, Kubernetes, and either Jenkins, Airflow, or SageMaker.
- Experience with MLOps practices is a plus.
- Solid understanding of data engineering concepts and data modeling techniques.
- Excellent problem-solving skills and attention to detail.
- Ability to work independently and lead tasks effectively.
- Strong communication and interpersonal skills.
