Experience Level: 10-12 years
Job Summary: We are seeking an experienced AWS Data Engineer/Lead Developer to join our team. The ideal candidate will have 10-12 years of experience in managing, developing, and optimizing data pipelines, with expertise in AWS services and data warehousing solutions. You will work with advanced technologies such as AWS Glue, S3, Lambda, PySpark, and databases like Aurora DB, Dynamo DB, and Redshift, while also overseeing CI/CD processes and scheduling tools like Stone branch.
Key Responsibilities:
- Design, develop, and optimize large-scale data pipelines using AWS services such as S3, Glue, Lambda, and Redshift.
- Build and maintain scalable data architectures and ETL processes.
- Develop solutions for data warehousing using Redshift and Aurora DB.
- Implement CI/CD pipelines for automated data integration and deployment.
- Collaborate with cross-functional teams to gather data requirements and translate them into technical specifications.
- Utilize PySpark for distributed data processing and implement data transformation processes.
- Manage data stores, including Aurora DB, Dynamo DB, and Redshift.
- Leverage Stone branch for job scheduling and monitoring.
- Ensure data security, performance optimization, and system reliability.
Key Requirements:
- 10-12 years of hands-on experience as a Data Engineer or in a similar role.
- Proficiency in AWS services, including S3, Glue, Lambda, and Redshift.
- Strong experience with SQL and data warehousing solutions.
- Expertise in PySpark for distributed data processing.
- Familiarity with CI/CD processes and tools.
- Experience with Aurora DB, Dynamo DB, and Redshift.
- Knowledge of job scheduling tools like Stone branch is preferred.
- Strong problem-solving skills and ability to work independently.
- Excellent communication and collaboration skills.
Preferred Qualifications:
- AWS certification is a plus.
- Experience in handling large-scale data sets and performance optimizat
