Job Summary:As an AWS Data Engineer, you will work on designing, developing, and maintaining our data infrastructure. This includes data pipelines, data lakes, and ETL processes on AWS, primarily leveraging AWS Glue. You will collaborate with data scientists, analysts, and other engineers to ensure data is readily accessible, accurate, and efficiently processed for analytics and business insights.
Key Responsibilities:
- Design, implement, and manage ETL/ELT pipelines using AWS Glue to process large datasets
- Build and maintain scalable data pipelines and data workflows on AWS, including ingestion, transformation, and storage solutions
- Develop and optimize AWS Glue scripts, jobs, crawlers, and workflows for seamless data integration and processing
- Work with stakeholders to understand data requirements and design data models for optimal storage, access, and performance
- Perform data validation and ensure data quality across pipelines
- Monitor and troubleshoot issues related to data ingestion, data transformations, and pipeline failures
- Optimize data processing and storage on AWS, leveraging services like S3, Redshift, and RDS alongside AWS Glue
- Implement data governance, security, and compliance policies for data pipelines
- Collaborate with the data engineering team to drive best practices in data management and infrastructure
Required Qualifications:
- Bachelor's degree in Computer Science, Engineering, or a related field advanced degrees preferred
- 3+ years of experience as a Data Engineer, with a focus on cloud data engineering solutions on AWS
- Proficiency in AWS Glue, including creating and managing Glue jobs, crawlers, and workflows
- Strong understanding of data lake architecture, data warehousing, and ETL processes
- Proficiency in Python and SQL, with experience in writing scripts for data transformation
- Experience with AWS services, including S3, Redshift, RDS, Lambda, and IAM
- Familiarity with data formats like JSON, Parquet, Avro, and ORC
- Strong analytical and problem-solving skills
- Experience with data quality and validation techniques
Preferred Qualifications:
- AWS certifications (e.g., AWS Certified Data Analytics - Specialty, AWS Certified Solutions Architect)
- Experience with additional ETL tools (e.g., Apache Spark, Apache Kafka)
- Knowledge of data governance frameworks and tools on AWS
- Exposure to DevOps practices and CI/CD for data pipelines
