Key Responsibilities
- Data Modeling:* Design and implement efficient data models, ensuring data accuracy and optimal performance.
- ETL Development:* Develop, maintain, and optimize ETL processes to extract, transform, and load data from various sources into our data warehouse.
- SQL Expertise:* Write complex SQL queries to extract, manipulate, and analyze data as needed.
- Python Development:* Develop and maintain Python scripts and applications to support data processing and automation.
- AWS Expertise:* Leverage your deep knowledge of AWS services, such as S3, Redshift, Glue, EMR, and Athena, to build and maintain data pipelines and infrastructure.
- Infrastructure as Code (IaC):* Experience with tools like Terraform or CloudFormation to automate the provisioning and management of AWS resources is a plus.
- Big Data Processing:* Knowledge of PySpark for big data processing and analysis is desirable.
- Source Code Management:* Utilize Git and GitHub for version control and collaboration on data engineering projects.
- Performance Optimization:* Identify and implement optimizations for data processing pipelines to enhance efficiency and reduce costs.
- Data Quality:* Implement data quality checks and validation procedures to maintain data integrity.
- Collaboration:* Work closely with data scientists, analysts, and other teams to understand data requirements and deliver high-quality data solutions.
- Documentation:* Maintain comprehensive documentation for all data engineering processes and projects.
Qualifications
At least 6-9 years of experience in data engineering roles, with a strong emphasis on AWS technologies.
Proficiency in data modeling, SQL, and Python.
Demonstrated expertise in AWS services, especially for data processing, extract transform and load.
Familiarity with Infrastructure as Code (IaC) tools.
Experience with PySpark and big data processing is a plus.
Strong version control skills using Git and GitHub.
Excellent problem-solving and communication skills.
Ability to work independently and in a team, taking ownership of projects and delivering on time.
