As a Data Engineer on the data foundation team, well rely on your expertise across multiple disciplines to develop, deploy and support data systems, data pipelines, data lakes, and lakehouses. Your ability to automate, operate, performance tune, and scale the data platform will be key to your success.
You will: - Collaborate with stakeholders to make effective use of core data assets
- Leverage a combination of tools, engines, libraries, and code to build data integrations
- Build integrations and pipelines loading both streaming and batched data
- Development of data platform and data pipeline operational and administrative tools in Python
- Load data to support defined data patterns and use cases
- Build integrations and pipeline unit, integration, quality, and performance tests
- Automate continuous deployment and testing of integrations and data pipelines
- Compose data engineering infrastructure as code and build automated infrastructure deployment workflows
- Monitor, operate, and optimize tools, systems, integrations, and pipelines
- Participate in peer code reviews, code approvals, and pull requests
- Help ensure appropriate data privacy and security
- Identify, recommend, and implement opportunities for improvement in efficiency, resilience, scale, security, and performance
What You Need to Get the Job Done (if you don't have all, apply anyway!) - Understanding of data lake, lakehouse, and data warehouse systems, and related technologies
- Knowledge and understanding of data formats, data patterns, models, and methodologies
- Experience developing data integrations, tools, and pipelines in Python
- Experience working with APIs
- Experience working with streaming technologies such as Kafka
- Ability to work with cloud infrastructure including resource scaling, S3, RDS, IAM, security groups, AMIs, cloudwatch, cloudtrail, and secrets manager
- Experience with expressing infrastructure as code leveraging tools like Terraform
- Experience with GitFlow and GitHub automation
- Git-based team coding workflows
