This role is for one of the Weekday's clients
We are seeking a highly skilled Senior Data Engineer to join our growing team. In this role, you'll play a key part in building and scaling our data platform to support complex data processing and analytics needs. You will lead the design and development of scalable systems that handle large volumes of batch and real-time data, driving data-driven insights and supporting business decisions across the organization. This is an exciting opportunity for someone passionate about cutting-edge data technologies, cloud-native environments, and making an impact through data infrastructure.
Key Responsibilities
- Data Platform Design and Development: Architect, build, and maintain a scalable, reliable data platform to support batch processing, real-time data streams, and advanced analytics, ensuring cost efficiency and optimized performance.
- Data Pipeline Management: Develop and manage scalable, resilient ETL/ELT pipelines for data ingestion, transformation, and distribution across the organization.
- Real-Time Data Processing: Design and implement real-time data processing solutions using technologies like Apache Kafka and Apache Flink, enabling low-latency, high-throughput data streams for business-critical applications.
- Data Storage Optimization: Lead efforts to design and optimize data lakes and warehouses, ensuring efficient data storage, retrieval, and cost-effective management in cloud environments.
- System Reliability and Monitoring: Implement monitoring solutions and data quality frameworks to ensure data accuracy, reliability, and security, proactively identifying and resolving performance bottlenecks.
- Cross-Team Collaboration: Work closely with data scientists, analysts, and product teams to deliver scalable data solutions that meet evolving business needs.
- Data Security and Compliance: Champion data security and compliance measures, ensuring data integrity and privacy.
- Technical Leadership: Provide technical leadership and mentorship to junior engineers, promoting best practices in system design, coding, and deployment.
- Automation and Infrastructure as Code (IaC): Automate data infrastructure management using tools like Terraform and Ansible to ensure consistency, repeatability, and reduced operational overhead.
Required Qualifications
- 6+ years of hands-on experience in data platform engineering, with expertise in architecting scalable data platforms.
- Proven experience in designing and implementing batch and real-time data pipelines in cloud environments (preferably AWS).
- Strong expertise in distributed data processing frameworks (e.g., Apache Spark, Presto) and real-time streaming technologies (Kafka, Flink).
- Experience with modern data lake architectures and formats like Hudi, Iceberg, or Delta Lake.
- Proficiency in Python and/or Java, with a focus on building robust, scalable code for complex data systems.
- Strong SQL and/or NoSQL skills.
- Hands-on experience with Infrastructure as Code tools like Terraform or Ansible for cloud infrastructure management.
- Experience in performance tuning and optimization of large-scale data systems.
- Strong ability to prioritize and scope tasks, with a focus on delivering incremental improvements and adapting to evolving project needs.
- Excellent communication and leadership skills, with experience mentoring junior engineers.
- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
Preferred Qualifications
- Experience with DataOps and CI/CD pipelines in data engineering.
- Experience building data platforms from the ground up.
- Knowledge of Kubernetes is a significant plus.
- Experience in a startup environment, with a willingness to innovate and adapt to changing business needs.
Skills: python,elt,apache spark,data lakes,presto,apache,data warehouses,java,nosql,terraform,etl,sql,ansible,infrastructure,apache kafka,apache flink,data engineering,data processing,data platform engineering
