Job Description
Team Leadership:
- Lead, mentor, and manage a team of data engineers.
- Foster a collaborative and innovative team culture.
- Provide guidance on data engineering best practices, methodologies, and technologies.
Data Architecture:
- Design and implement scalable, efficient, and robust data architectures.
- Collaborate with other teams to understand data requirements and ensure alignment with business goals.
- Evaluate and recommend data storage solutions, both on-premises and cloud-based.
ETL (Extract, Transform, Load) Processes:
- Oversee the development and maintenance of ETL processes for efficient data movement.
- Ensure data quality and integrity during the ETL processes.
- Optimize ETL workflows for performance and scalability.
Data Migration:
- Plan and execute data migration projects, ensuring minimal downtime and data integrity.
- Develop strategies for seamless transition between different data storage systems.
- Collaborate with relevant teams to address data mapping, transformation, and validation.
Data Warehousing:
- Manage the design and implementation of data warehouses.
- Ensure the integration of various data sources into the data warehouse.
- Collaborate with business intelligence teams to support reporting and analytics needs.
Data Integration:
- Implement data integration solutions to facilitate seamless data flow between different systems.
- Work with API integration and real-time data processing technologies.
- Ensure data consistency and accuracy across integrated systems.
Performance Monitoring and Optimization:
- Monitor and optimize data processing and storage systems for performance.
- Implement best practices for data partitioning, indexing, and query optimization.
- Troubleshoot and resolve performance issues in collaboration with IT teams.
Data Governance and Security:
- Implement and enforce data governance policies and standards.
- Collaborate with security teams to ensure data privacy and compliance with regulations.
- Establish and maintain access controls for data repositories.
Technical Skills:
- Proficient in SQL for database management and querying.
- Expertise in using PySpark/Python for data engineering tasks and automation.
- Have strong experience in Data Engineering and Data Migration with Large data volumes.
- Familiarity with version control systems and collaborative coding practices.
Qualifications:
- Bachelor's or Master's degree in a related field (e.g., Computer Science, Information Systems).
- Proven experience in data engineering, with a focus on technical leadership roles.
- Strong proficiency in data engineering tools and technologies (atleast two in: Snowflake, Databricks, Apache Spark, Hadoop, Kafka, Flink).
- Knowledge of cloud-based data solutions (AWS).
- Familiarity with data governance, privacy, and security principles.
- Excellent communication and interpersonal skills.
