Job Overview:
We are seeking a talented and motivated Senior Data Engineer with 6 to 10 years of experience to join our dynamic team. The ideal candidate will possess a strong background in designing, building, and maintaining scalable data pipelines and databases. You will work closely with engineering teams and product owner to translate requirements into resilient, secure, and high-performance software. Key responsibilities include designing, coding, testing, and implementing new or existing software products based on specifications. A crucial aspect of this role is a commitment to continuous learning and adaptability to evolving technologies and market trends.
Key Responsibilities:
- Data Pipeline Development:
- Design, build, and maintain robust and scalable data pipelines to process and transform large volumes of data from various sources.
- Implement ETL (Extract, Transform, Load) processes to ensure data is accurately and efficiently integrated into data warehouses or data lakes.
- Database Management:
- Develop, build and optimize database queries and schema designs to enhance performance and support analytical needs.
- Data Integration:
- Collaborate with cross-functional teams to gather data requirements and integrate data from diverse sources including APIs, flat files, and databases.
- Ensure data consistency, accuracy, and integrity across various systems.
- Data Quality and Monitoring:
- Implement data quality checks and monitoring processes to identify and resolve data issues proactively.
- Develop and maintain data documentation and metadata for transparency and reproducibility.
- Performance Tuning:
- Monitor and tune the performance of data pipelines and databases to ensure optimal operation and scalability.
- Troubleshoot and resolve data-related issues in a timely manner.
- Collaboration and Communication:
- Work closely with various stakeholders to understand their data needs and provide appropriate solutions.
- Communicate technical concepts and solutions effectively to both technical and non-technical team members.
- Continuous Learning and Adaptation:
- Stay current with emerging technologies, trends, and best practices in data engineering.
- Demonstrate a proactive attitude towards learning new tools and technologies, and adapt quickly to changes in the market and industry requirements.
Qualifications:
- Education:
- Bachelor's degree in Computer Science, IT, Engineering or a related field
- Experience:
- 6 to 10 years of hands-on experience in data engineering or a related role.
- Proficiency with data warehousing solutions and ETL tools.
- Technical Skills:
- Proficient in SQL: Extensive experience in writing complex queries, stored procedures, and functions in SQL and Azure SQL Managed Instance (SQL-MI).
- Data Pipeline Orchestration: Hands-on experience with data pipeline orchestration tools such as SSIS, Azure Data Factory, and Azure Synapse.
- Knowledge of Microsoft Fabric: Familiarity with end-to-end analytics and data platform Microsoft Fabric
- Programming Skills: Proficiency in Python and PySpark for data manipulation and analysis.
- Azure DevOps: Experience with Azure DevOps for version control, continuous integration, and deployment.
- Repository Management: Strong practical experience with code repositories such as Azure Repos and Bitbucket.
- Database Concepts: Solid understanding of Relational Database Management Systems (RDBMS) and data warehousing concepts.
- Cloud Platforms: Experience working with Azure
- Nice to have:
- Data Visualization: Knowledge / familiarity with data visualization tools like Power BI to create insightful and interactive reports.
- Programming Knowledge: Familiarity with additional programming languages such as .NET, Java, or front-end technologies like React/JavaScript is a plus.
- Project Management & Issue Tracking: Familiarity/exposure in using Jira - creating/managing stories, sub-tasks, tasks, bugs
- Soft Skills:
- Excellent problem-solving skills and attention to detail.
- Strong analytical and organizational abilities.
- Ability to work effectively both independently and as part of a team.
- Effective communication skills to interact with various stakeholders.
- Willingness to learn new technologies and adapt to rapid changes in the data engineering landscape.
