Data Pipeline Management:Design, construct, and maintain robust data pipelines.
Data Integration:Seamlessly integrate data from various sources, including APIs like Seccl and Morningstar, ensuring high data consistency and quality.
Data Architecture:Develop scalable data architectures to support advanced analytics and BI tools such as Power BI.
Microsoft Fabric Implementation:Actively participate in the migration to Microsoft Fabric, utilising its tools to enhance our data infrastructure.
Data Security and Governance:Ensure data security and compliance with relevant regulations, implementing robust data governance practices.
Data Quality Assurance:Manage data integrity and quality throughout its lifecycle.
Performance Optimisation:Improve data processes and pipelines for enhanced performance and scalability.
Monitoring and Troubleshooting:Monitor data workflows and address issues as they arise.
Collaborative Development:Work closely with data analysts, scientists, and other stakeholders to meet complex data requirements and drive innovation.
Essential Skills:
Programming:Strong proficiency in Python. Knowledge of C# is a plus.
Database Management:Extensive experience with SQL and familiarity with both relational and NoSQL databases.
ETL Tools and Big Data Technologies:Experience with tools like Azure Data Factory and big data frameworks such as Apache Spark.
Cloud Platforms:Deep experience with Azure, including Azure Synapse Analytics.
Microsoft Fabric:Knowledge of or experience with Microsoft Fabric as we transition our systems to this new platform.
Data Visualisation Tools:Skill in tools like Power BI for creating insightful visualisations.
Data Security and Governance:Understanding of data security principles and governance practices.
Problem-Solving:Excellent analytical and problem-solving skills.
Desirable Skills:
Azure AI Studio:Familiarity with Azure AI Studio and a willingness to embrace AI as we move to Microsoft Fabric.
AI and Machine Learning:Familiarity with developing and deploying AI/ML models is a plus.
Active Interest in Technology:Demonstrated interest in the latest Microsoft technologies, OpenAI, and a keen follower of technological developments.
Ideal Candidate Profile:
Educational Background:Bachelor's or master's degree in Computer Science, Information Technology, Data Science, or a related field.
Experience:Proven track record in data engineering with extensive experience in managing complex data pipelines and cloud environments.
Certifications:Certifications in data engineering, Azure, or Microsoft technologies are beneficial.
