Azure Data Engineer

Description of Duties:

Design, develop, and implement new data solutions on Azure/AWS to consume and deliver data solutions.

Maintain and support existing applications on Azure/AWS platforms.

Actively participate in daily team meetings and sprints.

Demonstrate proficiency in understanding data architecture design using Azure data engineering capabilities.

Client-facing role, requiring strong communication and presentation skills.

Technical Requirements:

Minimum of 1 year of experience in data transformations and ETL using Azure/AWS technologies.

Experience with CI/CD deployments (DataOps).

Knowledge of building real-time and batch data processing pipelines using Apache Spark, Azure Data Factory, Azure Databricks and Synapse will be beneficial.

Knowledge of Delta Lake for managing data lakes with reliability and performance. Familiarity with Databricks notebooks for collaborative development

Knowledge of security best practices for data handling and processing within Azure Databricks environment. Knowledge of security best practices for data handling and processing within Azure Databricks environment.

Excellent knowledge of relational and non-relational database modelling.

Proficiency in manipulating and combining large datasets from multiple sources.

Strong expertise in Python scripting.

Experience with SQL databases, such as MySQL and SQL Server.

Familiarity with SQL NoSQL databases, like MongoDB and CosmosDB.

Background in building ETL/Data Warehouse transformation processes.

Proficiency in writing complex SQL queries and creating data models.

Understanding of the software development lifecycle.

Excellent analytical and problem-solving skills.

Other Requirements:

Ability to work both independently as a self-starter and within a team environment.

Strong communication skills to effectively collaborate with delivery teams and business-facing teams, understand sometimes ambiguous needs, and translate them into clear, aligned requirements.

Client-facing role, requiring strong communication and presentation skills.

Good to Have:

Knowledge of PySpark.

Experience with medium to large projects.

Familiarity with agile work practices.

Passion for achieving goals and delivering results.

Machine Learning and Recommendation Systems model building knowledge.