Job Title : Azure Data Engineer
Location: Pune or Bangalore or Hyderabad (Remote/Hybrid)
Duration: Full time
Job description:
A role to design, build, and maintain data infrastructure in Microsoft Azure, ensuring data flows smoothly from various sources to destinations for analytics and reporting purposes.
Key Responsibilities:
- Design and Develop Data Pipelines:
- Build and optimize scalable data pipelines using Azure Data Factory (ADF) and Azure Synapse Analytics.
- Develop, maintain, and improve data integration, processing, and storage pipelines for ETL/ELT processes.
- Ensure data consistency, accuracy, and reliability across the pipelines.
- Data Modeling and Architecture:
- Design and implement data models (Star, Snowflake, and OLAP) based on business requirements.
- Collaborate with data architects to develop and refine the data warehouse/lake architecture using Azure Synapse Analytics or Azure Data Lake Storage.
- Data Transformation and Storage:
- Implement data transformation using Azure Databricks, SQL, and other tools.
- Manage and monitor data storage using Azure Data Lake and SQL Databases.
- Optimization and Performance Tuning:
- Optimize data processes for performance, scalability, and cost-efficiency in an Azure cloud environment.
- Security and Compliance:
- Implement data security best practices, including data encryption, role-based access control, and monitoring using Azure Security and Azure Monitor.
- Collaboration:
- Work closely with data scientists, business analysts, and other stakeholders to gather requirements and ensure data pipelines meet business needs.
- Monitoring and Maintenance:
- Set up automated monitoring, alerts, and logging for data pipelines.
- Troubleshoot and resolve data pipeline issues and ensure minimal downtime.
Technical Skills:
- ETL/ELT design and development experience using Azure Data Factory or similar tools.
- Expertise in SQL, PySpark, and Python for data manipulation and transformation.
- Proficient in Azure DevOps for CI/CD pipeline management and version control.
- Strong understanding of data warehousing and data lake architectures.
- Hands-on experience with Azure Databricks for big data processing and machine learning integration.
