Azure Data Engineer

1. Design, develop, and maintain scalable data pipelines and ETL processes using Databricks and PySpark.

2. Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and implement data solutions that align with business needs.

3. Optimize and tune existing data pipelines for performance and reliability.

4. Work closely with cloud-based services, particularly Azure or AWS, to deploy and manage data solutions in the cloud environment.

5. Implement data quality and data governance best practices to ensure the accuracy and reliability of the data.

6. Monitor and troubleshoot data pipelines, resolving issues promptly to minimize downtime and data loss.

7. Stay updated with the latest industry trends and technologies related to data engineering, Databricks, PySpark, and cloud platforms.

8. Collaborate with cross-functional teams to integrate data engineering solutions into various applications and systems.

9. Document data engineering processes, architectures, and configurations for reference and knowledge sharing.

10. Provide technical guidance and mentorship to junior data engineers in the team.

11. Design, develop, and maintain scalable data pipelines using Azure Data Factory, Azure Databricks, and other Azure data services.

12. Utilize Databricks for big data analytics and real-time data processing, ensuring optimal performance and efficiency.

13. Collaborate with data architects to design and implement effective data models and architectures to support business requirements.

14. Implement ETL processes for data extraction, transformation, and loading, ensuring the integrity and quality of the data.