Data Specialist

Lead Databricks Engineer

Experience 6 Years to 8 Years

Budget 8 LPA to 30 LPA

Location Magarpatta, Pune

Must Have Skills Azure Databricks, Pyspark, Unity Catlog

Roles and responsibilities -

1. Design and Develop Data Pipelines:

Create scalable data processing pipelines using Azure Databricks and PySpark.

Implement ETL (Extract, Transform, Load) processes to ingest, transform, and load data from various sources.

Collaborate with data engineers and architects to ensure efficient data movement and transformation.

2. Data Quality Implementation:

Establish data quality checks and validation rules within Azure Databricks.

Monitor data quality metrics and address anomalies promptly.

Work closely with data governance teams to maintain data accuracy and consistency.

3. Unity Catalog Integration:

Leverage Azure Databricks Unity Catalog to manage metadata, tables, and views.

Integrate Databricks assets seamlessly with other Azure services.

Ensure proper documentation and organization of data assets.

4. Delta Lake Expertise:

Understand and utilize Delta Lake, which provides ACID transactions and time travel capabilities on top of data lakes.

Implement Delta Lake tables for reliable data storage and versioning.

Optimize performance by leveraging Delta Lake features.

5. Performance Tuning and Query Optimization:

Profile and analyze query performance.

Optimize SQL queries, Spark jobs, and transformations for efficiency.

Tune resource allocation to achieve optimal execution times.

6. Resource Optimization:

Manage compute resources effectively within Azure Databricks clusters.

Scale clusters dynamically based on workload requirements.

Monitor resource utilization and cost efficiency.

7.Source System Integration:

Integrate Azure Databricks with various source systems (e.g., databases, data lakes, APIs).

Ensure seamless data ingestion and synchronization.

Handle schema evolution and changes in source data.

8Stored Procedure Conversion in Databricks:

Convert existing stored procedures (e.g., from SQL Server) into Databricks-compatible code.

Optimize and enhance stored procedures for better performance within Databricks

SSRS conversion experience

Required skills -

Minimum 5 years of experience architecting and building data platforms on Azure.

Proficiency in Azure Databricks, PySpark, and SQL.

Familiarity with Delta Lake concepts.

Certifications (preferred):

Microsoft Certified: Azure Data Engineer Associate or similar.

Databricks Certified Associate Developer for Apache Spark.