Vidpro Consultancy Services is seeking a Senior Databricks Engineer / Databricks Technical Lead / Data Architect with 5-10 years of experience.
Location: Bangalore, Chennai, Delhi, Pune, Kolkata and Gurgaon
Work Type: Hybrid
Mandatory Skills: Python/ PySpark / Spark with Azure/ Azure Databricks / Azure DataFactory
Primary Roles And Responsibilities
- Creating Modern Data Warehouse solutions using Databricks and AWS/Azure Stack
- Providing forward-thinking solutions in data engineering and analytics
- Collaborating with DW/BI leads for new ETL pipeline development requirements
- Troubleshooting issues in existing pipelines and addressing gaps
- Understanding reporting layer needs and developing data models
- Assisting team members with problem resolution and technical challenges
- Leading technical discussions with client architects and team members
- Orchestrating data pipelines in a scheduler via Airflow
Skills And Qualifications
- Bachelor's and/or master's degree in computer science or equivalent experience
- Minimum 3 years of experience in Data warehouse/ETL projects
- Deep understanding of Star and Snowflake dimensional modelling
- Strong knowledge of Data Management principles
- Proficiency in Databricks Data & AI platform and Databricks Delta Lake Architecture
- Hands-on experience in SQL, Python and Spark (PySpark)
- Experience with AWS/Azure stack and ETL with batch and streaming (Kinesis)
- Building ETL/data warehouse transformation processes
- Familiarity with Apache Kafka for use with streaming data/event-based data
- Working knowledge of Hadoop (incl. Hive, Pig, Impala) and non-relational/NoSQL data repositories (incl. MongoDB, Cassandra, Neo4J)
- Experience with structured and unstructured data including imaging & geospatial data
- Proficiency in RDBMS, complex SQL, PL/SQL, Unix Shell Scripting, performance tuning and troubleshooting
- Databricks Certified Data Engineer Associate/Professional Certification (Desirable)
- Comfortable working in a dynamic, fast-paced, innovative environment with several ongoing concurrent projects
- Experience working in Agile methodology
- Strong verbal and written communication skills
- Strong analytical and problem-solving skills with a high attention to detail
Skills: etl,circleci,rdbms,mongodb,hadoop,pyspark,pl/sql,databricks data & ai platform,azure,azure databricks,git,data warehouse,databricks delta lake architecture,sql,aws,pig,impala,apache kafka,unix shell scripting,data management,spark,neo4j,extract, transform, load (etl),hive,microsoft azure,azure datafactory,cassandra,python,pipelines,terraform
