As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining our clients data infrastructure. You will work mainly with snowflake and other variety of tools and technologies to manage and process data, optimize queries, and ensure data integrity. The ideal candidate will have extensive experience with SQL, Snowflake, data pipelines, and data processing tools, and a solid understanding of cloud platforms.
Key Responsibilities:
- SQL Query Writing & Scripting: Develop and optimize complex SQL queries for data extraction, transformation, and analysis
- Snowflake Expertise: Utilize Snowflake features such as SnowSQL, Bulk Copy, Snowpipe, Tasks, Streams, Time Travel, Cloning, Optimizer, Metadata Manager, data sharing, stored procedures, UDFs, and Snowsight, Coalesce for dynamic table management
- Data Pipeline & Data warehousing: Design, create, and orchestrate data pipelines, ETL & ELT processes using Snowflake and Airflow to ensure smooth data flow and processing. Build a data warehouse or data-mart for data analysis or further usage
- Programming Skills: Develop and maintain scripts and applications using Python, Java, or Scala. Intermediate proficiency in these languages is required
- Data Processing: Leverage data processing libraries such as NumPy and Pandas in Python, or equivalent libraries in other programming languages
- Cloud Storage: Work with cloud storage solutions across AWS, Azure, or GCP, ensuring effective data storage and management
- Data Streaming: Implement and manage data streaming solutions using Snowpipe and other relevant tools.
- Big Data Tools: Nice to have knowledge to utilize tools such as Spark, Spark MLlib, or Apache Kafka or any other tools for big data processing and analytics
- Basic ML Understanding: Apply basic machine learning principles and practices to enhance data processing and analysis
- Collaboration: Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet business needs
Qualifications:
- Experience: Minimum 5 years of experience in data engineering, with a strong focus on SQL, Snowflake, and data pipeline development and data warehousing
- Advanced proficiency with SQL and Snowflake utilities
- Intermediate programming skills in Python, Java, or Scala
- Having experience in using cloud storage solutions (AWS, Azure, GCP)
- Experience with data processing libraries (NumPy, Pandas) in Python or similar libraries in other programming languages
- Expertise in Apache Airflow for orchestrating data workflows
- Basic understanding of machine learning concepts
- Experience with Azure Databricks
- Familiarity with Snowpark, Apache Spark, and data streaming technologies
