Senior Data Engineer - (Google BigQuery, Apache Dataflow and Shell Scripting)
We are looking for a talented Data Engineer to join our team. As a Data Engineer, you will be responsible for designing, implementing, and maintaining robust data pipelines and infrastructure to support our data-driven initiatives. You will work closely with cross-functional teams to ensure that our data solutions meet the needs of the business.
Experience: 5-8 Yrs
Location: Pune, Bengaluru ,Hyderabad
Mandatory Skills:
- Proficiency in Google BigQuery for data warehousing and analytics.
- Experience with Apache Dataflow (Apache Beam) for building and managing data processing pipelines.
- Strong shell scripting skills for automation and scheduling tasks.
- Familiarity with Google Cloud Composer for workflow orchestration and management.
- Preferred experience with BI tools such as Power BI and Microstrategy for data visualization and reporting.
Key Responsibilities:
- Design, develop, and deploy data pipelines using Google BigQuery and Apache Dataflow to process large-scale datasets efficiently.
- Write and optimize shell scripts for automating data pipeline workflows and scheduling tasks.
- Utilize Google Cloud Composer for orchestrating complex workflows and managing dependencies.
- Collaborate with data analysts and business stakeholders to understand data requirements and translate them into technical solutions.
- Implement best practices for data governance, data quality, and data security within our data infrastructure.
- Troubleshoot and resolve issues related to data processing, performance, and scalability.
- Work closely with the BI team to integrate data pipelines with preferred BI tools such as Power BI and MicroStrategy for reporting and visualization.
- Stay current with industry trends and emerging technologies in data engineering and recommend opportunities for innovation and improvement.
- Document data pipelines, workflows, and technical specifications for knowledge sharing and future reference.
