Job Location Bangalore
Key Responsibilities
Data Engineer Senior Lead role is to coordinate, and execute all activities related to the requirements interpretation, design, and implementation of Data Analytics applications. This individual will apply proven industry and technology experience as well as communication skills, problem-solving skills, and knowledge of best practices to issues related to design, development, and deployment of mission-critical systems with a focus on quality application development and delivery.
Responsibilities
This role is key to the success of the Data Engineering capability at Telstra and will be responsible and accountable for the following:
- Lead the design, development, and maintenance of data pipelines using Spark, Python, Scala, and related technologies.
- Work with high volume data and ensure data quality and accuracy.
- Implement data security and privacy best practices to protect sensitive data.
- Collaborate with data scientists and business stakeholders to understand data needs and requirements.
- Develop and maintain documentation on data pipeline architecture, data models, and data workflows.
- Mentor and provide technical guidance to junior team members.
- Monitor and troubleshoot data pipelines to ensure they are performing optimally.
- Stay up to date with the latest developments in Azure, AWS, Spark, Python, Scala, and related technologies and apply them to solve business problems.
- Optimize data pipelines for cost and performance.
- Automate data processing tasks and workflows to reduce manual intervention.
- Ability to work in Agile Feature teams.
- Provide training and educate other team members around core capabilities and helps them deliver high quality solutions and deliverables/documentation.
- Self-Motivator to perform Design / Develop user requirements, test and deploy the changes into production.
Who we're looking for.
- Degree level IT qualifications in Software or Systems Engineering.
- Minimum 10 years of Big Data platform experience, consulting or technical lead / team lead role, with 3 years of team leadership experience as a solution lead, team lead, or similar roles implementing projects
- Hands-on experience on Python, Spark, SQL, REST APIs and Azure Cloud experience.
- Experience of working on File formats (Parquet/ORC/AVRO/Delta/Hudi etc.)
- Experience with high volume data processing and data streaming technologies
- Experience of using Orchestration tools like Control-m, Azure Data Factory, Airflow, Luigi to schedule jobs.
- Demonstrated experience leading data engineering projects and mentoring junior team members.
- Data engineer with expertise of working on Azure Cloud using Databricks, Kinesis/ Azure Event hub Flume/Kafka/Spark streaming, Azure Data Factory.
- Strong experience in data modelling, schema design, and ETL development using SQL and related technologies.
- Familiarity with data security and privacy best practices
- Good exposure on TDD
- Exposure on using CI tools like Git, Bitbucket, Github, Gitlab, Azure DevOps
- Exposure on using CD tools like Jenkins, Bamboo, Azure DevOps
- Exposure on Observability tools like Azure Monitor, Graphana etc
- Prior experience in building or working in team building reusable frameworks.
- Good understanding of Data Architecture and design principles. (Delta/Kappa/Lambda architecture)
- Exposure to Open AI, Azure App insights, Telemetry experience, LangChain, Azure cognitive search/Bing search, Gen AI and Chat boat.
- Exposure to Code Quality - Static and Dynamic code scans
- Experience in designing solutions for multiple large data warehouses with a good understanding of cluster and parallel architecture as well as high-scale or distributed RDBMS and/or knowledge on NoSQL platforms.
- Should be able to provided scalable and robust solution architecture depending on the business needs.
- Propose best practices/standards
- Programming & Databases - Java /Python/Scala/ SQL Procedure. Multi tenanted databases / Spark.
