Job Description
Job Location Bangalore
Key Responsibilities
As a Sr Data Platform Engineer/Data Platform Designer, your key responsibilities will be:
Design and Deliver Data Platforms: Align with our Data Strategy across various projects and functional areas.
System Integration: Collaborate with Enterprise Architects and Data Solution Architects to ensure seamless integration and robust data models.
Data Sharing Integrations: Partner with internal Data Platform teams and external partners to strategize and implement necessary integrations.
System Designs/Architecture: Propose designs for reliable data flows, integrating analytical batch processing, real-time data flows, and low-latency APIs.
Vendor Collaboration: Utilize your interpersonal skills to work with vendors, review product offerings, and integrate new products into our platform architecture.
Platform Management: Apply your data engineering expertise to manage and maintain data and automation platforms throughout their life cycles, ensuring information security.
CI/CD Pipelines: Develop and maintain pipelines for automated build, test, and deployment of code.
Automation: Identify and automate recurring platform administration tasks to enhance efficiency and effectiveness.
Technical Debt Management: Address technical debt to meet scalability and security requirements.
Feature Evaluation: Work with business users and technical resources to evaluate new features and architecture, contributing to solving challenging problems at scale.
Agile Development: Foster a culture of continuous delivery by supporting team planning activities, writing scrum team stories, and collaborating with team members to deliver work within a sprint.
Expertise Development: Seek out expertise and best practices both inside and outside the team to deepen your knowledge in data engineering.
Shift Work: Be open to working in shifts.
Strategy Contribution: Work with TPOs, Chapter Leads, and Group Owners to contribute to the team's strategy and roadmap.
Pipeline Optimisation: Collaborate with feature teams on optimisation and best practices.
Observability: Focus on user growth, data growth, cost, pipelines, data currency, error trends, service requests, and changes.
User Experience: Conduct user experience surveys and use the feedback to improve the platform.
Scalable Solutions: Ensure the design and delivery of solutions are scalable.
Cloud Cost Optimisation: Work on strategies for cloud cost optimisation.
Resiliency: Make sure the platform is resilient to various scenarios.
Security First Design: Prioritise security in all designs.
Task Automation: Identify mundane tasks and automate them for efficiency.
Team Support: Assist team members with issues and blockers.
Backlog Management: Manage and prioritise the backlog effectively.
Engineering Maturity: Work towards improving the maturity level of engineering practices.
AI/ML Platform Maintenance: Maintain and optimize AI/ML platforms, ensuring they are up-to-date and efficient.
AI/ML Best Practices: Stay updated with the latest trends and best practices in AI/ML, and implement them in our platforms.
AI/ML Collaboration: Collaborate with AI/ML teams to understand their needs and ensure the platform meets those needs.
Who We're Looking For
Experience: Minimum 8+ years of experience in data platforms and Azure technologies.
Role Experience: Proven experience as a data engineer or software engineer with a focus on data platforms and Azure technologies.
Azure Expertise: Strong expertise in Azure DevOps, Azure data services, including Azure Data Lake, Azure Databricks, Azure Synapse Analytics, Azure Data Factory, and Azure Cosmos DB. Familiarity with Azure ML, AWS Sage Maker, ML Flow is a plus.
Problem-Solving Skills: Strong problem-solving and analytical skills, with the ability to troubleshoot complex data issues.
Infrastructure-as-Code: Familiarity with infrastructure-as-code (IaC) technologies like Terraform, Bicep, or ARM Templates.
Scripting: Scripting experience - Python, Bash, PowerShell, or similar.
Data Security: Knowledge of data security, encryption, and compliance standards.
Version Control: Experience with version control systems like Git and collaborative development methodologies.
Data Engineering Concepts: Strong expertise in data engineering concepts and technologies, including data ingestion, ETL/ELT processes, and data integration techniques.
Communication Skills: Excellent communication and collaboration skills to work effectively with cross-functional teams.
Big Data Processing: Experience with big data processing frameworks such as Apache Spark or Azure Databricks.
System Administration: Experience with Linux and Windows Administration.
AI/ML Platform Maintenance: Experience in maintaining and optimizing AI/ML platforms.
AI/ML Best Practices: Staying updated with the latest trends and best practices in AI/ML.
AI/ML Collaboration: Experience in collaborating with AI/ML teams to understand their needs and ensure the platform meets those needs.
Good To Have
AI/ML and GenAI Technologies: Familiarity with AI/ML and GenAI technologies is highly desirable.
Improving Engineering Practices: Ability to leverage AI/ML and GenAI to improve engineering practices.
Task Automation: Proficiency in automating mundane tasks using AI/ML and GenAI, freeing up time for more complex problem-solving.
