Job Description: The Senior Data & AI Architect designs, develops, and implements complex data architectures, machine learning (ML) models, and AI-driven solutions using Azure (preferred), AWS, and GCP. This role requires hands-on expertise in data engineering, ML model development, big data frameworks, and cloud infrastructure. The architect will manage large-scale data processing, optimize ML pipelines, and deploy AI solutions while ensuring security, scalability, and performance.
Key Responsibilities:
Data & AI Architecture Design:
- Design scalable, reliable data pipelines using Azure Data Factory, Synapse, and Data Lake.
- Architect ML models with cloud integration for business use cases using Azure ML.
- Design data lakes/warehouses for batch and real-time processing.
- Integrate AI models with real-time data systems (Azure Event Hub, Stream Analytics).
Cloud Architecture:
- Build AI solutions using Azure AI services (Cognitive Services, ML, etc.).
- Implement strong data governance with Azure Data Catalog, Policy, and Security Center.
- Ensure high availability, fault tolerance, and disaster recovery using Azure services.
Performance Optimization:
- Optimize distributed data processing with Apache Spark on Azure Databricks.
- Improve query performance in Synapse with partitioning, indexing, and caching.
- Optimize ML model efficiency and inference times using Azure ML AutoML.
ML Lifecycle Management:
- Implement CI/CD pipelines for ML models using Azure DevOps.
- Apply MLOps for model monitoring, retraining, and lifecycle management.
- Ensure scalable model serving with Azure Kubernetes Service (AKS).
Real-time Data & Integration:
- Deploy real-time data pipelines with Azure Event Hub, Stream Analytics, and Kafka.
- Design data integration strategies using Azure Data Factory and APIs.
AI Solution Delivery:
- Lead AI product development (recommendation engines, predictive analytics).
- Drive end-to-end AI solution delivery, integrating models into business workflows.
Candidate Profile:
- 10+ years of experience in data architecture, data engineering, AI/ML, and cloud computing.
- Expertise in Azure (Data Factory, Synapse, ML, AKS); AWS/GCP experience is a plus.
- Strong knowledge of big data frameworks (Apache Spark, Databricks), Python, SQL, Terraform, and Kubernetes.
Key Attributes:
- Excellent problem-solving skills with a focus on optimizing performance, cost, and scalability.
- Experience designing secure, cloud-native AI systems.
- Ability to manage complex challenges independently in a fast-paced environment.
