Overview
The role of a Data Engineer at Gen AI is crucial for the organization's data infrastructure and analytics capabilities. The Data Engineer will be responsible for designing and developing scalable data pipelines, integrating data from various sources, and optimizing data flow and collection for cross-functional teams. This position plays a key role in shaping the data architecture and ensuring efficient data management for the organization's data-driven decision-making processes.
Key Responsibilities
- Design, construct, install, test, and maintain data management systems
- Develop and maintain scalable data pipelines for data analytics and processing
- Implement ETL processes to move data from various sources to data warehouses
- Collaborate with data scientists and analysts to understand data needs and deliver appropriate solutions
- Assess and develop data architecture strategies for the organization's data ecosystem
- Optimize and improve existing data pipelines and infrastructure
- Perform data modeling and schema design for various application requirements
- Ensure data quality and integrity by implementing best practices and data governance processes
- Automate data processes and monitor system performance and reliability
- Work closely with cross-functional teams to understand data requirements and provide technical solutions
- Troubleshoot and resolve data-related issues and provide technical support as needed
- Stay updated with the latest trends in data engineering and technologies
- Contribute to the continuous improvement of data engineering best practices and standards
Required Qualifications
- Strong experience in Python programming language - Must.
- Experience in REST APIs, Fast APIs, Graph APIs, SQL Alchemy.
- Good experience in Azure Services like DataLake, Azure SQL, Function App, Azure Cognitive Search.
- Good Understanding on Chunking, Embeddings, vectorization, indexing, Prompting, Hallucinations, RAG.
- Hands on experience in DevOps, create pull PRs and maintain code repositories.
- Strong communication skills and ability to collaborate effectively with team members.
Skills: python,sql,etl,data modeling,azure
