Job Title: Python Developer (ETL & Pipeline Specialist with Dagster Expertise)
Location: Dwarka,New Delhi
Employment Type: Full-Time
Experience Level: 3-5 Years
Job Summary:
We are seeking a highly skilled Python Developer with deep expertise in Dagster to design, develop, and manage ETL processes and data pipelines for web applications. The ideal candidate will leverage Dagster to orchestrate complex workflows, ensuring robust, scalable, and fault-tolerant backend systems. If you are passionate about data engineering and backend development with a focus on using Dagster for pipeline orchestration, we want to hear from you!
Key Responsibilities:
Dagster Pipeline Orchestration:
- Design, implement, and manage data pipelines using Dagster.
- Leverage Dagster's capabilities to schedule, monitor, and troubleshoot workflows efficiently.
- Ensure pipelines are highly reliable, fault-tolerant, and optimized for performance.
- Implement dynamic, modular pipelines that scale with business needs.
ETL Development and Data Processing:
- Develop and maintain efficient ETL processes to extract, transform, and load data from various sources.
- Integrate data workflows with web applications to ensure seamless data delivery.
- Monitor and enhance data quality, ensuring accuracy and consistency across all systems.
Backend Development:
- Build and maintain backend services and APIs using frameworks like Flask, FastAPI, or Django.
- Integrate pipeline outputs with backend systems for real-time application functionality.
- Implement secure and efficient data storage solutions using relational and NoSQL databases.
Optimization and Debugging:
- Optimize pipeline and backend performance for low-latency, high-throughput environments.
- Proactively identify and resolve bottlenecks and data quality issues in pipelines.
Collaboration and Documentation:
- Work closely with frontend developers, data engineers, and product teams to align pipeline and application requirements.
- Document Dagster pipeline configurations, ETL processes, and backend architecture.
Qualifications:
Education:
- Bachelor's or Master's degree in Computer Science, Software Engineering, or a related field.
Technical Skills:
- Proven expertise in Dagster for orchestrating and managing data pipelines.
- Strong proficiency in Python and its data manipulation libraries (e.g., Pandas, NumPy).
- Experience in backend development using frameworks like Flask, FastAPI, or Django.
- Familiarity with relational and NoSQL databases (e.g., PostgreSQL, MongoDB).
- Proficiency in developing and integrating RESTful APIs.
Preferred Skills:
- Experience with other pipeline orchestration tools (e.g., Apache Airflow, Prefect).
- Familiarity with Docker and CI/CD pipelines for deployment and scaling.
Soft Skills:
- Strong problem-solving and analytical skills.
- Excellent communication and teamwork abilities.
- Eagerness to learn and adapt to new tools and technologies.
What We Offer:
- Competitive salary and benefits.
- Opportunities to work on innovative projects using Dagster and other cutting-edge technologies.
- Support for professional development and learning.
To Apply:
Please submit your resume and a cover letter detailing your experience with Dagster, ETL processes, and backend development to [[Confidential Information]].
