Data QA Automation Engineer (Python)

Job Description:Data QA Automation Engineer (Python)

Software Requirements:

• Proficiency in Python programming with a strong understanding of its ecosystems and frameworks.
• Experience with data validation techniques and methodologies to ensure the accuracy and integrity of data.
• Hands-on experience with DB2 for database management and operations.
• Experience working with Snowflake for cloud-based data warehousing solutions.
• Proficiency in Databricks for big data processing and analytics on a unified data platform.

Overall Responsibilities:

• Develop and maintain scalable data pipelines and build out new API integrations to support continuing increases in data volume and complexity.
• Write high-quality, efficient, testable code in Python and other object-oriented languages.
• Collaborate with cross-functional teams to understand data needs, gather requirements, and implement data solutions.
• Conduct data validation to ensure the accuracy and quality of data through all pipelines.
• Implement processes and systems to monitor data quality, ensuring production data is always accurate and available for key stakeholders and business processes that depend on it.
• Perform data analysis required to troubleshoot data-related issues and assist in the resolution of data issues.
• Work closely with a team of frontend and backend engineers, product managers, and analysts.
• Define and manage SLA for all data sets in allocated areas of ownership.

Technical Skills:

Programming and Scripting:

Strong knowledge of Python and its libraries for data analysis (Pandas, NumPy, SciPy).

Database Management:

• Experience with SQL databases such as DB2.
• Familiarity with Snowflake and Databricks platforms.

Data Handling:

Understanding of data modeling, data access, and data storage techniques.

Version Control:

Ability to use version control systems such as Git.

API and Data Formats:

Knowledge of RESTful APIs and various data formats (JSON, XML).

Development Methodologies:

Experience with Agile/Scrum development methodologies.

Experience:

• Minimum of 3-7 years of experience in a similar role, working with Python, data validation, DB2, Snowflake, and Databricks.
• Proven track record of designing and deploying high-quality, scalable, and reliable data solutions.
• Experience working with large-scale data-driven applications is a plus.
• Day-to-Day Activities:
• Develop, test, and deploy automated scripts for data validation and pipeline monitoring.
• Collaborate with team members to identify and resolve data quality issues.
• Participate in daily stand-ups and sprint planning meetings.
• Perform code reviews and provide constructive feedback to peers.
• Continuously monitor data pipelines and address any anomalies promptly.
• Document technical specifications and maintain up-to-date project documentation.
• Engage in continuous learning and stay updated with the latest industry trends and technologies.

Qualification:

• Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field, or equivalent work experience.

Soft Skills:

• Strong analytical and problem-solving skills.
• Excellent verbal and written communication skills.
• Ability to work in a fast-paced, team-oriented environment.
• Strong attention to detail and commitment to delivering high-quality work.
• Proactive and self-motivated with the ability to manage multiple tasks efficiently.