Responsibilities:
As a focused Data Engineer / Python Developer, you will be responsible for extracting and ingesting data from websites using web crawling tools. In this role, you will own the creation process of these tools, services, and workflows to improve crawl/ scrape analysis, reports, and data management. We will rely on you to test the data and the scrape to ensure accuracy and quality. You will own the process to identify and rectify any issues with breaks and scale scrapes as needed.
Requirements:
- Passionate about Data Engineering
- Confident in running large scale web scrapes
- Solid Python and JavaScript knowledge
- Familiarity with the Chrome DevTools and how to use each of its features
- Ability to work independently and juggle priorities to thrive in a fast-paced environment
- Familiarity with Linux/UNIX, HTTP, HTML, and Networking
- Good understanding of website structure and composition
- Familiarity with techniques and tools for crawling, extracting and processing
- Familiarity with system monitoring/administration tools
- Familiarity with version control, open-source practices, and code review
- Familiarity with applications designed to display archived web content
- Great communication skills (written and Spoken in English)
- Master's Degree in Computer Science or a related field or the equivalent demonstrated experience
