This role is for one of the Weekday's clients
We are seeking a talented and motivated Software Development Engineer (SDE) with expertise in data scraping to join our team. The ideal candidate will have strong experience in designing and implementing web scraping solutions, data extraction techniques, and working with structured and unstructured data. You will collaborate closely with the data engineering and product teams to ensure high-quality, reliable data scraping for business insights and decision-making.
Key Responsibilities
- Develop, maintain, and optimize web scraping tools to extract large volumes of data from multiple websites and APIs.
- Design and implement scalable data pipelines for efficient data collection, storage, and processing.
- Write clean, efficient, and maintainable code for data scraping and parsing.
- Implement solutions for dynamic website scraping, handling AJAX requests, Captchas, and other challenges.
- Ensure data quality by validating, cleaning, and formatting scraped data.
- Collaborate with cross-functional teams to understand data requirements and integrate data into the company's data infrastructure.
- Monitor and troubleshoot scraping jobs, ensuring timely and accurate data collection.
- Stay up-to-date with the latest trends and technologies in web scraping, automation, and data extraction.
Required Skills And Qualifications
- Bachelors degree in Computer Science, Engineering, or a related field.
- 2+ years of experience in data scraping, web crawling, or data extraction.
- Proficiency in programming languages like Python, Java, or Node.js.
- Strong knowledge of libraries and frameworks such as BeautifulSoup, Scrapy, Selenium, Puppeteer, or Playwright.
- Experience working with APIs (RESTful, GraphQL) and understanding of HTTP/HTTPS protocols.
- Hands-on experience with database systems (SQL, NoSQL) for storing and processing large data sets.
- Familiarity with cloud services like AWS, GCP, or Azure for scaling data scraping solutions.
- Experience handling dynamic web pages, Captchas, and rate-limiting challenges.
- Solid understanding of data structures, algorithms, and software design principles.
- Strong debugging, problem-solving, and optimization skills.
Preferred Qualifications
- Experience in building scalable, distributed data scraping systems.
- Familiarity with DevOps practices, CI/CD pipelines, and version control (Git).
- Exposure to big data tools such as Hadoop, Spark, or Kafka.
- Knowledge of web automation and bots development.
Skills: web scraping,data,data extraction
