Web Crawling & Scraping Engineer
Primary stack
Nice to have's
Job description
About the Position
We seek a highly skilled and motivated Web Crawling & Scraping Engineer. We crawl over 140k news websites daily and we are looking for someone who is passionate about Web Crawling the same way we are. Curious to explore new ways of handling difficult website cases and automating our crawling techniques.
Responsibilities
- Crawling Platform: Design, construct, test, and maintain robust, reliable, and scalable crawling pipeline infrastructure.
- Automation: Add an automatic way of fixing non-working crawlers.
- Metrics: Provide metrics on website coverage.
- Data Pipeline: Design, construct, test, and maintain robust, reliable, and data pipeline infrastructure.
- Automation and Unit Tests: Implement automation and unit tests.
- Optimization: Optimize server performance and resource utilization of crawling infrastructure.
- Review and Improvement: Regularly review and improve system performance and scalability.
- Collaboration and Documentation: Maintain accurate and up-to-date documentation of server configurations, procedures, and policies.
- Technical Support: Provide technical support and training to team members as needed.
Example Tasks
- Introduce a new automatic way of crawling a website that does not work with existing techniques.
- Come up with an idea on how to verify why a specific crawler stopped working and fix it automatically.
- Use LLM methods to improve crawling methods.
Requirements
- Proven experience as a Web Crawling & Scraping Engineer or similar role.
- Proficiency in Web Scraping and Web Crawling Techniques.
- Experience with Streaming/batch data processing frameworks such as RabbitMQ.
- Solid knowledge of SQL and NoSQL databases.
- Proficiency with Kubernetes and Docker.
- Strong problem-solving skills.
- Excellent communication and collaboration skills.
Nice to Have
- Experience with ElasticSearch (OpenSearch).
Your KPIs
- Number of non-working crawlers per website (should be small).
- The time between a crawler goes down and we come up with a fix (should be small).
We Offer
- Competitive salary and equity.
- Up to 24 days of vacation & 16 days of sick leave/holidays (all fully paid).
- Learning and development compensation.
- One meeting-free day per week.
- Co-working Budget.
- Training Budget.
- We provide all the necessary equipment to work comfortably and efficiently from home.
- Yearly company retreats (2024 - Canary Islands, 2023 - French Alpes).
Needed Tools
- Scrapy
- Crawlee
Apply Now
Other Roles
Technical Account Executive
Support enterprise clients during rapid growth with freedom, trust, and hands-on experience working with top global companies.
SDR (Sales Development Representative)
Identify and qualify leads through outreach, schedule demos, and support sales and marketing efforts while keeping records updated in the CRM.
© NewsCatcher. This job description was sourced from the employer's public career page. TheJob is not the employer — we index the posting and route candidates to the source. All content rights and hiring decisions belong to the employer.
Mission: To help companies always catch important news.What We Do: NewsCatcher is building the largest and most accurate database of news articles published online. We crawl millions of web pages daily to detect over 1,000,000 online news articles, empowering businesses with real-time, global news data. Our advanced API is designed for seamless integration and customizable news analytics, enabling companies to harness the power of news and stay informed at all times.Team: Our dedicated team consists of 18 specialists committed to delivering the most comprehensive and up-to-date news data available. We are proud to be a part of the Y Combinator Summer 2022 batch (YC S22).