This directory contains a compilation of key players offering APIs for data extraction. I write a brief overview of each service. I also include a link to the official website, pricing, and other relevant information. Consider this work in progress. 🛠
Web scraping API is a type of tool that has relatively recently emerged, designed to simplify the process of web scraping by handling proxies, browsers, and CAPTCHAs. It allows to (almost) effortlessly extract HTML from any webpage via a straightforward API call, instead of spending time fiddling around proxies and implementing retry logic manually.
Anton SidashinAs a web developer and CTO with over 15 years of experience, I am passionate about building profitable small SaaS products and pursuing Go-to-Market strategy for them. My areas of expertise include high performance, networking technology and APIs, SRE, automation using Puppeteer.js, web scraping, and SQL databases.
My blog posts
Choosing a proxy for web scraping
Once you're familiar with basic web scraping tools like Scrapy, and you've scraped your first 1-2 websites, you'll probably get your first ban because your IP address has made too many requests
Building modern and production ready Node.js API: my boring setup
I know how this desire to try out a new flashy npm package in a quick project can easily paralyze my will and the whole project's progress over exploring new docs and code. So in this writeup I would like to give appreciation to all the tools and techniques that have proven to be effective for me.