Web scraping is a topic in which I am deeply interested. I have completed numerous successful web scraping projects and have launched several products in this field. Most of these projects were developed using Node.js, Puppeteer, cURL, and Playwright.
The journey began when several fellow B2B SaaS founders expressed a common desire: to seamlessly enrich their signup processes with pertinent company data. They were frustrated with the inaccessibility of the Crunchbase API, which seemed only available to Fortune 500 behemoths with deep pockets.
Puppeteer is an incredibly useful tool for automating web browsers. It allows to run headless (or non-headless) Chrome instances, automatically interacting with websites and pages in ways that would normally require manual input from a user or other scripts. In a lot of cases (particularly in web scraping tasks) it
Executing http(s) request via proxy might be helpful in a lot of cases, this helps to make your http request look like it was executed from a different country or location. Setting proxy in node-fetch Node.js package is not simple as in Axios (where we can set a
There is a number of ways to perform web requests in Node.js: node-fetch, axios, got, superagent Node.js can perform HTTP requests without additional packages While I don't ever use this approach because of it's poor developer ergonomics (using EventEmitter to collect the response data is just too verbose
Okay, let's admit it - web scraping via Puppeteer and Playwright is the most versatile and flexible way of web scraping nowadays. Unfortunately it's also the most cumbersome, time consuming way of scraping, and sometimes it feels a little bit like voodoo magic. This is a post about my long
Web scraping is a big and hot topic now, and PHP is a pretty fast language which is convenient for rapid prototyping and is wildly popular across web developers. I have pretty extensive experience building complex scrapers in Node.js, but before this I spent a lot of years actively
