Executing http(s) request via proxy might be helpful in a lot of cases, this helps to make your http request look like it was executed from a different country or location.

Setting proxy in node-fetch Node.js package is not simple as in Axios (where we can set a proxy by passing simple JS object with options), in node-fetch we need to pass an Agent with proxy set up, so it is a bit more manual work. But, this is also a good thing, because we can use latest and greatest proxy package from npm for proxy support, instead of relying on what axios maintainers have chosen.

Let's use https://github.com/delvedor/hpagent for proxy agent creation, this is a pretty fresh proxy package with good keep-alive support (so proxy won't close the connection with our server, which might result in better performance). There are popular older proxy packages available on npm, but they have some sort of problems, for example the package that I waas using a lot is   https://www.npmjs.com/package/proxy-agent and it was last updated more than a year ago, it really lacks support from maintainers.

Unfortunately, hpagent does not support socks5 proxies now so if you have socks5 proxies I'd recommend to use https://www.npmjs.com/package/proxy-agent.

Installation

npm i node-fetch hpagent

Setting HTTP proxy for node-fetch

import fetch from 'node-fetch';

// use HttpsProxyAgent only if you have https proxy
const { HttpProxyAgent, HttpsProxyAgent } = require('hpagent');

const response = await fetch('https://target-website.com', {
  agent: new HttpProxyAgent({
    keepAlive: true,
    keepAliveMsecs: 1000,
    maxSockets: 256,
    maxFreeSockets: 256,
    scheduling: 'lifo',
    proxy: 'http://proxy-username:pw@proxy-host:port'
  })
});

const body = await response.text();

console.log(body);

hpagent additional options

hpagent accepts a lot of options, for example for basic auth, or for certificates. Read more in hpagent docs on their Github.

const agent = new HttpProxyAgent({
  keepAlive: true,
  keepAliveMsecs: 1000,
  maxSockets: 256,
  maxFreeSockets: 256,
  proxy: 'https://localhost:8080',
  proxyConnectOptions: {
    headers: {
      'Proxy-Authorization': 'Basic YWxhZGRpbjpvcGVuc2VzYW1l',
    },
    ca: [ fs.readFileSync('custom-proxy-cert.pem') ]
  }
})

node-fetch vs Axios vs Got vs Superagent

Node.js has a number of cool networking packages for HTTP requests. Which one to choose for regular usage and for web scraping needs? Read my overview of these libraries with usage examples and lost of cons for each of them.