How to set proxy in node-fetch

Executing http(s) request via proxy might be helpful in a lot of cases, this helps to make your http request look like it was executed from a different country or location.

Setting proxy in node-fetch Node.js package is not simple as in Axios (where we can set a proxy by passing simple JS object with options), in node-fetch we need to pass an Agent with proxy set up, so it is a bit more manual work. But, this is also a good thing, because we can use latest and greatest proxy package from npm for proxy support, instead of relying on what axios maintainers have chosen.

Let's use https://github.com/delvedor/hpagent for proxy agent creation, this is a pretty fresh proxy package with good keep-alive support (so proxy won't close the connection with our server, which might result in better performance). There are popular older proxy packages available on npm, but they have some sort of problems, for example the package that I was using a lot is   https://www.npmjs.com/package/proxy-agent and it was last updated more than a year ago, it really lacks support from maintainers.

Unfortunately, hpagent does not support socks5 proxies now so if you have socks5 proxies I'd recommend to use https://www.npmjs.com/package/proxy-agent.

Installation

npm i node-fetch hpagent

Setting HTTP proxy for node-fetch

import fetch from 'node-fetch';

// import HttpsProxyAgent in case you have https:// proxy 
// (do not confuse with https:// target website protocol)
const { HttpProxyAgent } = require('hpagent');

const response = await fetch('https://target-website.com', {
  agent: new HttpProxyAgent({
    keepAlive: true,
    keepAliveMsecs: 1000,
    maxSockets: 256,
    maxFreeSockets: 256,
    scheduling: 'lifo',
    proxy: 'http://proxy-username:pw@proxy-host:port'
  })
});

const body = await response.text();

console.log(body);

hpagent additional options

hpagent accepts a lot of options, for example for basic auth, or for certificates. Read more in hpagent docs on their Github.

const agent = new HttpProxyAgent({
  keepAlive: true,
  keepAliveMsecs: 1000,
  maxSockets: 256,
  maxFreeSockets: 256,
  proxy: 'https://localhost:8080',
  proxyConnectOptions: {
    headers: {
      'Proxy-Authorization': 'Basic YWxhZGRpbjpvcGVuc2VzYW1l',
    },
    ca: [ fs.readFileSync('custom-proxy-cert.pem') ]
  }
})

hpagent integrates nicely with all popular npm packages which receive node Agent instance for proxies (so, it works with Got, too!). If you don't now what is node.js "Agent" and you are curious, I'd recommend to read this StackOverflow post: https://stackoverflow.com/questions/37471335/node-js-relationship-among-http-server-http-agent-sockets-and-http-request and node.js docs reference page related to HTTP requests.

node-fetch vs Axios vs Got vs Superagent

Node.js has a number of cool networking packages for HTTP requests. Which one to choose for regular usage and for web scraping needs? Read my overview of these libraries with usage examples and lost of cons for each of them.

If you are doing web scraping and node-fetch and got packages are already not good enough for your tasks, see my setting proxy in Puppeteer article.