Using Puppeteer with Rotating Proxies

Puppeteer is a Node.js library that allows you to control Chrome browser from JS code. Most things that you can do manually in the browser can be done using Puppeteer. Here are a few examples to get you started:

  • Generate screenshots and PDFs of pages.
  • Crawl a SPA and generate pre-rendered content (i.e. "SSR").
  • Automate form submission, UI testing, keyboard input, etc.
  • Create an up-to-date, automated testing environment. Run your tests directly in the latest version of Chrome using the latest JavaScript and browser features.
  • Capture a timeline trace of your site to help diagnose performance issues.

If you need to scrape SPA or some website with heavy javascript usage - puppeteer in many cases is the way to go. It can also save a lot of time in many other situations because of simple high level API.

Here is an example of how Puppeteer can be used in combination with our rotating proxy server. The code snippet below also shows how you can set additional headers to control BotProxy (please refer our docs for full information about all supported control headers and APIs).

const puppeteer = require('puppeteer');

puppeteer.launch({
    args: ['--proxy-server=x.botproxy.net:8080']
}).then(async browser => {
    const page = await browser.newPage();
    page.authenticate({
        username: 'proxy-user',
        password: 'proxy-password'
    });
    await page.setExtraHTTPHeaders({
        'X-BOTPROXY-COUNTRY': 'US',
        'X-BOTPROXY-SESSION': '123'
    });
    await page.goto('http://httpbin.org/ip');
    // other actions...
    let content = await page.content();
    console.log(content);
    await browser.close();
});

Note that adding extra headers only works when you access plain http resources. For https resources you need to use username API or configure desired settings in your account dashboard for the proxy user.

As you can see it is very easy to start using rotating proxies in your existing puppeteer projects and will require only a coupe of lines of code.



BotProxy: Anonymous HTTP Rotating Proxy

Plug BotProxy into your web scraping application and it will automatically route your parsers requests through one of our outgoing proxy servers. Multiple locations, fresh IP addresses every day. Typical integrations take less than 5 minutes into any script or application.

Start Free Trial or learn more