Documentation

BotProxy Rotating HTTP Proxy allows you to distribute your requests through one of 15+ geographical locations around the world. Currently we have locations in US, Canada, Europe, Asia and Australia. Each location change its IP address daily. We also provide access through 1000+ open proxy servers.

BotProxy is an ideal solution to perform regular data update from many different websites or when you need to easily change geographical location of your requests.

Quick Start

Select your programming language to see how to integrate BotProxy into your application. Each request should be authenticated by either sending your user key and its password or by whitelisting origin IP address (can be done in your account dashboard).

curl --proxy x.botproxy.net:8080 --proxy-user user-key:key-password "https://httpbin.org/ip"
#!/usr/bin/env node
require('request-promise')({
    url: 'https://httpbin.org/ip',
    proxy: 'http://user-key:[email protected]:8080'
}).then(function(data){ console.log(data); }, function(err){ console.error(err); });
package example;

import org.apache.http.HttpHost;
import org.apache.http.client.fluent.*;

public class Example {
    public static void main(String[] args) throws Exception {
        HttpHost proxy = new HttpHost("x.botproxy.net", 8080);
        String res = Executor.newInstance()
            .auth(proxy, "user-key", "key-password")
            .execute(Request.Get("https://httpbin.org/ip").viaProxy(proxy))
            .returnContent().asString();
        System.out.println(res);
    }
}
using System;
using System.Net;

class Example
{
    static void Main()
    {
        var client = new WebClient();
        client.Proxy = new WebProxy("x.botproxy.net:8080");
        client.Proxy.Credentials = new NetworkCredential("user-key", "key-password");
        Console.WriteLine(client.DownloadString("https://httpbin.org/ip"));
    }
}
Imports System.Net

Module Example
    Sub Main()
        Dim Client As New WebClient
        Client.Proxy = New WebProxy("http://x.botproxy.net:8080")
        Client.Proxy.Credentials = New NetworkCredential("user-key", "key-password")
        Console.WriteLine(Client.DownloadString("https://httpbin.org/ip"))
    End Sub
End Module
<?php
$curl = curl_init('https://httpbin.org/ip');
curl_setopt($curl, CURLOPT_PROXY, 'http://x.botproxy.net:8080');
curl_setopt($curl, CURLOPT_PROXYUSERPWD, 'user-key:key-password');
curl_exec($curl);
?>

Standard Library example

#!/usr/bin/env python
import urllib.request
opener = urllib.request.build_opener(
    urllib.request.ProxyHandler(
        {'http': 'http://user-key:[email protected]:8080',
         'https': 'http://user-key:[email protected]:8080'}))
print(opener.open('https://httpbin.org/ip').read())

Using python requests

import requests

res = requests.get(
    'http://httpbin.org/ip',
    proxies={
        'http': 'http://user-key:[email protected]:8080',
        'https': 'http://user-key:[email protected]:8080'
        },
    headers={
        'X-BOTPROXY-COUNTRY': 'US'
        })
print(res.text)
#!/usr/bin/ruby

require 'uri'
require 'net/http'

uri = URI.parse('https://httpbin.org/ip')
proxy = Net::HTTP::Proxy('x.botproxy.net', 8080, 'user-key', 'key-password')

req = Net::HTTP::Get.new(uri.path)

result = proxy.start(uri.host,uri.port) do |http|
    http.request(req)
end

puts result.body
#!/usr/bin/perl
use LWP::UserAgent;
my $agent = LWP::UserAgent->new();
$agent->proxy(['http', 'https'], "http://user-key:key-password\@x.botproxy.net:8080");
print $agent->get('https://httpbin.org/ip')->content();
  1. Click the Chrome menu on the browser toolbar.
  2. Select "Settings".
  3. Click "Show advanced settings".
  4. In the "Network" section, click "Change proxy settings".
  5. Click "LAN settings".
  6. Select the "Use a proxy server for your LAN" check box.
  7. Enter "Address": x.botproxy.net
  8. Enter "Port": 8080
  9. Save changes by pressing "OK"

IP Rotation

When you use BotProxy proxy server without specifying additional parameters your outgoing IP address is changed every minute. The new IP address is assigned from the pool of proxies from locations authorized to your account. You can view current proxy IP addresses on Locations tab in your account dashboard. Proxies in all locations are renewed every 24 hours.

It is possible to change IP with every request. You need to pass X-BOTPROXY-SESSION header with any random value (or use username API) or by setting session age limit to 1 second in proxy user settings. So a new session with a next available IP is created with each request. But we don't recommend doing so. Proxy sessions are used to control what external IP is used by a connected client. If you use rotating proxy that changes IP address with every request in a browser (or other browser automation tools like selenium, or headless chrome) then when you openinig a single page different page resources are loaded from different IP addresses. This can be easily tracked by anti bot systems which analyze access logs and session cookies. And all proxy IPs will be blocked very soon. Thus to prevent proxy detection we stick with one IP for some period of time emulating regular user behavior.

Please also note that if you specify location, country or session id using HTTP headers or username IP then current session is renewed with every request and only expired if inactive for 1 minute. The outgoing IP address will not be changed.

Open Proxy Rotation

When using OpenProxy location (xo) all available proxies are sorted by their response time. So when you start the first session you'll be using the fastest available proxy. Next session will give you second fastest proxy and so on until you reach the slowest proxy from our list. Then rotation is restarted from the beginning.

You can start rotation from the beginning of the list at any time by providing X-BOTPROXY-RESET-ROTATION header.

BotProxy API

Client: Your machine running the scraping code. It connects to the Super Proxy to request URLs you wish to scrape.

Super Proxy: BotProxy endpoint that distributes your requests through our network to the BotProxy outgoing proxy servers.

Proxy Peer: The final server that sends your URL request to to the target website. This is the IP that the website sees, it is a server that is part of the BotProxy network.

Using the system — Overview

You send your requests to the Super Proxy, and the Super Proxy relays them through a outgoing Proxy Peer on our network. You can specify to the Super Proxy which country the Proxy Peer should be in, when to force a switch to a different IP address within the country, and set other parameters. The control is done by specifying X-BOTPROXY-COUNTRY or X-BOTPROXY-LOCATION headers within your request.

Obtaining address of Super Proxy

Super Proxies are available under x.botproxy.net. This DNS name resolves to the set of currently fastest Super Proxies.

Session: IP persistence

In many cases you may want to run several parallel sessions from the same Client, use different Proxy Peers per session, and be able to change them when needed. This can be done by adding the session ID to the proxy request headers X-BOTPROXY-SESSION. Generate the random number on thread startup, and change it when you want to change the Proxy Peer assigned for the thread's connection. Session ID can be any random string/counter: requests with same session string will use the same Proxy Peer (as long as possible); requests with different session strings will be assigned different Proxy Peers.

To force an IP change, just modify the session ID.

If an assigned Proxy Peer becomes unavailable, the Super Proxy will change it automatically, even if you do not change the session ID.

HTTP Headers Specifications

All headers starting with X-BOTPROXY- are stripped out from original request so they will not be sent to target website. Response headers are only added to plain HTTP response as we can't inject any data into secure communication. For HTTPS requests botproxy response headers are added immediatelly after HTTP/1.0 200 Connection established response from proxy. Currently the following API request and response headers are supported:

HeaderDescriptionValues
Request headers
X-BOTPROXY-LOCATION Limit outgoing proxy IPs to specified proxy location. Only specified outgoing location is used. It has precedence over country header so if this is set, country header is ignored. Location code. See location list in your account dashboard.
X-BOTPROXY-COUNTRYLimit outgoing proxy IPs to specified country. Only locations in this country will be used. ISO country code of the available proxy locations. See location list in your account dashboard.
X-BOTPROXY-SESSION Session identifier. All requests with the same session ID will use the same external IP address. Session will be reset after 1 minute inactivity time. If you need longer sessions you need to make keep alive requests. Any alphanumeric string up to 50 bytes long
X-BOTPROXY-CONNECT-TIMEOUT Connection timeout when connecting to upstream proxy. Default value is 30 seconds. Useful when using OpenProxies to control how much time to wait for open proxy to establish connection. Timeout in seconds
X-BOTPROXY-RESET-ROTATION Reset rotation index. Next proxy will be selected from the beginning of the list of available proxies sorted by response time. string: "true"
Response headers
X-BOTPROXY-IP IP address of the outgoing proxy server. This is the address that a remote website sees when processing your requests through botproxy. IPv4 Address

Please note, that when you run an HTTPS request you should send control headers separately from normal request headers. Normal http headers are encrypted and we cannot extract them from your request. Send botproxy control headers immediately after CONNECT line together with proxy authentication headers:

CONNECT httpbin.org:443 HTTP/1.1
X-BOTPROXY-SESSION: 123456
Proxy-Authorization: Basic bG90...

HTTP/1.0 200 Connection established
X-BOTRPOXY-IP: xx.xx.xx.xx
[encrypted data]

If you cannot insert additional headers in HTTPS request you can try the username API from the section below

Username API

In many cases there is no way to add custom HTTP headers. We provide additional username API to control outgoing locations using only proxy authentication credentials. To achive this you should add special suffixes to your proxy access username as follows. Let's say your proxy access user username is pxu77-0

  • Specify COUNTRY by adding +CN suffix to username. For example to limit outgoing locations to US only use pxu77-0+US as your proxy username
  • Specify LOCATION by adding +loc suffix: use pxu77-0+us-fl as your proxy username
  • Optionally you can add SESSION identifier to country or location by adding one more suffix as follows: pxu77-0+US+123456 or pxu77-0+us-fl+123456

The complete usage example to route outgoing traffic through US locations only and within session 123456 is:

curl --proxy x.botproxy.net:8080 --proxy-user "pxu77-0+us+123456:password" "https://httpbin.org/ip"

Locations

Currently we have proxy peers in datacenters in the following locations:

  • Sydney, Australia
  • Toronto, Canada
  • Frankfurt, Germany
  • Paris, France
  • Bangalore, India
  • Amsterdam, Netherlands
  • Singapore, Singapore
  • London, United Kingdom
  • US Central, Iowa, USA
  • US East, New York, USA
  • US West, San Francisco, USA
  • Open Proxies (~1000 IPs, various countries)

You can choose locations according to your subscription plan. Each location rotates IP addresses every 24 hours approx at 5am UTC.

Open Proxies

We maintain a list of known open proxy servers. If you choose "Open Proxies" location your request will be routed through one of these proxies. Your requests are forwarded through less reliable & higher latency servers. Open proxies IPs are located in many different countries around the world. These open proxy IPs typically do not stay online very long, and are not operated on reliable infrastructure. The tradeoff for this lack of reliability is a huge increase in quantity & variability of IP addresses.

The list is regularly updated and contains around 1000 working IP addresses around the world. If any request through an open proxy fails due to a proxy error, that error is recorded, and the request is re-tried up to 3 more times, using a different proxy for each retry. Failed proxy is immediatelly removed from the list. You can use X-BOTPROXY-CONNECT-TIMEOUT header to control how long to wait for connection to open proxy.

By default Open Proxies location is disabled on a proxy user. You need to manually enable it in your account dashboard.

Unmetered Bandwith: We provide an API to obtain proxies from the OpenProxy location. You can use the proxies directly to avoid counting traffic against you plan limit.

Secure Web Proxy

A secure web proxy is a web proxy that the browser communicates with via SSL, as opposed to clear text. In insecure public networks, such as airports or cafes, browsing over HTTP may leave the user vulnerable to cookie stealing, session hijacking or worse. A secure web proxy can add a significant layer of defense in these cases. Diagrams below explain different scenarios in details. Please note: not all clients (browsers) support secure proxies. Refer your client documentation and set HTTPS proxy as needed.

To use secure proxy in your applications you need to first establish a TLS connection to our secure proxy and then initiate regular HTTP proxy connection over established encrypted connection.

Find below an explanation of how requests go when using different kind of proxies:

HTTP request using HTTP proxy

path of HTTP request through HTTP Proxy

In this case request is sent from your client in plain text thus allowing your local network administrator or an attacker to intercept it and access all the data being sent. This is the least secure option.

HTTPS request using HTTP proxy

path of HTTPS request through HTTP Proxy

Target host name alongside your BotProxy proxy user credentials are transferred in plain text. Your client requests to create a tunnel to the specific host and port with CONNECT command and then makes secure communication via this tunnel. Parties on local network cannot intercept or hijack any data being transferred between you and remote host.

HTTP request using Secure (HTTPS) proxy

path of HTTP request through HTTPS Proxy

In this case your client establishes secure connection with our proxy server and no one can know what data is being transferred between you and target host. Moreover nobody in local network can know what is the target host. Even though the target host can be accessed using plain text HTTP protocol.

HTTPS request using Secure (HTTPS) proxy

path of HTTPS request through HTTPS Proxy

Requests and data are encrypted all the way through. Most secure option.

Additional Documentation and Examples

Demo Screencast

Watch the screencast below to get the idea of using BotProxy username API to control your outgoing IP address.

Limitations

Our proxies are not designed to query a single website on a high volume (more than 1GB/day) or request rate. We have a limited number of static IPs that are renewed every 24 hours. For example if you want to run a bot to automatically query Google all available IPs will be banned with CAPTCHA pretty soon and it will not work. You'll have to wait 24 hours to get fresh IPs. Only run such queries with "Open Proxies" location.

Use our datacenter proxies to hide your real location, avoid block by geolocation or API request/rate limits by IP, etc.

Please also check out Acceptable Use Policy. We ask that you please do not engage in prohibited activity on our network.