diff --git a/docs/aiohttp.rst b/docs/aiohttp.rst index 5a522a7..a3e0503 100644 --- a/docs/aiohttp.rst +++ b/docs/aiohttp.rst @@ -1,28 +1,154 @@ aiohttp ======= -The `aiohttp `_ library is an async HTTP client/server framework for Python. This page describes how to use aiohttp with proxies and how to interact with proxy headers. +The `aiohttp `_ library is an async HTTP client/server framework for Python. This page describes how to use aiohttp with proxies and how to send and receive custom proxy headers. + +Getting Started +--------------- + +This section shows you how to quickly get up and running with proxy headers in aiohttp. + +**Prerequisites:** + +1. Install the packages: + + .. code-block:: bash + + pip install python-proxy-headers aiohttp + +2. Import the module: + + .. code-block:: python + + from python_proxy_headers import aiohttp_proxy + +**Quick Example - Send and Receive Proxy Headers:** + +.. code-block:: python + + import asyncio + from python_proxy_headers import aiohttp_proxy + + async def main(): + async with aiohttp_proxy.ProxyClientSession() as session: + async with session.get( + 'https://api.ipify.org?format=json', + proxy='http://PROXYHOST:PORT', + proxy_headers={'X-ProxyMesh-Country': 'US'} + ) as response: + # Access the response data + data = await response.json() + print(data) # {"ip": "..."} + + # Access proxy response headers + print(response.headers.get('X-ProxyMesh-IP')) + + asyncio.run(main()) + +That's it! The ``ProxyClientSession`` handles sending your custom headers to the proxy and makes proxy response headers available in the response. Using Proxies with aiohttp --------------------------- aiohttp provides built-in support for proxies through the ``proxy`` parameter in request methods. You can specify a proxy URL for each request. -Basic Proxy Usage -~~~~~~~~~~~~~~~~~ +Basic Proxy Usage (Standard aiohttp) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -To use a proxy with aiohttp, you can pass the ``proxy`` parameter to any request method: +To use a proxy with standard aiohttp: .. code-block:: python import aiohttp - async with aiohttp.ClientSession() as session: - async with session.get('https://api.ipify.org?format=json', - proxy="http://PROXYHOST:PORT") as r: - text = await r.text() + import asyncio + + async def main(): + async with aiohttp.ClientSession() as session: + async with session.get( + 'https://api.ipify.org?format=json', + proxy='http://PROXYHOST:PORT' + ) as response: + text = await response.text() + print(text) + + asyncio.run(main()) This routes the request through the specified proxy server. +Proxy Authentication +~~~~~~~~~~~~~~~~~~~~ + +To use a proxy that requires authentication: + +.. code-block:: python + + import aiohttp + import asyncio + + async def main(): + async with aiohttp.ClientSession() as session: + # Method 1: Include credentials in the URL + async with session.get( + 'https://api.ipify.org?format=json', + proxy='http://username:password@PROXYHOST:PORT' + ) as response: + text = await response.text() + + # Method 2: Use proxy_auth parameter + auth = aiohttp.BasicAuth('username', 'password') + async with session.get( + 'https://api.ipify.org?format=json', + proxy='http://PROXYHOST:PORT', + proxy_auth=auth + ) as response: + text = await response.text() + + asyncio.run(main()) + +Session-Level Proxy +~~~~~~~~~~~~~~~~~~~ + +You can set a default proxy for all requests in a session: + +.. code-block:: python + + import aiohttp + import asyncio + + async def main(): + async with aiohttp.ClientSession( + proxy='http://PROXYHOST:PORT', + proxy_auth=aiohttp.BasicAuth('user', 'pass') + ) as session: + # All requests will use this proxy + async with session.get('https://api.ipify.org?format=json') as response: + text = await response.text() + + asyncio.run(main()) + +Environment Variables +~~~~~~~~~~~~~~~~~~~~~ + +aiohttp can read proxy settings from environment variables when ``trust_env=True``: + +.. code-block:: bash + + export HTTP_PROXY="http://PROXYHOST:PORT" + export HTTPS_PROXY="http://PROXYHOST:PORT" + +.. code-block:: python + + import aiohttp + import asyncio + + async def main(): + async with aiohttp.ClientSession(trust_env=True) as session: + # Will automatically use proxies from environment variables + async with session.get('https://api.ipify.org?format=json') as response: + text = await response.text() + + asyncio.run(main()) + Sending Custom Proxy Headers ----------------------------- @@ -31,28 +157,45 @@ While it's not documented, aiohttp does support passing in custom proxy headers .. code-block:: python import aiohttp - async with aiohttp.ClientSession() as session: - async with session.get('https://api.ipify.org?format=json', - proxy="http://PROXYHOST:PORT", - proxy_headers={'X-ProxyMesh-Country': 'US'}) as r: - text = await r.text() + import asyncio + + async def main(): + async with aiohttp.ClientSession() as session: + async with session.get( + 'https://api.ipify.org?format=json', + proxy='http://PROXYHOST:PORT', + proxy_headers={'X-ProxyMesh-Country': 'US'} + ) as response: + text = await response.text() + + asyncio.run(main()) The ``proxy_headers`` parameter allows you to send custom headers to the proxy server. This is useful for controlling proxy behavior, such as selecting a specific country or IP address. Receiving Proxy Response Headers --------------------------------- -However, if you want to get proxy response headers, you should use our extension module ``python_proxy_headers.aiohttp_proxy``: +Standard aiohttp does not expose proxy response headers from the CONNECT request. To get proxy response headers, use our extension module ``python_proxy_headers.aiohttp_proxy``: .. code-block:: python + import asyncio from python_proxy_headers import aiohttp_proxy - async with aiohttp_proxy.ProxyClientSession() as session: - async with session.get('https://api.ipify.org?format=json', - proxy="http://PROXYHOST:PORT", - proxy_headers={'X-ProxyMesh-Country': 'US'}) as r: - text = await r.text() - proxy_ip = r.headers['X-ProxyMesh-IP'] + + async def main(): + async with aiohttp_proxy.ProxyClientSession() as session: + async with session.get( + 'https://api.ipify.org?format=json', + proxy='http://PROXYHOST:PORT', + proxy_headers={'X-ProxyMesh-Country': 'US'} + ) as response: + data = await response.json() + + # Proxy response headers are now available + proxy_ip = response.headers.get('X-ProxyMesh-IP') + print(f"Request was made through: {proxy_ip}") + + asyncio.run(main()) The ``ProxyClientSession`` extends the standard ``ClientSession`` to make proxy response headers available in the response headers. This allows you to access information from the proxy server, such as the IP address that was assigned to your request. @@ -68,11 +211,13 @@ Here's a complete example showing how to use aiohttp with proxy headers: async def main(): async with aiohttp_proxy.ProxyClientSession() as session: - async with session.get('https://api.ipify.org?format=json', - proxy="http://PROXYHOST:PORT", - proxy_headers={'X-ProxyMesh-Country': 'US'}) as r: - data = await r.json() - proxy_ip = r.headers.get('X-ProxyMesh-IP') + async with session.get( + 'https://api.ipify.org?format=json', + proxy='http://PROXYHOST:PORT', + proxy_headers={'X-ProxyMesh-Country': 'US'} + ) as response: + data = await response.json() + proxy_ip = response.headers.get('X-ProxyMesh-IP') print(f"Your IP: {data['ip']}") print(f"Proxy IP: {proxy_ip}") @@ -116,7 +261,7 @@ The main entry point for using proxy headers with aiohttp. This class extends `` from python_proxy_headers.aiohttp_proxy import ProxyClientSession async with ProxyClientSession() as session: - async with session.get('https://example.com', proxy="http://PROXYHOST:PORT") as r: + async with session.get('https://example.com', proxy='http://PROXYHOST:PORT') as r: proxy_ip = r.headers.get('X-ProxyMesh-IP') The ``ProxyClientSession`` constructor accepts all the same arguments as ``aiohttp.ClientSession``, and automatically sets: @@ -172,4 +317,3 @@ The extension classes work together in the following flow: 4. **ProxyClientResponse** merges the proxy headers into the response's ``headers`` property This allows proxy response headers to be transparently available in your application without any special handling. - diff --git a/docs/httpx.rst b/docs/httpx.rst index 0b583eb..970d430 100644 --- a/docs/httpx.rst +++ b/docs/httpx.rst @@ -1,61 +1,199 @@ httpx ===== -`HTTPX `_ is a fully featured HTTP client for Python 3, which provides sync and async APIs, and support for both HTTP/1.1 and HTTP/2. This page describes how to use httpx with proxies and how to interact with proxy headers. +`HTTPX `_ is a fully featured HTTP client for Python 3, which provides sync and async APIs, and support for both HTTP/1.1 and HTTP/2. This page describes how to use httpx with proxies and how to send and receive custom proxy headers. + +Getting Started +--------------- + +This section shows you how to quickly get up and running with proxy headers in httpx. + +**Prerequisites:** + +1. Install the packages: + + .. code-block:: bash + + pip install python-proxy-headers httpx + +2. Import the module: + + .. code-block:: python + + from python_proxy_headers import httpx_proxy + +**Quick Example - Send and Receive Proxy Headers:** + +.. code-block:: python + + import httpx + from python_proxy_headers import httpx_proxy + + # Create a proxy with custom headers + proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'}) + + # Make a request using our helper function + r = httpx_proxy.get('https://api.ipify.org?format=json', proxy=proxy) + + # Access the response data + print(r.json()) # {"ip": "..."} + + # Access proxy response headers + print(r.headers.get('X-ProxyMesh-IP')) # The IP address assigned by the proxy + +That's it! The ``httpx_proxy`` module handles sending your custom headers to the proxy and makes proxy response headers available in the response. Using Proxies with httpx ------------------------ -httpx provides built-in support for proxies through the ``httpx.Proxy`` class. You can create a proxy object and use it with httpx clients. +httpx provides built-in support for proxies through the ``proxy`` parameter. You can pass a proxy URL to the client. -Basic Proxy Usage -~~~~~~~~~~~~~~~~~ +Basic Proxy Usage (Standard httpx) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To use a proxy with standard httpx: + +.. code-block:: python + + import httpx + + # Simple proxy URL + with httpx.Client(proxy='http://PROXYHOST:PORT') as client: + r = client.get('https://api.ipify.org?format=json') + print(r.json()) + +This routes requests through the specified proxy server. + +Proxy Authentication +~~~~~~~~~~~~~~~~~~~~ + +To use a proxy that requires authentication: + +.. code-block:: python + + import httpx + + # Include credentials in the URL + with httpx.Client(proxy='http://username:password@PROXYHOST:PORT') as client: + r = client.get('https://api.ipify.org?format=json') + +Different Proxies for HTTP and HTTPS +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -httpx supports sending proxy headers by default, though it's not documented. You can use the ``httpx.Proxy`` class with custom headers: +You can configure different proxies for HTTP and HTTPS using mounts: .. code-block:: python import httpx - from httpx import HTTPProxyTransport + + proxy_mounts = { + 'http://': httpx.HTTPTransport(proxy='http://localhost:8030'), + 'https://': httpx.HTTPTransport(proxy='http://localhost:8031'), + } + with httpx.Client(mounts=proxy_mounts) as client: + r = client.get('https://api.ipify.org?format=json') + +SOCKS Proxies +~~~~~~~~~~~~~ + +httpx supports SOCKS proxies with an additional dependency: + +.. code-block:: bash + + pip install httpx[socks] + +.. code-block:: python + + import httpx + + with httpx.Client(proxy='socks5://user:pass@host:port') as client: + r = client.get('https://api.ipify.org?format=json') + +Sending Custom Proxy Headers +---------------------------- + +httpx supports sending proxy headers by default through the ``httpx.Proxy`` class: + +.. code-block:: python + + import httpx + proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'}) - transport = HTTPProxyTransport(proxy=proxy) - with httpx.Client(mounts={'http://': transport, 'https://': transport}) as client: + + with httpx.Client(proxy=proxy) as client: r = client.get('https://api.ipify.org?format=json') -This creates a proxy with custom headers and uses it with an httpx client. +The ``headers`` parameter on the ``Proxy`` object allows you to send custom headers to the proxy server. Receiving Proxy Response Headers --------------------------------- -But to get response headers from a proxy server, you need to use our extension module ``python_proxy_headers.httpx_proxy``: +Standard httpx does not expose proxy response headers from the CONNECT request. To get response headers from a proxy server, use our extension module ``python_proxy_headers.httpx_proxy``: .. code-block:: python import httpx from python_proxy_headers.httpx_proxy import HTTPProxyTransport + proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'}) transport = HTTPProxyTransport(proxy=proxy) + with httpx.Client(mounts={'http://': transport, 'https://': transport}) as client: r = client.get('https://api.ipify.org?format=json') - r.headers['X-ProxyMesh-IP'] + # Proxy response headers are now available + proxy_ip = r.headers.get('X-ProxyMesh-IP') + print(f"Request was made through: {proxy_ip}") The ``HTTPProxyTransport`` class from our extension module extends the standard transport to make proxy response headers available in the response headers. Helper Methods -------------- -This module also provides helper methods similar to ``requests`` for convenience: +For simpler use cases, this module provides helper methods similar to ``requests``: .. code-block:: python import httpx from python_proxy_headers import httpx_proxy + proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'}) + + # GET request r = httpx_proxy.get('https://api.ipify.org?format=json', proxy=proxy) - r.headers['X-ProxyMesh-IP'] + + # Access proxy response headers + print(r.headers.get('X-ProxyMesh-IP')) + +The helper module supports all standard HTTP methods: -The helper module supports all standard HTTP methods: ``get``, ``post``, ``put``, ``delete``, ``patch``, ``head``, and ``options``. +.. code-block:: python + + import httpx + from python_proxy_headers import httpx_proxy + + proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'}) + + # GET request + r = httpx_proxy.get('https://api.example.com', proxy=proxy) + + # POST request + r = httpx_proxy.post('https://api.example.com', json={'key': 'value'}, proxy=proxy) + + # PUT request + r = httpx_proxy.put('https://api.example.com/resource/1', json={'name': 'updated'}, proxy=proxy) + + # PATCH request + r = httpx_proxy.patch('https://api.example.com/resource/1', json={'status': 'active'}, proxy=proxy) + + # DELETE request + r = httpx_proxy.delete('https://api.example.com/resource/1', proxy=proxy) + + # HEAD request + r = httpx_proxy.head('https://api.example.com', proxy=proxy) + + # OPTIONS request + r = httpx_proxy.options('https://api.example.com', proxy=proxy) Async Support ------------- @@ -65,13 +203,21 @@ httpx supports async requests, so we provide an async extension too: .. code-block:: python import httpx + import asyncio from python_proxy_headers.httpx_proxy import AsyncHTTPProxyTransport - proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'}) - transport = AsyncHTTPProxyTransport(proxy=proxy) - async with httpx.AsyncClient(mounts={'http://': transport, 'https://': transport}) as client: - r = await client.get('https://api.ipify.org?format=json') - r.headers['X-ProxyMesh-IP'] + async def main(): + proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'}) + transport = AsyncHTTPProxyTransport(proxy=proxy) + + async with httpx.AsyncClient(mounts={'http://': transport, 'https://': transport}) as client: + r = await client.get('https://api.ipify.org?format=json') + + # Access proxy response headers + proxy_ip = r.headers.get('X-ProxyMesh-IP') + print(f"Proxy IP: {proxy_ip}") + + asyncio.run(main()) The ``AsyncHTTPProxyTransport`` works just like the sync version but for async clients. @@ -102,6 +248,7 @@ Asynchronous Example .. code-block:: python import httpx + import asyncio from python_proxy_headers.httpx_proxy import AsyncHTTPProxyTransport async def main(): @@ -115,45 +262,8 @@ Asynchronous Example print(f"Your IP: {data['ip']}") print(f"Proxy IP: {proxy_ip}") - import asyncio asyncio.run(main()) -Using Helper Methods -~~~~~~~~~~~~~~~~~~~~ - -For simpler use cases, you can use the helper methods: - -.. code-block:: python - - import httpx - from python_proxy_headers import httpx_proxy - - proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'}) - - # GET request - r = httpx_proxy.get('https://api.example.com', proxy=proxy) - - # POST request - r = httpx_proxy.post('https://api.example.com', json={'key': 'value'}, proxy=proxy) - - # PUT request - r = httpx_proxy.put('https://api.example.com/resource/1', json={'name': 'updated'}, proxy=proxy) - - # PATCH request - r = httpx_proxy.patch('https://api.example.com/resource/1', json={'status': 'active'}, proxy=proxy) - - # DELETE request - r = httpx_proxy.delete('https://api.example.com/resource/1', proxy=proxy) - - # HEAD request - r = httpx_proxy.head('https://api.example.com', proxy=proxy) - - # OPTIONS request - r = httpx_proxy.options('https://api.example.com', proxy=proxy) - - # Access proxy response headers - proxy_ip = r.headers.get('X-ProxyMesh-IP') - Streaming Responses ~~~~~~~~~~~~~~~~~~~~ @@ -187,10 +297,62 @@ Proxy headers are custom HTTP headers that can be used to communicate with proxy The exact headers available depend on your proxy provider. Check your proxy provider's documentation for the specific headers they support. +API Reference +------------- + +Helper Functions +~~~~~~~~~~~~~~~~ + +* ``request(method, url, *, proxy=None, **kwargs)`` - Make a request with the specified method +* ``get(url, *, proxy=None, **kwargs)`` - Make a GET request +* ``post(url, *, proxy=None, **kwargs)`` - Make a POST request +* ``put(url, *, proxy=None, **kwargs)`` - Make a PUT request +* ``patch(url, *, proxy=None, **kwargs)`` - Make a PATCH request +* ``delete(url, *, proxy=None, **kwargs)`` - Make a DELETE request +* ``head(url, *, proxy=None, **kwargs)`` - Make a HEAD request +* ``options(url, *, proxy=None, **kwargs)`` - Make an OPTIONS request +* ``stream(method, url, *, proxy=None, **kwargs)`` - Stream a response (context manager) + +HTTPProxyTransport +~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + from python_proxy_headers.httpx_proxy import HTTPProxyTransport + + transport = HTTPProxyTransport( + proxy=proxy, # httpx.Proxy object or proxy URL string + verify=True, # SSL verification + cert=None, # Client certificate + trust_env=True, # Trust environment variables + http1=True, # Enable HTTP/1.1 + http2=False, # Enable HTTP/2 + limits=DEFAULT_LIMITS, + **kwargs + ) + +AsyncHTTPProxyTransport +~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + from python_proxy_headers.httpx_proxy import AsyncHTTPProxyTransport + + transport = AsyncHTTPProxyTransport( + proxy=proxy, # httpx.Proxy object or proxy URL string + verify=True, # SSL verification + cert=None, # Client certificate + trust_env=True, # Trust environment variables + http1=True, # Enable HTTP/1.1 + http2=False, # Enable HTTP/2 + limits=DEFAULT_LIMITS, + **kwargs + ) + httpcore Integration -------------------- -Our httpx helper module internally provides extension classes for `httpcore `_, for handling proxy headers over tunnel connections. These classes extend httpcore's internal proxy implementation to capture and merge proxy response headers. +Our httpx extension module internally provides extension classes for `httpcore `_, for handling proxy headers over tunnel connections. These classes extend httpcore's internal proxy implementation to capture and merge proxy response headers. Extension Classes ~~~~~~~~~~~~~~~~~ @@ -225,4 +387,3 @@ Internal Usage These classes are used internally by ``HTTPProxyTransport`` and ``AsyncHTTPProxyTransport``. When you create a transport with a proxy, it automatically uses ``HTTPProxyHeaders`` (or ``AsyncHTTPProxyHeaders``) as the connection pool, which in turn uses ``ProxyTunnelHTTPConnection`` (or ``AsyncProxyTunnelHTTPConnection``) for HTTPS tunnel connections. If you're building custom functionality on top of httpcore, you can import and use these classes directly, but note that they depend on httpcore's internal APIs which may change between versions. - diff --git a/docs/requests.rst b/docs/requests.rst index 3c4bca6..a089262 100644 --- a/docs/requests.rst +++ b/docs/requests.rst @@ -1,49 +1,165 @@ requests ======== -The `requests `_ library is a simple HTTP library for Python. This page describes how to use requests with proxies and how to interact with proxy headers. +The `requests `_ library is a simple HTTP library for Python. This page describes how to use requests with proxies and how to send and receive custom proxy headers. + +Getting Started +--------------- + +This section shows you how to quickly get up and running with proxy headers in requests. + +**Prerequisites:** + +1. Install the packages: + + .. code-block:: bash + + pip install python-proxy-headers requests + +2. Import the module: + + .. code-block:: python + + from python_proxy_headers import requests_adapter + +**Quick Example - Send and Receive Proxy Headers:** + +.. code-block:: python + + from python_proxy_headers import requests_adapter + + # Make a request with custom proxy headers + r = requests_adapter.get( + 'https://api.ipify.org?format=json', + proxies={ + 'http': 'http://PROXYHOST:PORT', + 'https': 'http://PROXYHOST:PORT' + }, + proxy_headers={'X-ProxyMesh-Country': 'US'} + ) + + # Access the response data + print(r.json()) # {"ip": "..."} + + # Access proxy response headers + print(r.headers.get('X-ProxyMesh-IP')) # The IP address assigned by the proxy + +That's it! The ``requests_adapter`` module handles sending your custom headers to the proxy and makes proxy response headers available in the response. Using Proxies with requests --------------------------- The requests library provides built-in support for proxies through the ``proxies`` parameter. You can specify proxies for HTTP and HTTPS requests separately. -Basic Proxy Usage -~~~~~~~~~~~~~~~~~ +Basic Proxy Usage (Standard requests) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -To use a proxy with requests, you can pass the ``proxies`` parameter to any request method: +To use a proxy with standard requests: .. code-block:: python import requests + proxies = { 'http': 'http://PROXYHOST:PORT', 'https': 'http://PROXYHOST:PORT' } r = requests.get('https://api.ipify.org?format=json', proxies=proxies) + print(r.json()) This routes the request through the specified proxy server. You can specify different proxies for HTTP and HTTPS if needed. -Sending Custom Proxy Headers ------------------------------ +Proxy Authentication +~~~~~~~~~~~~~~~~~~~~ + +To use a proxy that requires authentication: + +.. code-block:: python + + import requests + + proxies = { + 'http': 'http://username:password@PROXYHOST:PORT', + 'https': 'http://username:password@PROXYHOST:PORT' + } + r = requests.get('https://api.ipify.org?format=json', proxies=proxies) + +Environment Variables +~~~~~~~~~~~~~~~~~~~~~ -The requests adapter builds on our ``urllib3_proxy_manager`` module to make it easy to pass in proxy headers and receive proxy response headers. +requests can also read proxy settings from environment variables: -Using the Extension Module -~~~~~~~~~~~~~~~~~~~~~~~~~~ +.. code-block:: bash -To send custom proxy headers and receive proxy response headers, use our ``python_proxy_headers.requests_adapter`` module: + export HTTP_PROXY="http://PROXYHOST:PORT" + export HTTPS_PROXY="http://PROXYHOST:PORT" + +Then in your code: + +.. code-block:: python + + import requests + + # Will automatically use proxies from environment variables + r = requests.get('https://api.ipify.org?format=json') + +Session-Level Proxies +~~~~~~~~~~~~~~~~~~~~~ + +You can configure proxies at the session level: + +.. code-block:: python + + import requests + + session = requests.Session() + session.proxies = { + 'http': 'http://PROXYHOST:PORT', + 'https': 'http://PROXYHOST:PORT' + } + r = session.get('https://api.ipify.org?format=json') + +Sending Custom Proxy Headers +---------------------------- + +Standard requests does not provide an easy way to send custom headers to the proxy (as opposed to the target server). Our ``requests_adapter`` module makes this easy: .. code-block:: python from python_proxy_headers import requests_adapter - r = requests_adapter.get('https://api.ipify.org?format=json', - proxies={'http': 'http://PROXYHOST:PORT', - 'https': 'http://PROXYHOST:PORT'}, - proxy_headers={'X-ProxyMesh-Country': 'US'}) - r.headers['X-ProxyMesh-IP'] + + r = requests_adapter.get( + 'https://api.ipify.org?format=json', + proxies={ + 'http': 'http://PROXYHOST:PORT', + 'https': 'http://PROXYHOST:PORT' + }, + proxy_headers={'X-ProxyMesh-Country': 'US'} + ) -The ``requests_adapter`` module supports all the standard requests methods: ``get``, ``post``, ``put``, ``delete``, ``patch``, ``head``, and ``options``. +The ``proxy_headers`` parameter allows you to send custom headers to the proxy server. This is useful for controlling proxy behavior, such as selecting a specific country or IP address. + +Receiving Proxy Response Headers +--------------------------------- + +Standard requests does not expose proxy response headers from the CONNECT request. When using the ``requests_adapter`` module, proxy response headers are automatically available in the response headers: + +.. code-block:: python + + from python_proxy_headers import requests_adapter + + r = requests_adapter.get( + 'https://api.ipify.org?format=json', + proxies={ + 'http': 'http://PROXYHOST:PORT', + 'https': 'http://PROXYHOST:PORT' + }, + proxy_headers={'X-ProxyMesh-Country': 'US'} + ) + + # Access proxy response headers + proxy_ip = r.headers.get('X-ProxyMesh-IP') + print(f"Request was made through: {proxy_ip}") Available Methods ~~~~~~~~~~~~~~~~~ @@ -54,6 +170,11 @@ All standard requests methods are available through the adapter: from python_proxy_headers import requests_adapter + proxies = { + 'http': 'http://PROXYHOST:PORT', + 'https': 'http://PROXYHOST:PORT' + } + # GET request r = requests_adapter.get('https://api.example.com', proxies=proxies, @@ -73,26 +194,26 @@ All standard requests methods are available through the adapter: # DELETE request r = requests_adapter.delete('https://api.example.com', - proxies=proxies, - proxy_headers={'X-ProxyMesh-Country': 'US'}) - -All parameters that work with standard requests methods also work with the adapter methods, including ``params``, ``data``, ``json``, ``headers``, ``cookies``, ``auth``, and more. - -Receiving Proxy Response Headers ---------------------------------- - -When using the ``requests_adapter`` module, proxy response headers are automatically available in the response headers: - -.. code-block:: python - - from python_proxy_headers import requests_adapter - r = requests_adapter.get('https://api.ipify.org?format=json', - proxies=proxies, - proxy_headers={'X-ProxyMesh-Country': 'US'}) + proxies=proxies, + proxy_headers={'X-ProxyMesh-Country': 'US'}) - # Access proxy response headers - proxy_ip = r.headers.get('X-ProxyMesh-IP') - print(f"Proxy IP: {proxy_ip}") + # PATCH request + r = requests_adapter.patch('https://api.example.com', + json={'status': 'active'}, + proxies=proxies, + proxy_headers={'X-ProxyMesh-Country': 'US'}) + + # HEAD request + r = requests_adapter.head('https://api.example.com', + proxies=proxies, + proxy_headers={'X-ProxyMesh-Country': 'US'}) + + # OPTIONS request + r = requests_adapter.options('https://api.example.com', + proxies=proxies, + proxy_headers={'X-ProxyMesh-Country': 'US'}) + +All parameters that work with standard requests methods also work with the adapter methods, including ``params``, ``data``, ``json``, ``headers``, ``cookies``, ``auth``, ``timeout``, and more. Proxy Headers Overview ---------------------- @@ -115,8 +236,17 @@ For better connection pooling and cookie handling, you can use ``ProxySession``: from python_proxy_headers.requests_adapter import ProxySession with ProxySession(proxy_headers={'X-ProxyMesh-Country': 'US'}) as session: - r = session.get('https://api.example.com', proxies=proxies) + session.proxies = { + 'http': 'http://PROXYHOST:PORT', + 'https': 'http://PROXYHOST:PORT' + } + + # All requests through this session will include proxy headers + r = session.get('https://api.example.com') proxy_ip = r.headers.get('X-ProxyMesh-IP') + + # Make multiple requests with the same session + r2 = session.get('https://api.example.com/other') Or you can manually mount the adapter to a standard requests Session: @@ -129,9 +259,33 @@ Or you can manually mount the adapter to a standard requests Session: session.mount('http://', HTTPProxyHeaderAdapter(proxy_headers={'X-ProxyMesh-Country': 'US'})) session.mount('https://', HTTPProxyHeaderAdapter(proxy_headers={'X-ProxyMesh-Country': 'US'})) - r = session.get('https://api.example.com', proxies=proxies) + session.proxies = { + 'http': 'http://PROXYHOST:PORT', + 'https': 'http://PROXYHOST:PORT' + } + + r = session.get('https://api.example.com') proxy_ip = r.headers.get('X-ProxyMesh-IP') +API Reference +------------- + +Helper Functions +~~~~~~~~~~~~~~~~ + +The module provides convenience functions that mirror the standard ``requests`` API: + +* ``request(method, url, proxy_headers=None, **kwargs)`` - Make a request with the specified method +* ``get(url, proxy_headers=None, **kwargs)`` - Make a GET request +* ``post(url, proxy_headers=None, **kwargs)`` - Make a POST request +* ``put(url, proxy_headers=None, **kwargs)`` - Make a PUT request +* ``patch(url, proxy_headers=None, **kwargs)`` - Make a PATCH request +* ``delete(url, proxy_headers=None, **kwargs)`` - Make a DELETE request +* ``head(url, proxy_headers=None, **kwargs)`` - Make a HEAD request +* ``options(url, proxy_headers=None, **kwargs)`` - Make an OPTIONS request + +Each function creates a temporary ``ProxySession`` with the specified ``proxy_headers`` and makes the request. All standard requests parameters (``params``, ``data``, ``json``, ``headers``, ``cookies``, ``auth``, ``proxies``, ``timeout``, etc.) are supported. + Extension Classes ----------------- @@ -182,20 +336,3 @@ Extends ``requests.Session`` with pre-configured ``HTTPProxyHeaderAdapter`` inst * ``proxy_headers`` (dict, optional): A dictionary of custom headers to send to the proxy server. The ``ProxySession`` automatically mounts ``HTTPProxyHeaderAdapter`` instances for both ``http://`` and ``https://`` URL schemes, so all requests through the session will use proxy header support. - -Helper Functions ----------------- - -The module provides convenience functions that mirror the standard ``requests`` API: - -* ``request(method, url, proxy_headers=None, **kwargs)`` - Make a request with the specified method -* ``get(url, proxy_headers=None, **kwargs)`` - Make a GET request -* ``post(url, proxy_headers=None, **kwargs)`` - Make a POST request -* ``put(url, proxy_headers=None, **kwargs)`` - Make a PUT request -* ``patch(url, proxy_headers=None, **kwargs)`` - Make a PATCH request -* ``delete(url, proxy_headers=None, **kwargs)`` - Make a DELETE request -* ``head(url, proxy_headers=None, **kwargs)`` - Make a HEAD request -* ``options(url, proxy_headers=None, **kwargs)`` - Make an OPTIONS request - -Each function creates a temporary ``ProxySession`` with the specified ``proxy_headers`` and makes the request. All standard requests parameters (``params``, ``data``, ``json``, ``headers``, ``cookies``, ``auth``, ``proxies``, etc.) are supported. - diff --git a/docs/urllib3.rst b/docs/urllib3.rst index f6135d8..c7613ad 100644 --- a/docs/urllib3.rst +++ b/docs/urllib3.rst @@ -1,35 +1,98 @@ urllib3 ======= -The `urllib3 `_ library is a powerful HTTP client for Python. This page describes how to use urllib3 with proxies and how to interact with proxy headers. +The `urllib3 `_ library is a powerful HTTP client for Python. This page describes how to use urllib3 with proxies and how to send and receive custom proxy headers. + +Getting Started +--------------- + +This section shows you how to quickly get up and running with proxy headers in urllib3. + +**Prerequisites:** + +1. Install the package: + + .. code-block:: bash + + pip install python-proxy-headers urllib3 + +2. Import the module: + + .. code-block:: python + + from python_proxy_headers import urllib3_proxy_manager + +**Quick Example - Send and Receive Proxy Headers:** + +.. code-block:: python + + from python_proxy_headers import urllib3_proxy_manager + + # Create a proxy manager that captures proxy response headers + proxy = urllib3_proxy_manager.ProxyHeaderManager( + 'http://PROXYHOST:PORT', + proxy_headers={'X-ProxyMesh-Country': 'US'} + ) + + # Make a request + r = proxy.request('GET', 'https://api.ipify.org?format=json') + + # Access the response data + print(r.data.decode()) # {"ip": "..."} + + # Access proxy response headers (these come from the proxy, not the target server) + print(r.headers.get('X-ProxyMesh-IP')) # The IP address assigned by the proxy + +That's it! The ``ProxyHeaderManager`` handles sending your custom headers to the proxy and makes proxy response headers available in the response. Using Proxies with urllib3 -------------------------- urllib3 provides built-in support for proxies through the ``urllib3.ProxyManager`` class. You can create a proxy manager that routes all requests through a proxy server. -Basic Proxy Usage -~~~~~~~~~~~~~~~~~ +Basic Proxy Usage (Standard urllib3) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -To use a proxy with urllib3, you can use the standard ``urllib3.ProxyManager``: +To use a proxy with standard urllib3: .. code-block:: python import urllib3 + proxy = urllib3.ProxyManager('http://PROXYHOST:PORT') r = proxy.request('GET', 'https://api.ipify.org?format=json') + print(r.data.decode()) + +This routes requests through the specified proxy server. + +Proxy Authentication +~~~~~~~~~~~~~~~~~~~~ + +To use a proxy that requires authentication: + +.. code-block:: python + + import urllib3 + + # Include credentials in the URL + proxy = urllib3.ProxyManager('http://username:password@PROXYHOST:PORT') + r = proxy.request('GET', 'https://api.ipify.org?format=json') -This creates a proxy manager that will route all requests through the specified proxy server. +Or use the ``proxy_headers`` parameter to set the ``Proxy-Authorization`` header manually. Sending Custom Proxy Headers -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +---------------------------- -If you just want to send custom proxy headers, but don't need to receive proxy response headers, then you can use ``urllib3.ProxyManager`` with the ``proxy_headers`` parameter: +If you just want to send custom proxy headers, but don't need to receive proxy response headers, you can use ``urllib3.ProxyManager`` with the ``proxy_headers`` parameter: .. code-block:: python import urllib3 - proxy = urllib3.ProxyManager('http://PROXYHOST:PORT', proxy_headers={'X-ProxyMesh-Country': 'US'}) + + proxy = urllib3.ProxyManager( + 'http://PROXYHOST:PORT', + proxy_headers={'X-ProxyMesh-Country': 'US'} + ) r = proxy.request('GET', 'https://api.ipify.org?format=json') The ``proxy_headers`` parameter allows you to send custom headers to the proxy server. This is useful for controlling proxy behavior, such as selecting a specific country or IP address. @@ -41,30 +104,58 @@ The ``proxy_headers`` parameter allows you to send custom headers to the proxy s Receiving Proxy Response Headers --------------------------------- -To get proxy response headers, use our extension module ``python_proxy_headers.urllib3_proxy_manager``: +Standard urllib3 does not expose proxy response headers from the CONNECT request. To get proxy response headers, use our extension module ``python_proxy_headers.urllib3_proxy_manager``: .. code-block:: python from python_proxy_headers import urllib3_proxy_manager + proxy = urllib3_proxy_manager.ProxyHeaderManager('http://PROXYHOST:PORT') r = proxy.request('GET', 'https://api.ipify.org?format=json') - r.headers['X-ProxyMesh-IP'] + + # Proxy response headers are now available + proxy_ip = r.headers.get('X-ProxyMesh-IP') + print(f"Request was made through: {proxy_ip}") The ``ProxyHeaderManager`` extends the standard ``ProxyManager`` to make proxy response headers available in the response headers. This allows you to access information from the proxy server, such as the IP address that was assigned to your request. Sending and Receiving Proxy Headers ------------------------------------ -You can also pass ``proxy_headers`` into our ``ProxyHeaderManager`` as well. For example, you can pass back the same ``X-ProxyMesh-IP`` header to ensure you get the same IP address on subsequent requests: +You can combine sending custom headers with receiving proxy response headers: .. code-block:: python from python_proxy_headers import urllib3_proxy_manager - proxy = urllib3_proxy_manager.ProxyHeaderManager('http://PROXYHOST:PORT', proxy_headers={'X-ProxyMesh-IP': 'previous-ip-address'}) + + # Send country preference, receive assigned IP + proxy = urllib3_proxy_manager.ProxyHeaderManager( + 'http://PROXYHOST:PORT', + proxy_headers={'X-ProxyMesh-Country': 'US'} + ) r = proxy.request('GET', 'https://api.ipify.org?format=json') - r.headers['X-ProxyMesh-IP'] + + # Check which IP was assigned + assigned_ip = r.headers.get('X-ProxyMesh-IP') + print(f"Assigned IP: {assigned_ip}") + +You can also pass back the same ``X-ProxyMesh-IP`` header to ensure you get the same IP address on subsequent requests: + +.. code-block:: python -This allows you to both send custom headers to the proxy and receive proxy response headers in a single request. + from python_proxy_headers import urllib3_proxy_manager + + # First request - get an IP + proxy = urllib3_proxy_manager.ProxyHeaderManager('http://PROXYHOST:PORT') + r = proxy.request('GET', 'https://api.ipify.org?format=json') + assigned_ip = r.headers.get('X-ProxyMesh-IP') + + # Second request - request the same IP + proxy = urllib3_proxy_manager.ProxyHeaderManager( + 'http://PROXYHOST:PORT', + proxy_headers={'X-ProxyMesh-IP': assigned_ip} + ) + r = proxy.request('GET', 'https://api.ipify.org?format=json') Helper Function ~~~~~~~~~~~~~~~ @@ -75,7 +166,10 @@ The module also provides a convenience function for creating a ``ProxyHeaderMana from python_proxy_headers.urllib3_proxy_manager import proxy_from_url - proxy = proxy_from_url('http://PROXYHOST:PORT', proxy_headers={'X-ProxyMesh-Country': 'US'}) + proxy = proxy_from_url( + 'http://PROXYHOST:PORT', + proxy_headers={'X-ProxyMesh-Country': 'US'} + ) r = proxy.request('GET', 'https://api.ipify.org?format=json') r.headers['X-ProxyMesh-IP'] @@ -92,6 +186,39 @@ Proxy headers are custom HTTP headers that can be used to communicate with proxy The exact headers available depend on your proxy provider. Check your proxy provider's documentation for the specific headers they support. +API Reference +------------- + +ProxyHeaderManager +~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + from python_proxy_headers.urllib3_proxy_manager import ProxyHeaderManager + + proxy = ProxyHeaderManager( + proxy_url, # The proxy URL (e.g., 'http://proxy.example.com:8080') + proxy_headers=None, # Optional dict of headers to send to the proxy + **kwargs # Additional arguments passed to urllib3.ProxyManager + ) + +**Parameters:** + +* ``proxy_url`` (str): The URL of the proxy server +* ``proxy_headers`` (dict, optional): Headers to send to the proxy in the CONNECT request +* ``**kwargs``: Additional arguments passed to the parent ``ProxyManager`` class + +proxy_from_url +~~~~~~~~~~~~~~ + +.. code-block:: python + + from python_proxy_headers.urllib3_proxy_manager import proxy_from_url + + proxy = proxy_from_url(url, **kwargs) + +A convenience function that creates a ``ProxyHeaderManager`` from a URL string. + Internal Classes ---------------- @@ -124,4 +251,3 @@ The ``HTTPSProxyConnectionPool`` class extends ``HTTPSConnectionPool`` to automa 3. Merges these headers into the response headers when ``urlopen()`` is called This ensures that proxy response headers are automatically available in the response object returned to your application. -