Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 55 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Changelog

## 0.9.0

- `no_proxy` support — bypass the proxy for specific hosts, domains, IPs, or CIDRs
- `no_proxy` re-evaluated on every redirect hop
- `--rate-limit` CLI flag for batch mode

## 0.8.0

- `no_proxy` initial implementation (per-request)

## 0.7.0

- Fix duplicate Host header on pooled HTTP/2 connections

## 0.6.0

- Multipart file uploads (`files=` parameter, httpx-style)
- Reject `body` and `files` together with `ValueError`

## 0.5.0

- `blasthttp.mock` submodule for test fixtures (drop-in `BlastHTTP` replacement)
- httpx-style Response API — lazy `body`, `cookies`, `raw_headers`, `hash`
- `peer_ip` on every response and redirect hop

## 0.4.0

- `request_batch_stream()` — async iterator that yields results as they complete

## 0.3.0

- `blasthttp.h2` — permissive HPACK encoder/decoder and HTTP/2 frame builders
- HPACK decoder with dynamic table tracking
- RawConnection inherits client rate limiter

## 0.2.0

- Async Python API via `pyo3-async-runtimes`
- `RawConnection` for byte-level TCP/TLS I/O
- ALPN protocol negotiation on `raw_connect()`
- Proxy support for raw connections (HTTP CONNECT + SOCKS5)

## 0.1.1

- Initial release
- Batch HTTP client with connection pooling
- Custom OpenSSL 3.3.2 with legacy cipher support
- HTTP/2 via ALPN negotiation
- TLS cert extraction (CN, SANs, issuer)
- Response hashing (MD5, SHA256, MurmurHash3)
- Python bindings with retries, rate limiting, and download
- `resolve_ip` (DNS pinning) and `request_target` (request-line override)
- CLI with batch mode and JSON output
79 changes: 78 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ cargo add blasthttp

- **Batch connection reuse** — connections are pooled and reused within a batch, dramatically reducing overhead when scanning many URLs on the same hosts
- **Rust performance** — async I/O, zero-copy where possible, and native concurrency give significant speed improvements over pure Python HTTP clients
- **SSL cert info on every request** — extracts CN, SANs, and issuer during the TLS handshake that's already happening, eliminating the need for a separate sslcert connection
- **SSL cert info on every request** — extracts CN, SANs, emails, issuer, validity dates, and fingerprint during the TLS handshake that's already happening, eliminating the need for a separate sslcert connection
- **All TLS ciphers available by default** — custom-compiled OpenSSL 3.3.2 with legacy provider baked in (RC4, 3DES, export ciphers, SSLv3) so you can connect to anything
- **No cert validation by default** — offensive-first: connects to self-signed, expired, and misconfigured TLS without extra config
- **HTTP/2 support** — automatic via ALPN negotiation, falls back to HTTP/1.1
Expand Down Expand Up @@ -79,7 +79,10 @@ Output is JSON (one object per response), including status, headers, redirect ch
"cert_info": {
"common_name": "example.com",
"sans": ["example.com", "www.example.com"],
"emails": [],
"issuer": "DigiCert Global G2",
"not_before": "2024-01-15T00:00:00Z",
"not_after": "2025-02-15T23:59:59Z",
"fingerprint_sha256": "a0b1c2..."
},
"hash": {
Expand All @@ -103,6 +106,7 @@ Output is JSON (one object per response), including status, headers, redirect ch
| `-d, --data` | Request body | |
| `-l, --list` | File of URLs for batch mode | |
| `-c, --concurrency` | Max concurrent requests (batch) | `50` |
| `--rate-limit` | Requests per second (batch mode) | unlimited |
| `-L, --follow-redirects` | Follow redirects | off |
| `--max-redirects` | Max redirect hops | `10` |
| `-t, --timeout` | Request timeout (seconds) | `10` |
Expand Down Expand Up @@ -179,6 +183,7 @@ asyncio.run(main())
| `redirect_chain` | `list[RedirectHop]` | each hop carries its own `peer_ip` |
| `elapsed_ms` / `elapsed` | `int` / `timedelta` | total request time |
| `request` | `Request` | original `request.url` and `request.method` |
| `debug_log` | `list[str]` | internal debug/timing lines |
| `json()` | callable | `json.loads(self.text)` |
| `raise_for_status()` | callable | raises `HTTPStatusError` on 4xx/5xx |

Expand Down Expand Up @@ -210,6 +215,78 @@ Two ways to issue the same workload — pick whichever matches how you want to c

`request_batch_stream` yields up to 1000 results per chunk, or whatever has accumulated after ~200ms — the timeout flushes partial chunks so the consumer is never starved when results trickle in slowly. Both functions accept the same `(configs, concurrency=50, rate_limit=None)` arguments and respect `set_rate_limit()` identically.

### `request()` parameters

All parameters except `url` are optional:

| Parameter | Type | Notes |
|---|---|---|
| `url` | `str` | Target URL |
| `method` | `str` | HTTP method (default `GET`) |
| `headers` | `list[tuple[str, str]]` | Request headers |
| `body` | `str \| bytes` | Request body (mutually exclusive with `files`) |
| `files` | `dict` | Multipart form-data (httpx-style `files=`; see below) |
| `timeout` | `int` | Request timeout in seconds |
| `follow_redirects` | `bool` | Follow redirects |
| `max_redirects` | `int` | Max redirect hops |
| `verify_certs` | `bool` | Enable TLS cert validation (default `False`) |
| `proxy` | `str` | HTTP/SOCKS proxy URL |
| `no_proxy` | `list[str]` | Hosts that bypass the proxy |
| `cipher_string` | `str` | OpenSSL cipher string |
| `min_tls_version` | `str` | Minimum TLS version (`"1.0"`–`"1.3"`) |
| `max_tls_version` | `str` | Maximum TLS version (`"1.0"`–`"1.3"`) |
| `retries` | `int` | Number of retries on failure |
| `retry_wait_min_ms` | `int` | Minimum backoff between retries (ms) |
| `retry_wait_max_ms` | `int` | Maximum backoff between retries (ms) |
| `max_body_size` | `int` | Max response body bytes to download |
| `request_target` | `str` | Override the HTTP request-line URI |
| `resolve_ip` | `str` | Connect to this IP (DNS pinning / `curl --resolve`) |

`BatchConfig` accepts the same parameters (pass them as constructor kwargs).

### Multipart file uploads

Pass `files` to `request()` or `BatchConfig` for httpx-style multipart form-data. `body` and `files` are mutually exclusive — passing both raises `ValueError`.

```python
r = await client.request(
"https://example.com/upload",
method="POST",
files={"document": ("report.pdf", pdf_bytes, "application/pdf")},
)
```

The `Content-Type: multipart/form-data` boundary header is set automatically.

### Retries

`request()`, `download()`, and `BatchConfig` support automatic retries with exponential backoff:

```python
r = await client.request(
"https://flaky-api.example.com/data",
retries=3,
retry_wait_min_ms=100,
retry_wait_max_ms=5000,
)
```

### `download()` parameters

`download(url, path, ...)` saves a response body to a file. It always follows redirects (up to 10 hops).

| Parameter | Type | Notes |
|---|---|---|
| `url` | `str` | Target URL |
| `path` | `str` | Local file path to write |
| `max_size` | `int` | Max response body bytes |
| `timeout` | `int` | Request timeout in seconds |
| `verify_certs` | `bool` | Enable TLS cert validation |
| `proxy` | `str` | HTTP/SOCKS proxy URL |
| `no_proxy` | `list[str]` | Hosts that bypass the proxy |
| `headers` | `list[tuple[str, str]]` | Request headers |
| `retries` | `int` | Number of retries on failure |

### DNS Pinning & Request-Line Control

Use `resolve_ip` to connect to a specific IP while keeping the original hostname for SNI and Host header — like `curl --resolve`:
Expand Down
Loading