feat: peer-to-peer model distribution

[adurham]

Summary

When more than one node in a cluster needs the same model, this lets nodes that already hold the weights serve them to peers over the local network instead of every node fetching independently from HuggingFace. On a Thunderbolt-meshed cluster, a 200 GB model is pulled from the internet exactly once and then fans out at link-local speed.

The PR is two commits, both reviewable independently:

1. feat: peer-to-peer model distribution — the file server, peer discovery, curl-based fetch with hash verification, and the concurrency cap.
2. refactor: rename download cancel→pause and surface DownloadPaused state — splits the existing "cancel" download state into a real DownloadPaused so the worker plan loop doesn't auto-restart paused downloads. Adds POST /download/pause alongside /download/cancel (the latter is kept for backwards compat).

How it works

• New src/exo/download/file_server.py exposes a small aiohttp server on EXO_FILE_SERVER_PORT (default 52416) that streams files from any directory listed in EXO_MODELS_DIRS / EXO_MODELS_READ_ONLY_DIRS. Supports HTTP Range for resumable downloads.
• New src/exo/download/peer_discovery.py walks the global download_status for any peer in DownloadCompleted and picks the best routable IPv4 address from that peer's NodeNetworkInfo, ranking thunderbolt > maybe_ethernet > ethernet > everything else and skipping IPv6, loopback, link-local.
• StartDownload / DownloadModel carry an optional repo_url. When set, the shard downloader skips HF and uses curl (-C - for resume, -f for fail-fast, -D <file> to capture response headers) to pull from the peer. Per-file parallelism is bumped to 16 concurrent files for the peer path since the bottleneck is local NIC throughput, not HF rate limits.

Security stance

Documented in detail in docs/p2p-model-distribution.md. Headlines:

• Path-traversal defense. Resolved file paths are pinned to the specific <model_dir>/<normalized>/ subdirectory — ..-traversal that would escape into a sibling model under the same root is rejected with 404 (an earlier is_relative_to(model_dir)-only check missed this). Tested at the HTTP-protocol level with raw sockets so client-side URL normalization can't hide a regression.
• Range header robustness. Malformed Range headers (bytes=abc-, multi-range, suffix-form, etc.) are silently ignored rather than crashing the handler. Pre-fix this was a one-line DoS — int("abc") raised ValueError and aiohttp returned 500.
• Hash verification. When the source has a <file>.sha256 sidecar (written by the existing HF integrity-check path after it verifies HF's etag), the file server emits X-File-SHA256. The receiver captures the header in the same curl round-trip via -D, hashes the downloaded bytes, refuses to rename .partial → final on mismatch, and the outer retry loop re-fetches. On success the receiver writes its own sidecar so it can in turn pass verified hashes to other peers. Catches transmission corruption and disk-side decay; trust boundary remains "every node in the cluster is trusted."
• Concurrency cap. EXO_FILE_SERVER_MAX_CONCURRENCY (default 64). Excess in-flight serves get 503 Retry-After: 1 and the receiver's existing retry loop handles it (curl re-runs and -C - resumes — no bytes wasted).
• Configurable bind host. EXO_FILE_SERVER_BIND_HOST (default 0.0.0.0). Set to 127.0.0.1 to disable P2P serving on a node, or to a specific interface IP to narrow exposure.
• Error responses do not echo request paths. A 404 says Not found, not File not found: <user-controlled-input>.
• No reverse-proxy assumption. Inter-cluster traffic is the use case; rate-limiting at the reverse-proxy layer doesn't fit the topology.

Tests

61 new tests, all passing alongside the existing 360. Notable:

• test_peer_discovery.py — 16 cases covering interface priority, routability filtering, multi-peer fallthrough.
• test_file_server.py — 32 cases including 4 raw-socket protocol-level tests that exercise path traversal and malformed Range without aiohttp client-side URL normalization.
• test_p2p_download.py — 13 end-to-end cases through a real ephemeral file_server, including hash mismatch handling, sidecar absence (backward compat), and _parse_x_file_sha256 redirect-chain handling.

Test plan

• CI green (basedpyright, ruff, pytest)
• Reviewer sanity-check: docs/p2p-model-distribution.md accurately describes the security stance
• Reviewer sanity-check: hash-verification flow makes sense given the existing download_file_with_retry retry semantics
• Optional manual test: 2-node cluster, place an instance of a model neither node has, confirm node A pulls from HF and node B pulls from node A

Related

Fork-side issue tracking from when this was scoped: docs/upstream-prs.md (private) — this is the "P2P model distribution" entry.