hft-hunter

Find the AWS EC2 instances that reach a crypto exchange with the lowest possible latency.

hft-hunter launches fleets of candidate EC2 instances across many instance types and racks (cluster placement groups), measures protocol-level latency to the exchange (TCP, TLS, HTTP, WebSocket, ICMP, UDP), ranks the candidates by their p99 latency, and automatically keeps the winners while terminating the losers. The result is a shortlist of the fastest instance types / placement groups on which you should run your production trading system.

1. How the pieces fit together

There are three programs, and it matters which machine each one runs on:

Program	Runs on	Who starts it	What it does
`hunter` (CLI)	Your operator machine — your laptop, or a small always-on control box	You, by hand	The control plane. Launches/terminates EC2 fleets, runs the collector, ranks results, keeps winners.
`hunter-agent`	Each AWS EC2 probe instance (the candidate servers you are evaluating)	Automatically, at instance boot	Repeatedly probes the exchange endpoints from that instance and uploads the measurements to the collector. You never install it by hand — the fleet launch does it for you.
collector	A host reachable from the EC2 probe instances, inside the same VPC	You, by hand (or embedded in `hunter hunt`)	Receives probe measurements over HTTP and stores them as JSONL on disk. It is just `hunter serve-collector`.

And two categories of servers:

Target servers = the exchange endpoints, e.g. api.binance.com, stream.binance.com, ws.okx.com. You deploy nothing here. These are external and are only measured.
Scanned / candidate servers = the AWS EC2 instances that hft-hunter launches for you. The hunter-agent runs on these, measuring how fast this specific instance in this specific rack can talk to the target servers.

Data flow:

  your operator machine                AWS VPC
  ┌───────────────────┐        ┌──────────────────────────────┐
  │ hunter (CLI)       │  EC2   │  probe instance   ─┐          │        exchange
  │  hunter fleet …    │───────▶│   hunter-agent    │ measures  │──────▶ api.binance.com
  │  hunter hunt …     │  API   │  probe instance   │ latency   │        ws.okx.com …
  │  hunter report …   │        │   hunter-agent   ─┘           │        (target servers)
  └───────────────────┘        │        │ uploads results       │
            ▲                   │        ▼                       │
            │  read JSONL       │  collector (hunter             │
            └───────────────────┤  serve-collector) ◀───────────┘
                                └──────────────────────────────┘

2. What you need before you start

An AWS account with EC2 permissions, and credentials configured on your operator machine (aws configure, an SSO profile, or environment variables — anything the AWS SDK understands).
Terraform ≥ 1.5 (only to create the one-time network baseline).
A Rust toolchain on your operator machine, only so you can build the two binaries once.
Somewhere to publish the built hunter-agent binary so EC2 instances can download it at boot (an S3 bucket or any HTTPS URL works).

Building the code is a one-time setup step, not the daily workflow. If you only want to sanity check latency from your current machine, jump to Appendix A: local-only quick probe.

3. Step-by-step: run a full AWS hunt

This is the main workflow. Follow it top to bottom the first time.

Step 1 — Build the two binaries (one time)

On your operator machine:

# The control-plane CLI you will run locally (native build is fine).
cargo build --release -p hunter

# The probe agent that will run ON the Linux EC2 instances.
# Build it for Linux so EC2 can execute it. If your operator machine is
# already Linux x86_64, a plain release build is enough:
cargo build --release -p hunter-agent
# On macOS / a different arch, cross-compile for the EC2 architecture, e.g.:
#   rustup target add x86_64-unknown-linux-gnu
#   cargo build --release -p hunter-agent --target x86_64-unknown-linux-gnu

The hunter binary lands in target/release/hunter; the agent in target/release/hunter-agent (or under target/<triple>/release/).

Step 2 — Publish the agent binary

Upload the hunter-agent binary to a URL the EC2 instances can reach at boot. For example:

aws s3 cp target/release/hunter-agent s3://your-bucket/hunter-agent
# then note its download URL, e.g. a presigned/HTTPS URL:
#   https://your-bucket.s3.amazonaws.com/hunter-agent

Remember this URL — it becomes agent_url in the config. At launch, each EC2 instance downloads the agent from here and starts it as a systemd service.

Step 3 — Create the AWS network baseline (one time)

The static VPC / subnet / security group / IAM profile is provisioned once with Terraform. The per-instance lifecycle is handled later by the hunter CLI, not Terraform.

cd deploy/terraform
terraform init
terraform apply     # review the plan, then confirm

# Copy these outputs — you paste them into the config in the next step:
terraform output subnet_id
terraform output security_group_id

You also need an AMI id (Amazon Linux 2023) for your region, and optionally an EC2 key pair name if you want SSH access to the instances.

Step 4 — Decide where the collector runs, and start it

The collector must be reachable from the probe instances, over the port in the security group (9900 by default, open inside the VPC). Practically that means running it on a host inside the VPC (a tiny always-on EC2 box, or your operator machine if it is reachable from the subnet).

On that collector host:

hunter serve-collector --listen 0.0.0.0:9900 --results-dir results

Note the address as reachable from EC2 (a private VPC IP, e.g. 10.77.1.10:9900) — that becomes collector_addr in the config.

Shortcut: hunter hunt (Step 6) starts an embedded collector for you. If you use hunt, you can skip running a separate collector — just make sure collector_addr points at wherever the hunter hunt process is reachable from EC2.

Step 5 — Fill in `config/hunter.toml`

Edit the [fleet] section with the values gathered above:

[fleet]
region = "ap-northeast-1"
ami_id = "ami-xxxxxxxxxxxxxxxxx"           # AL2023 AMI in your region
subnet_id = "subnet-xxxxxxxxxxxxxxxxx"     # terraform output subnet_id
security_group_id = "sg-xxxxxxxxxxxxxxxxx" # terraform output security_group_id
key_name = "hft-hunter"                    # optional; your EC2 key pair
instance_types = ["c7i.xlarge", "c6in.xlarge", "c7gn.xlarge"]
count_per_type = 5                         # instances launched per type
max_instances = 50                         # hard cost guardrail
agent_url = "https://your-bucket.s3.amazonaws.com/hunter-agent"  # Step 2
collector_addr = "10.77.1.10:9900"         # collector as reachable from EC2
dry_run = false                            # set true to validate without launching

The [[targets]] blocks define which exchange endpoints to measure, and the [hunt] block controls the automated keep-the-winners loop (see Section 4).

Step 6 — Run the hunt

The one-command, end-to-end path:

hunter hunt --config config/hunter.toml

This will:

Start an embedded collector.
Launch the first wave of EC2 instances (one cluster placement group per instance type). Each instance boots, tunes its OS/NIC, downloads and runs hunter-agent.
Wait wait_secs while agents probe the exchange and upload results.
Rank the instances by p99 latency and print a Markdown table.
Keep the top keep_top winners, terminate the rest, then (for further rounds) add more instances into the winning placement groups and re-test — hunting for the fastest rack.

At the end you have the winning instance ids / placement groups: that is where you should run your production trading system.

Step 7 — Inspect and clean up

# Re-render a report from collected results at any time:
hunter report --input results --format md          # Markdown table
hunter report --input results --format csv --top 3  # top 3 as CSV
hunter report --input results --format json         # JSON

# If you ran waves manually (see below) and want to stop paying for them:
hunter fleet terminate --config config/hunter.toml --wave w1

4. What `hunter hunt` actually does

The behaviour is driven by the [hunt] block in config/hunter.toml:

[hunt]
wave = "w1"           # label used for tags and placement group names
wait_secs = 300       # how long to wait for agents to report each round
keep_top = 3          # how many winners to keep each round
rounds = 2            # 1 = single round; >1 refines the winning racks
expand_per_round = 4  # instances added to each winning placement group per round

Each round: wait → rank by p99 → keep the top keep_top, terminate the losers → expand the winning placement groups with expand_per_round fresh instances → repeat. This progressively concentrates your candidates in the fastest racks.

5. Manual fleet control (instead of `hunt`)

If you prefer to drive each stage yourself:

# 1. Start the collector on a VPC-reachable host (see Step 4).
hunter serve-collector --listen 0.0.0.0:9900 --results-dir results

# 2. Launch a wave (optionally override instance types / count from the CLI).
hunter fleet launch --config config/hunter.toml --wave w1
hunter fleet launch --config config/hunter.toml --wave w1 --types c7i.xlarge,c6in.xlarge --count 4

# 3. Watch the instances come up.
hunter fleet status --config config/hunter.toml --wave w1

# 4. Once enough results have arrived, rank them.
hunter report --input results --format md --top 5

# 5. Terminate the wave when you are done.
hunter fleet terminate --config config/hunter.toml --wave w1

6. Probe kinds

Each target lists the protocols to measure in its protocols = [...] field:

Kind	Meaning
`tcp_connect`	Time to complete the TCP three-way handshake
`tls_handshake`	Time to complete the TLS handshake
`http`	HTTP time-to-first-byte (uses the target's `http_path`)
`ws_ping`	WebSocket ping/pong round trip (uses `ws_path`)
`ws_order_create`	WebSocket `CREATE_ORDER` -> `BOOKED` round trip (needs a mock/order endpoint)
`ws_order_cancel`	WebSocket `CANCEL_ORDER` -> `DONE` round trip (needs a mock/order endpoint)
`icmp`	ICMP echo (ping) round trip
`udp_rtt`	UDP echo round trip

Ready-made target sets live in config/targets/binance.toml and config/targets/okx.toml.

Appendix A: local-only quick probe

You do not need AWS to try hft-hunter. To measure latency straight from the machine you are on right now:

# Discover the IPs behind an exchange hostname and their TCP connect times.
hunter discover --host api.binance.com

# Run every probe in the config from this machine and print a ranking.
hunter probe --config config/hunter.toml

To exercise the WebSocket order round-trip probe locally, start the bundled mock exchange first:

mock-exchange --listen 127.0.0.1:9001 --udp-listen 127.0.0.1:9002
# then point a target at 127.0.0.1:9001 with
# protocols = ["ws_ping", "ws_order_create", "ws_order_cancel"]

mock-exchange is only a local testing/BDD helper — it plays no role in a real AWS hunt.

Appendix B: configuration files

File	Purpose
`config/hunter.toml`	Main config: probe targets, `[fleet]` AWS settings, `[collector]`, `[hunt]` workflow
`config/targets/binance.toml`	Standalone Binance target set (REST + WS)
`config/targets/okx.toml`	Standalone OKX target set (REST + WS public/private)
`deploy/terraform/`	One-time AWS baseline (VPC, subnet, security group, IAM)

Appendix C: building, testing, and contributing

For development work on hft-hunter itself:

just setup       # install dev tools (cargo-shear, cargo-sort, typos-cli)
just build       # cargo build --workspace
just lint        # typos + markdown + sort check + fmt check + clippy + shear
just test        # unit tests + proptest
just bdd         # cucumber-rs BDD scenarios (stats, probe, report)
just test-all    # test + bdd
just docs        # cargo doc --no-deps --open

Workspace layout

bin/
  hunter          Control-plane CLI (discover, probe, serve-collector, fleet, hunt, report)
  hunter-agent    Probe daemon that runs on EC2 instances
  mock-exchange   Local mock WS + UDP exchange for testing/BDD
crates/
  common          Shared domain types (ProbeKind, Target, ProbeRecord, ...)
  probe           TCP/TLS/HTTP/WS/ICMP/UDP probing engine
  stats           HDR-histogram nanosecond latency statistics
  collect         Axum collector service + JSONL storage + EC2 IMDS metadata
  report          p99 ranking and Markdown/CSV/JSON rendering
  cloud           AWS EC2 fleet orchestration (launch, keep/kill, placement groups)
  mockex          Mock exchange server library

License

Apache-2.0

Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
.cargo		.cargo
.github/workflows		.github/workflows
bin		bin
config		config
crates		crates
deploy/terraform		deploy/terraform
features		features
scripts		scripts
.agnix.toml		.agnix.toml
.gitignore		.gitignore
.rumdl.toml		.rumdl.toml
.rustfmt.toml		.rustfmt.toml
.typos.toml		.typos.toml
AGENTS.md		AGENTS.md
CLAUDE.md		CLAUDE.md
Cargo.toml		Cargo.toml
Justfile		Justfile
README.md		README.md
cliff.toml		cliff.toml

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Repository files navigation

hft-hunter

1. How the pieces fit together

2. What you need before you start

3. Step-by-step: run a full AWS hunt

Step 1 — Build the two binaries (one time)

Step 2 — Publish the agent binary

Step 3 — Create the AWS network baseline (one time)

Step 4 — Decide where the collector runs, and start it

Step 5 — Fill in `config/hunter.toml`

Step 6 — Run the hunt

Step 7 — Inspect and clean up

4. What `hunter hunt` actually does

5. Manual fleet control (instead of `hunt`)

6. Probe kinds

Appendix A: local-only quick probe

Appendix B: configuration files

Appendix C: building, testing, and contributing

Workspace layout

License

About

Releases

Packages

Contributors

Languages

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

hft-hunter

1. How the pieces fit together

2. What you need before you start

3. Step-by-step: run a full AWS hunt

Step 1 — Build the two binaries (one time)

Step 2 — Publish the agent binary

Step 3 — Create the AWS network baseline (one time)

Step 4 — Decide where the collector runs, and start it

Step 5 — Fill in config/hunter.toml

Step 6 — Run the hunt

Step 7 — Inspect and clean up

4. What hunter hunt actually does

5. Manual fleet control (instead of hunt)

6. Probe kinds

Appendix A: local-only quick probe

Appendix B: configuration files

Appendix C: building, testing, and contributing

Workspace layout

License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Step 5 — Fill in `config/hunter.toml`

4. What `hunter hunt` actually does

5. Manual fleet control (instead of `hunt`)