Hft datageneration vllm sched

.gitignore

@@ -38,3 +38,5 @@ htmlcov/
 .claude
 venv
 takehome
+.venv311/
+.python-version

benchmarks/hft/HFT_HEADROOM_TOOLS.md

@@ -0,0 +1,81 @@
+# HFT Task A (headroom report) + Task B (apply + prove) — tools and plan
+
+Two scripts + one library that deliver Task A now (no pod) and set up Task B
+(needs pod). Drop the three files in benchmarks/hft/.
+
+## Files
+- hft_library.yaml         — 3 candidate interventions in the real
+                             InterventionSpec schema (review: null pending Adit)
+- headroom_kernel_rank.py  — Task A: ranks them by the real predict_delta()
+- (Task B uses gitm/optimizer/apply.py — already in the repo)
+
+## Task A — run the headroom report (NO POD NEEDED)
+
+    cd benchmarks/hft
+    python3 headroom_kernel_rank.py
+
+Output: ranked list of 3 candidate interventions, each with coverage %,
+predicted fractional delta, and predicted Δ events/sec. Current ranking:
+
+    1  hft_parquet_h2d_overlap        89.9% cover   ~22.5%   +4.65M eps
+    2  hft_multistream_symbol_shards  42.1% cover    ~6.3%   +1.31M eps
+    3  hft_groupby_scan_fuse          10.1% cover    ~0.8%   +0.17M eps
+
+This satisfies Task A's "done when": >=3 candidates, each with a predicted
+Δ events/sec tied to a residual (the covered kernels).
+
+## What's measured vs estimated (say this to Adit)
+
+MEASURED (real):
+  - kernel times / coverage % — from the seed-42 nsys profile
+  - the predict_delta() function — the same one the optimizer loop uses
+
+ESTIMATED (literature, cited per spec, NOT measured on our workload):
+  - expected_delta_mean (0.25 / 0.15 / 0.08) — from cuDF / NVIDIA docs
+  - therefore the predicted-Δ magnitudes are estimates
+
+So: the RANKING is defensible (driven by real coverage). The NUMBERS are
+estimates until Task B measures them. The "#1 gets us to 25.4M / over target"
+line depends on the 0.25 estimate holding; at the low end (0.10) it's ~22.5M,
+still short. Present it as "predicted to potentially cross target, pending
+measurement," not a promise.
+
+## Task B — apply + prove the #1 intervention (NEEDS POD)
+
+Goal: apply hft_parquet_h2d_overlap to the harness, prove it speeds things up
+WITHOUT changing the derived metrics (VWAP/microprice), via a checksum gate.
+
+The pieces that already exist in the repo:
+  - gitm/optimizer/apply.py    — apply_intervention(spec, applicator,
+                                 min_keep_delta): snapshot -> apply -> measure
+                                 -> keep|rollback. Use this as the gate.
+  - The Applicator seam         — you implement snapshot/apply/measure/restore
+                                 for the harness (e.g. toggle prefetch depth).
+
+What you build for Task B:
+  1. A checksum: run harness, hash the derived-metric output (e.g. sha256 of
+     the sorted VWAP/microprice arrays). This is the correctness gate.
+  2. An HarnessApplicator implementing the Applicator protocol:
+       - snapshot(): record current reader settings + baseline checksum
+       - apply(spec): set reader_prefetch_depth=2 (overlap decode/H2D)
+       - measure(spec): re-run harness, return (eps_after - eps_before)/eps_before
+       - restore(): put settings back
+  3. Wrap it: apply_intervention(spec, HarnessApplicator(), min_keep_delta=0.0)
+     BUT also assert checksum_after == checksum_before, else force rollback.
+  4. Provenance report: claim (headroom #1) -> evidence (nsys coverage) ->
+     intervention (prefetch_depth=2) -> delta (measured eps before/after) +
+     checksum match. If checksum mismatches, report NO speedup (correctness
+     beats speed).
+
+Task B's "done when": provenance report shows a measured speedup with
+checksum-identical output, and reports no win if the checksum mismatches.
+
+HONEST SCOPE: Task B is real engineering — implementing the overlap in the
+harness (cuDF chunked/prefetched Parquet read), wiring the Applicator, and
+measuring on the pod. It is NOT a copy-paste job. Do it once pod access is
+back, and expect to iterate on the actual cuDF read path.
+
+## Validation done locally
+headroom_kernel_rank.py was run against the real seed-42 kernel sums and
+produces the ranked table above using the repo's real predict_delta(). The
+ranking logic is verified; the magnitudes carry the estimate caveat above.

benchmarks/hft/HFT_TASK_TOOLS.md

@@ -0,0 +1,66 @@
+# HFT Task 1 + Task 3 — tools and run order
+
+Three scripts that finish the predicted-graph integration (Task 1) and the
+Granger causality analysis (Task 3). All three need the pod (the data and
+nsys live there). Run them in this order.
+
+## What's measured vs estimated (read this first)
+
+The predicted-graph residuals depend on byte counts. Right now hft_graph.py
+uses ESTIMATES:
+  - compression_ratio = 3.0  (a guess; zstd-1 on int columns is ~2.5-4x)
+  - per-stage bytes_moved (assumed multiples of the uncompressed size)
+
+The observed kernel times ARE real (from nsys). So the residual MAGNITUDES
+(the "592x" numbers) are only as good as the byte estimates. Step 1 below
+replaces the compression-ratio guess with a measured value. The Granger
+RANKING (step 3) is more robust to the byte estimates than the magnitudes,
+because it depends on relative timing across shards, not absolute residual
+size — but at 20 samples it's still suggestive, not conclusive.
+
+## Step 1 — measure the real compression ratio (no nsys, just file reads)
+
+    python3 measure_compression.py /workspace/hft_numba_seed42
+
+Copy the printed "compression_ratio (parquet meta)" value and replace
+compression_ratio=3.0 in HFTDatasetSpec (benchmarks or gitm/planner/hft_graph.py).
+Re-run `python3 -m gitm.planner.hft_residual_demo` to get corrected magnitudes.
+
+## Step 2 — profile 20 shards separately (needs nsys + GPU)
+
+    bash profile_per_shard.sh /workspace/hft_numba_seed42 /workspace/shard_profiles 20
+
+Profiles 20 single-shard harness runs (5M events each) -> 20 nsys profiles ->
+20 samples per kernel op. This is the part that takes a while (20 nsys runs).
+
+WHY 20 AND NOT 5: grangercausalitytests at maxlag=2 needs >= 8 observations
+or statsmodels raises "Maximum allowable lag is 0". Even at 8-20 the p-values
+are noisy. 20 is a practical floor; more shards = more reliable.
+
+## Step 3 — run Granger causality on the residual series
+
+    python3 parse_shard_profiles.py /workspace/shard_profiles
+
+Parses the 20 CSV summaries, builds a per-op residual time series against the
+predicted graph, and runs gitm.optimizer.attribution.attribute(). Output is a
+ranked table of (cause -> effect, p_value). A low p for
+zstd_decompress -> parquet_decode would confirm decompression is upstream of
+the decode slowdown, not just correlated with it.
+
+## Validated locally
+
+The parse + Granger chain was tested with 20 synthetic shards carrying a known
+zstd->parquet lagged dependence. attribute() correctly ranked
+zstd_decompress -> parquet_decode at p=0.0000 and the unrelated merge_sort
+pairs as non-significant (p>0.18). So the code path works; on real data the
+result depends on what the actual per-shard timings show.
+
+## Honest limitations to mention to Adit
+
+1. compression_ratio is estimated until step 1 is run on real files.
+2. Granger at n=20 is suggestive, not conclusive — more shards would help.
+3. roofline still has no decompression-bound category; the residuals are vs
+   the memory-bandwidth floor, which these ops were never going to hit. The
+   "right" fix is a new prediction category calibrated against nvCOMP zstd
+   throughput numbers — that's a proposal for Adit, not something to merge
+   into the shared roofline.py unilaterally.

benchmarks/hft/datasets.md

@@ -44,3 +44,32 @@ make manifest      # -> manifest.yaml (sha256 + byte count per Parquet file)
 make verify        # re-hash $GITM_SCRATCH/staging/hft and confirm byte-identical
 ```
 Manifest sha256: <!-- TODO: paste after first freeze -->
+
+## How It Was Built
+
+### Generator
+Written from scratch in C++ (`benchmarks/hft/generator/main.cpp`).
+Built using CMake against pyarrow's bundled Arrow/Parquet C++ libraries.
+
+### Arrival Model
+Hawkes process with parameters:
+- mu=100 (baseline rate)
+- alpha=0.6 (excitation per event)
+- beta=0.8 (decay rate)
+Inter-arrival times sampled as dt = -log(uniform) / lambda (inverse transform sampling).
+
+### Price Model
+Integer random walk in tick space. Mid price moves +1 or -1 on each trade event (type=2).
+
+### Output
+Written in 5M event chunks to avoid OOM on large datasets.
+Same seed always produces identical bytes (mt19937_64 RNG).
+
+### Manifest
+Generated by `benchmarks/hft/gen_manifest.py`.
+Contains sha256 and byte count per file for reproducibility verification.
+
+### Profiling
+Benchmarked using `benchmarks/hft/harness.py` (cuDF/CuPy GPU pipeline).
+Profiled with NVIDIA Nsight Systems (nsys).
+Results documented in `benchmarks/hft/results.md`.

benchmarks/hft/gen_manifest.py

@@ -9,7 +9,7 @@
 files = []
 
 for seed in seeds:
-    p = data_root / f"datasets/hft/hft_1b_seed{seed}/part0.parquet"
+    p = data_root / f"datasets/hft/hft_1b_seed{seed}/part-00000.parquet"
     sha256 = hashlib.sha256(p.read_bytes()).hexdigest()
     size   = p.stat().st_size
     files.append({

benchmarks/hft/generator_comparison.md

@@ -0,0 +1,112 @@
+# HFT Generator Comparison Report
+
+## Overview
+I tested and built four different approaches to generate 1B HFT events, progressively improving speed while maintaining data realism. The final version is 7.4x faster than C++ while keeping exact Hawkes timestamps.
+
+## What Makes the Data Realistic
+
+The realism comes from three things, not from Numba or GPU:
+
+1. Hawkes process for timestamps - in real markets, one big trade triggers more trades. Events cluster in bursts. Hawkes models this: every event jumps the arrival rate up by alpha (0.6), then it decays back to baseline at rate beta (0.8). Simple uniform random gaps would be unrealistic.
+
+2. Realistic event mix - real order books have ~55% new orders, ~35% cancellations, ~10% trades. We use exactly this distribution instead of uniform random.
+
+3. Price random walk - price moves up or down by 1 tick every time a trade happens, simulating real market microstructure.
+
+Numba and GPU do NOT change the realism. They just make the same math run faster.
+
+## Why Numba
+
+The Hawkes process has a sequential dependency - each timestamp depends on the previous lambda value:
+
+    lambda_i = MU + (lambda_{i-1} - MU) * exp(-BETA * dt_i) + ALPHA
+
+This cannot be parallelized directly. In plain Python this loop runs at 0.7M/sec. Numba JIT compiles the exact same loop to native CPU machine code using LLVM - running at 34.8M/sec. 53x faster, same exact output.
+
+Attempted fully vectorized GPU Hawkes using parallel prefix scan but thinning approximation produced non-monotonic timestamps. Numba is the correct solution - exact, fast, stable.
+
+## Generators Tested
+
+### 1. C++ Generator (benchmarks/hft/generator/main.cpp)
+Sequential loop per event. Hawkes process for timestamps. Writes to Parquet in 5M chunks to avoid OOM. Single threaded.
+
+- Time for 1B events: ~10 minutes
+- Events/sec: ~1.67M/sec
+- Hawkes: exact
+- Event mix: uniform random
+
+### 2. GPU Single Thread (CuPy)
+Generates all 6 fields for 5M events at once on GPU. Transfers to CPU, writes to Parquet. Sequential - waits for write before next chunk. No Hawkes.
+
+- Time for 1B events: ~4m 25s
+- Events/sec: ~3.8M/sec
+- Hawkes: simple uniform gaps
+- Event mix: uniform random
+
+### 3. GPU Parallel simple
+GPU generates chunk N while 3 CPU threads write simultaneously using ThreadPoolExecutor. GPU never waits for disk. Still no Hawkes.
+
+- Time for 1B events: 1m 18s
+- Events/sec: 12.9M/sec
+- Hawkes: simple uniform gaps
+- Event mix: uniform random
+
+### 4. Numba + GPU Parallel (final version)
+Best of all approaches combined:
+- Numba JIT compiles Hawkes loop to native CPU - 53x faster than plain Python, exact output
+- CuPy generates symbol, side, price, size, type on GPU simultaneously
+- Realistic event mix: 55% add, 35% cancel, 10% trade
+- 3 parallel write threads - disk never idle
+
+- Time for 1B events: ~1m 22s
+- Events/sec: ~12.4M/sec
+- Hawkes: exact
+- Event mix: realistic 55/35/10
+
+## Results
+
+| Generator | Time for 1B | Events/sec | Speedup vs C++ | Exact Hawkes | Realistic Mix |
+|---|---|---|---|---|---|
+| Adit Python (numpy) | ~30 min | 0.56M/sec | 0.3x | No | No |
+| C++ CPU | ~10 min | 1.67M/sec | 1x | Yes | No |
+| GPU single thread | ~4m 25s | 3.8M/sec | 2.3x | No | No |
+| GPU parallel simple | 1m 18s | 12.9M/sec | 7.7x | No | No |
+| Numba + GPU parallel | 1m 22s | 12.4M/sec | 7.4x | Yes | Yes |
+
+## Why GPU Parallel is Fast
+
+Three things happen simultaneously:
+- GPU generates fields for chunk N using thousands of GPU cores
+- Write thread 1 writes chunk N-2 to disk
+- Write thread 2 writes chunk N-1 to disk
+
+Nothing waits. Like a pit crew where refueling, tire change, and wing adjustment all happen at once.
+
+## Bottleneck at Each Stage
+
+| Stage | C++ | GPU single | GPU parallel | Numba + GPU |
+|---|---|---|---|---|
+| Timestamp generation | slow CPU loop | uniform only | uniform only | Numba JIT fast |
+| Field generation | slow CPU loop | GPU fast | GPU fast | GPU fast |
+| Parquet write | sequential | sequential | 3x parallel | 3x parallel |
+| Bottleneck | generation | write | disk bandwidth | disk bandwidth |
+
+## RNG Speed Test (5M numbers)
+- numpy CPU: 0.204s = 24M/sec
+- CuPy GPU: 0.000s = 10B+/sec
+- Speedup: 423x
+
+## Three Seeds Validated
+
+| Seed | Rows | Files | Events/sec |
+|---|---|---|---|
+| 42 | 1,000,000,000 | 200 | 12.4M/sec |
+| 43 | 1,000,000,000 | 200 | 12.4M/sec |
+| 44 | 1,000,000,000 | 200 | 12.4M/sec |
+
+Seed variance: 0% - identical throughput across all seeds.
+
+## Conclusion
+Final generator (Numba + GPU + parallel writes) is 7.4x faster than C++ with exact Hawkes timestamps and realistic event mix. The key insight is that realism and speed are not tradeoffs - Numba makes exact Hawkes fast enough to match GPU field generation speed.
+
+Next step: explore Arrow IPC format instead of Parquet to reduce write overhead further.