Skip to content

Add replay connector, shared capture scripts, and replay throughput improvements#32

Merged
jillesvangurp merged 7 commits intomainfrom
replay-connector
Apr 2, 2026
Merged

Add replay connector, shared capture scripts, and replay throughput improvements#32
jillesvangurp merged 7 commits intomainfrom
replay-connector

Conversation

@jillesvangurp
Copy link
Copy Markdown
Contributor

@jillesvangurp jillesvangurp commented Apr 2, 2026

Summary

This PR adds the replay connector and shared hub-feed logging tools, and it now also carries the hub-side performance work needed to replay dense traces reliably.

What Changed

  • Adds connectors/replay as a diagnostic connector that replays logged location_updates NDJSON back into the hub.
  • Supports timestamp correction to replay from "now", acceleration via --acceleration-factor, and synthetic interpolation via --interpolation-rate-hz.
  • Adds replay publish batching via --batch-window-ms and --max-batch-size to reduce WebSocket overhead.
  • Moves the reusable WebSocket logging and feed-inspection helpers into the shared root scripts/ project and removes connector-local duplicates.
  • Updates connector and repository docs to use the shared scripts and describe replay as a diagnostic trace-replay tool.
  • Improves hub ingest and fan-out performance with:
    • lazy transforms
    • async native publication and decision-processing stages
    • a future-ready insertion point for filtered track processing before fence/collision evaluation
    • batched WebSocket fan-out
    • drop-on-overload behavior for non-critical queues instead of synchronous backpressure
    • runtime drop counters and larger configurable buffers
    • collision prefiltering against active motions

Why

The repository needed a connector-agnostic way to capture and replay traces for diagnostics and regression testing, and dense replay traces exposed that the hub was doing too much synchronous work on the ingest path.

Replay Verification

Using the same captured OpenSky trace replayed with --interpolation-rate-hz 10.0:

  • scheduled emissions: 13,192
  • earlier replay branch baseline: 256 observed location items before the perf work
  • first async hub pass: 8,639 observed location items
  • current branch after batching and fan-out changes: 13,192 observed location items
  • fence events matched the live capture at 126

Validation

  • uv lock --project connectors/replay
  • uv lock --project scripts
  • python3 -m py_compile scripts/*.py connectors/replay/*.py
  • just bootstrap
  • just generate
  • just check
  • live OpenSky capture and replay validation against the local demo stack

@jillesvangurp jillesvangurp changed the title [codex] Add replay connector and shared hub scripts Add replay connector, shared capture scripts, and replay throughput improvements Apr 2, 2026
@jillesvangurp jillesvangurp marked this pull request as ready for review April 2, 2026 10:03
@jillesvangurp jillesvangurp merged commit 661c3bc into main Apr 2, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant