Skip to content

Improve replay throughput and asynchronous event fan-out #33

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jillesvangurp merged 6 commits into from
Apr 2, 2026
Merged

Improve replay throughput and asynchronous event fan-out #33

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
545a9a2
Add replay connector and shared hub scripts
jillesvangurp Apr 2, 2026
7011209
Fix repository docs links
jillesvangurp Apr 2, 2026
acce245
Make derived location processing asynchronous
jillesvangurp Apr 2, 2026
57a9683
Merge branch 'performance-tweaks' into replay-connector
jillesvangurp Apr 2, 2026
4c1da21
Merge branch 'replay-connector' into performance-tweaks
jillesvangurp Apr 2, 2026
da5c58f
Improve replay throughput and async fan-out
jillesvangurp Apr 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 20 additions & 14 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Open RTLS Hub is an OpenAPI-first Go implementation of an OMLOX-ready location hub. It provides OMLOX `/v2` REST resources, OMLOX companion MQTT and WebSocket surfaces, and hub-mediated RPC control-plane support for location-driven integrations.

The hub is vendor-neutral and environment-driven. It runs with Postgres, MQTT, and JWT-based access control, and it follows a contract-first workflow with the normative REST contract in [specifications/openapi/omlox-hub.v0.yaml](/Users/jillesvangurp/git/open-rtls/open-rtls-hub/specifications/openapi/omlox-hub.v0.yaml).
The hub is vendor-neutral and environment-driven. It runs with Postgres, MQTT, and JWT-based access control, and it follows a contract-first workflow with the normative REST contract in [specifications/openapi/omlox-hub.v0.yaml](specifications/openapi/omlox-hub.v0.yaml).

Key capabilities:
- OMLOX `/v2` REST resources and ingestion endpoints
Expand All @@ -14,7 +14,7 @@ Key capabilities:
- Dockerized local runtime for Postgres, Mosquitto, Dex, and the hub
- `just` workflows for bootstrap, code generation, validation, and compose operations
- Unit tests and Testcontainers-based integration coverage
- Connector demonstrators under [`connectors/`](/Users/jillesvangurp/git/open-rtls/open-rtls-hub/connectors)
- Connector demonstrators under [`connectors/`](connectors)

## Omlox

Expand Down Expand Up @@ -54,19 +54,25 @@ Notes:
- `just compose-logs` tails compose services

## Software Docs
- [docs/index.md](/Users/jillesvangurp/git/open-rtls/open-rtls-hub/docs/index.md)
- [docs/architecture.md](/Users/jillesvangurp/git/open-rtls/open-rtls-hub/docs/architecture.md)
- [docs/configuration.md](/Users/jillesvangurp/git/open-rtls/open-rtls-hub/docs/configuration.md)
- [docs/auth.md](/Users/jillesvangurp/git/open-rtls/open-rtls-hub/docs/auth.md)
- [docs/rpc.md](/Users/jillesvangurp/git/open-rtls/open-rtls-hub/docs/rpc.md)
- [docs/index.md](docs/index.md)
- [docs/architecture.md](docs/architecture.md)
- [docs/configuration.md](docs/configuration.md)
- [docs/auth.md](docs/auth.md)
- [docs/rpc.md](docs/rpc.md)

## Connector Demonstrators
- [connectors/README.md](/Users/jillesvangurp/git/open-rtls/open-rtls-hub/connectors/README.md)
- [connectors/local-hub/README.md](/Users/jillesvangurp/git/open-rtls/open-rtls-hub/connectors/local-hub/README.md)
- [connectors/gtfs/README.md](/Users/jillesvangurp/git/open-rtls/open-rtls-hub/connectors/gtfs/README.md)
- [connectors/opensky/README.md](/Users/jillesvangurp/git/open-rtls/open-rtls-hub/connectors/opensky/README.md)
- [connectors/README.md](connectors/README.md)
- [connectors/local-hub/README.md](connectors/local-hub/README.md)
- [connectors/gtfs/README.md](connectors/gtfs/README.md)
- [connectors/opensky/README.md](connectors/opensky/README.md)

## Utility Scripts
- [scripts/log_locations.py](scripts/log_locations.py)
- [scripts/log_fence_events.py](scripts/log_fence_events.py)
- [scripts/log_collision_events.py](scripts/log_collision_events.py)
- [scripts/check_fence_alignment.py](scripts/check_fence_alignment.py)

## Engineering Docs
- [engineering/index.md](/Users/jillesvangurp/git/open-rtls/open-rtls-hub/engineering/index.md)
- [engineering/testing.md](/Users/jillesvangurp/git/open-rtls/open-rtls-hub/engineering/testing.md)
- [engineering/openapi-governance.md](/Users/jillesvangurp/git/open-rtls/open-rtls-hub/engineering/openapi-governance.md)
- [engineering/index.md](engineering/index.md)
- [engineering/testing.md](engineering/testing.md)
- [engineering/openapi-governance.md](engineering/openapi-governance.md)
5 changes: 4 additions & 1 deletion cmd/hub/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,8 @@ func runWithRuntime(ctx context.Context, rt runtimeDeps) error {
LocationTTL: cfg.StateLocationTTL,
ProximityTTL: cfg.StateProximityTTL,
DedupTTL: cfg.StateDedupTTL,
NativeLocationBuffer: cfg.NativeLocationBuffer,
DerivedLocationBuffer: cfg.DerivedLocationBuffer,
MetadataReconcileInterval: cfg.MetadataReconcileInterval,
CollisionsEnabled: cfg.CollisionsEnabled,
CollisionStateTTL: cfg.CollisionStateTTL,
Expand All @@ -190,7 +192,7 @@ func runWithRuntime(ctx context.Context, rt runtimeDeps) error {
if eventBus != nil {
var ch <-chan hub.Event
var unsubscribeMQTTPublisher func()
ch, unsubscribeMQTTPublisher = eventBus.Subscribe(128)
ch, unsubscribeMQTTPublisher = eventBus.Subscribe(cfg.EventBusSubscriberBuffer)
mqttPublisherDone := runEventPublisher(ctx, logger, ch, rt.eventPublisherHandle(mq))
cleanupMQTTPublisher = func() {
unsubscribeMQTTPublisher()
Expand Down Expand Up @@ -271,6 +273,7 @@ func runWithRuntime(ctx context.Context, rt runtimeDeps) error {
cfg.WebSocketReadTimeout,
cfg.WebSocketPingInterval,
cfg.WebSocketOutboundBuffer,
cfg.EventBusSubscriberBuffer,
cfg.CollisionsEnabled,
)
r.Get("/v2/ws/socket", wsHub.Handle)
Expand Down
9 changes: 5 additions & 4 deletions connectors/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,11 @@ Connector projects in this repository should:
- prefer the hub's existing OMLOX interfaces over private integration paths
- keep bootstrap utilities and runtime connectors in the same project when they
depend on the same upstream metadata
- reuse the shared local hub runtime under [`connectors/local-hub`](/Users/jillesvangurp/git/open-rtls/open-rtls-hub/connectors/local-hub) when they need a local demo stack
- reuse the shared local hub runtime under [`connectors/local-hub`](local-hub) when they need a local demo stack

Available connector demos:

- [`connectors/local-hub`](/Users/jillesvangurp/git/open-rtls/open-rtls-hub/connectors/local-hub): reusable local hub, Postgres, Dex, and Mosquitto stack
- [`connectors/gtfs`](/Users/jillesvangurp/git/open-rtls/open-rtls-hub/connectors/gtfs): GTFS-RT vehicle updates and station fence bootstrap
- [`connectors/opensky`](/Users/jillesvangurp/git/open-rtls/open-rtls-hub/connectors/opensky): OpenSky aircraft positions with airport-sector fences
- [`connectors/local-hub`](local-hub): reusable local hub, Postgres, Dex, and Mosquitto stack
- [`connectors/gtfs`](gtfs): GTFS-RT vehicle updates and station fence bootstrap
- [`connectors/opensky`](opensky): OpenSky aircraft positions with airport-sector fences
- [`connectors/replay`](replay): diagnostic NDJSON trace replay with timestamp correction, acceleration, and interpolation
19 changes: 7 additions & 12 deletions connectors/gtfs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,6 @@ The checked-in defaults target the live Grand Dole network:
`location_updates` over WebSocket
- `station_polygons.py`: creates one station `Zone` and one station `Fence` per
station in the hub
- `scripts/log_locations.py`: subscribes to `location_updates` and writes NDJSON
- `scripts/log_fence_events.py`: subscribes to `fence_events` and writes NDJSON
- `scripts/log_collision_events.py`: subscribes to `collision_events` and writes NDJSON
- `scripts/check_geofence_alignment.py`: compares logged locations against the
current station fences
- `hub_client.py`: shared REST and WebSocket helper code
- `gtfs_support.py`: GTFS parsing, GTFS-RT decoding, and station polygon
generation helpers
Expand All @@ -33,7 +28,7 @@ The checked-in defaults target the live Grand Dole network:
## Shared Local Hub

This demo uses the shared local runtime in
[`connectors/local-hub`](/Users/jillesvangurp/git/open-rtls/open-rtls-hub/connectors/local-hub).
[`connectors/local-hub`](../local-hub).

Start it with:

Expand Down Expand Up @@ -103,12 +98,12 @@ uv run --project connectors/gtfs python connectors/gtfs/station_polygons.py --en
uv run --project connectors/gtfs python connectors/gtfs/connector.py --env-file connectors/gtfs/.env.local
```

7. Optional: record live WebSocket topics to NDJSON:
7. Optional: record live WebSocket topics to NDJSON with the shared root scripts:

```bash
uv run --project connectors/gtfs python connectors/gtfs/scripts/log_locations.py --env-file connectors/gtfs/.env.local
uv run --project connectors/gtfs python connectors/gtfs/scripts/log_fence_events.py --env-file connectors/gtfs/.env.local
uv run --project connectors/gtfs python connectors/gtfs/scripts/log_collision_events.py --env-file connectors/gtfs/.env.local
uv run --project scripts python scripts/log_locations.py --env-file connectors/gtfs/.env.local --output connectors/gtfs/logs/location_updates.ndjson
uv run --project scripts python scripts/log_fence_events.py --env-file connectors/gtfs/.env.local --output connectors/gtfs/logs/fence_events.ndjson
uv run --project scripts python scripts/log_collision_events.py --env-file connectors/gtfs/.env.local --output connectors/gtfs/logs/collision_events.ndjson
```

For a single GTFS-RT fetch during local testing:
Expand Down Expand Up @@ -188,10 +183,10 @@ can emit `fence_events` when vehicle trackables enter or leave those station
polygons. Subscribe to `fence_events` or `fence_events:geojson` to observe
arrival and departure behavior.

The `scripts/` directory also includes a simple alignment checker:
The shared root `scripts/` directory also includes a simple alignment checker:

```bash
uv run --project connectors/gtfs python connectors/gtfs/scripts/check_geofence_alignment.py --env-file connectors/gtfs/.env.local
uv run --project scripts python scripts/check_fence_alignment.py --env-file connectors/gtfs/.env.local --locations-log connectors/gtfs/logs/location_updates.ndjson
```

## Limitations
Expand Down
195 changes: 0 additions & 195 deletions connectors/gtfs/scripts/check_geofence_alignment.py
View file
Open in desktop

This file was deleted.

14 changes: 5 additions & 9 deletions connectors/opensky/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,6 @@ Alternate built-in presets:
- `airport_fences.py`: creates airport and apron-sector zones/fences from presets
- `opensky_support.py`: OpenSky polling and preset helpers
- `hub_client.py`: hub REST and WebSocket helpers
- `scripts/log_locations.py`: subscribes to `location_updates` and writes NDJSON
- `scripts/log_fence_events.py`: subscribes to `fence_events` and writes NDJSON
- `scripts/log_collision_events.py`: subscribes to `collision_events` and writes NDJSON
- `scripts/check_fence_alignment.py`: compares logged aircraft locations against current hub fences
- `.env.example`: environment template
- `pyproject.toml`: `uv`-managed Python project metadata
- `uv.lock`: locked Python dependency set
Expand Down Expand Up @@ -92,18 +88,18 @@ uv run --project connectors/opensky python connectors/opensky/airport_fences.py
uv run --project connectors/opensky python connectors/opensky/connector.py --env-file connectors/opensky/.env.local
```

7. Optional: log live WebSocket topics:
7. Optional: log live WebSocket topics with the shared root scripts:

```bash
uv run --project connectors/opensky python connectors/opensky/scripts/log_locations.py --env-file connectors/opensky/.env.local
uv run --project connectors/opensky python connectors/opensky/scripts/log_fence_events.py --env-file connectors/opensky/.env.local
uv run --project connectors/opensky python connectors/opensky/scripts/log_collision_events.py --env-file connectors/opensky/.env.local
uv run --project scripts python scripts/log_locations.py --env-file connectors/opensky/.env.local --output connectors/opensky/logs/location_updates.ndjson
uv run --project scripts python scripts/log_fence_events.py --env-file connectors/opensky/.env.local --output connectors/opensky/logs/fence_events.ndjson
uv run --project scripts python scripts/log_collision_events.py --env-file connectors/opensky/.env.local --output connectors/opensky/logs/collision_events.ndjson
```

8. Check how close captured aircraft positions came to the airport fences:

```bash
uv run --project connectors/opensky python connectors/opensky/scripts/check_fence_alignment.py --env-file connectors/opensky/.env.local
uv run --project scripts python scripts/check_fence_alignment.py --env-file connectors/opensky/.env.local --locations-log connectors/opensky/logs/location_updates.ndjson
```

## Hub Mapping
Expand Down
Loading