Skip to content

Add local MQTT broker and IoT hardware ingress pipeline#75

Merged
d3mocide merged 3 commits into
mainfrom
claude/audit-tinygs-integration-fSkyf
May 22, 2026
Merged

Add local MQTT broker and IoT hardware ingress pipeline#75
d3mocide merged 3 commits into
mainfrom
claude/audit-tinygs-integration-fSkyf

Conversation

@d3mocide
Copy link
Copy Markdown
Owner

Adds Mosquitto as a core container (always-on, no profile flag) and a new
mqtt_subscriber poller that dispatches incoming messages to per-normalizer
handlers. Replaces the unstable TinyGS REST-only integration with a proper
local-node MQTT path while keeping the REST poller as an opt-in fallback.

Infrastructure:

  • docker-compose.yml: mosquitto service (eclipse-mosquitto:2), mosquitto_data
    volume, LAN port 1883 (MQTT_PORT), poller now depends on mosquitto health
  • infra/mosquitto/mosquitto.conf: anonymous, plaintext, local-network-only

Data pipeline (poller):

  • pollers/mqtt_subscriber.py: groups sources by broker, one aiomqtt connection
    per broker, MQTT wildcard topic matching, reconnects on failure
  • normalizers/tinygs_mqtt.py: tinygs_satellite entities + satellite_contact
    events from local TinyGS node packets (SNR, RSSI, satPos, decoded payload)
  • normalizers/rtl_433.py: rf_sensor entities at home coords from RTL_433 SDR
    decoded frames (temperature, humidity, wind, power, etc.)
  • normalizers/meshtastic_mqtt.py: mesh_node entities from Meshtastic JSON MQTT
    uplink; text messages fenced from MeshCore by source_url prefix "mqtt:"
  • normalizers/ais_mqtt.py: vessel entities via AIS-catcher MQTT output,
    delegates to existing normalize_ais_catcher, alternative to WebSocket poller
  • db.py: add write_mesh_message() for Meshtastic MQTT chat persistence

Config layer (both backend and poller):

  • config_loader.py (x2): MqttSourceEntry model, added to SourcesConfig
  • config_writer.py: add/remove/update_mqtt_entry keyed by name (not url)
  • config_sync.py: _sync_mqtt_sources with full config/user diff semantics
  • db/init/10_mqtt_sources.sql: mqtt_sources table (no credential columns)

API:

  • routers/sources.py: GET/POST/PATCH/DELETE /sources/mqtt endpoints
  • routers/admin.py: tinygs_satellite and rf_sensor added to _TYPE_TO_POLLER
  • db/models.py: MqttSource ORM model

Auth: boolean auth_enabled gate per source; credentials from env vars
MQTT_{UPPER_SNAKE_NAME}_USERNAME/PASSWORD — never stored in DB or YAML

Frontend:

  • storeTypes.ts: tinygs_satellite and rf_sensor added to EntityTypeFilter
  • store.ts: defaults (true) and TTLs (3600s satellite, 900s sensor)
  • MapOverlay.tsx: entityFilterRef initializer updated to include new types

Config docs:

  • sources.example.yml: mqtt_sources section with all four source types
  • .env.example: MQTT_PORT, auth credential naming convention, updated
    TINYGS_ENABLED comment pointing users toward local MQTT path

claude added 3 commits May 21, 2026 04:05
Adds Mosquitto as a core container (always-on, no profile flag) and a new
mqtt_subscriber poller that dispatches incoming messages to per-normalizer
handlers. Replaces the unstable TinyGS REST-only integration with a proper
local-node MQTT path while keeping the REST poller as an opt-in fallback.

Infrastructure:
- docker-compose.yml: mosquitto service (eclipse-mosquitto:2), mosquitto_data
  volume, LAN port 1883 (MQTT_PORT), poller now depends on mosquitto health
- infra/mosquitto/mosquitto.conf: anonymous, plaintext, local-network-only

Data pipeline (poller):
- pollers/mqtt_subscriber.py: groups sources by broker, one aiomqtt connection
  per broker, MQTT wildcard topic matching, reconnects on failure
- normalizers/tinygs_mqtt.py: tinygs_satellite entities + satellite_contact
  events from local TinyGS node packets (SNR, RSSI, satPos, decoded payload)
- normalizers/rtl_433.py: rf_sensor entities at home coords from RTL_433 SDR
  decoded frames (temperature, humidity, wind, power, etc.)
- normalizers/meshtastic_mqtt.py: mesh_node entities from Meshtastic JSON MQTT
  uplink; text messages fenced from MeshCore by source_url prefix "mqtt:"
- normalizers/ais_mqtt.py: vessel entities via AIS-catcher MQTT output,
  delegates to existing normalize_ais_catcher, alternative to WebSocket poller
- db.py: add write_mesh_message() for Meshtastic MQTT chat persistence

Config layer (both backend and poller):
- config_loader.py (x2): MqttSourceEntry model, added to SourcesConfig
- config_writer.py: add/remove/update_mqtt_entry keyed by name (not url)
- config_sync.py: _sync_mqtt_sources with full config/user diff semantics
- db/init/10_mqtt_sources.sql: mqtt_sources table (no credential columns)

API:
- routers/sources.py: GET/POST/PATCH/DELETE /sources/mqtt endpoints
- routers/admin.py: tinygs_satellite and rf_sensor added to _TYPE_TO_POLLER
- db/models.py: MqttSource ORM model

Auth: boolean auth_enabled gate per source; credentials from env vars
MQTT_{UPPER_SNAKE_NAME}_USERNAME/PASSWORD — never stored in DB or YAML

Frontend:
- storeTypes.ts: tinygs_satellite and rf_sensor added to EntityTypeFilter
- store.ts: defaults (true) and TTLs (3600s satellite, 900s sensor)
- MapOverlay.tsx: entityFilterRef initializer updated to include new types

Config docs:
- sources.example.yml: mqtt_sources section with all four source types
- .env.example: MQTT_PORT, auth credential naming convention, updated
  TINYGS_ENABLED comment pointing users toward local MQTT path
TinyGS nodes can only connect to one MQTT broker at a time (local or
mqtt.tinygs.com), making local integration impossible without losing
cloud connectivity. Sunset the entire integration.

- Delete poller/pollers/tinygs.py, poller/normalizers/tinygs_mqtt.py
- Delete frontend TinyGS layer files (TinyGSLayer.tsx, buildTinyGSLayer.ts)
- Remove tinygs_station / tinygs_satellite from entity type filters,
  store TTLs, sidebar counts, EntityDetail, EntitySearch, CommsPanel,
  MapOverlay, useWebSocket, SignalQualityChart
- Remove tinygs_station from admin.py _TYPE_TO_POLLER
- Remove TINYGS_ENABLED env var and all TinyGS config_loader/config_sync
  references
- Update sources.example.yml and .env.example to remove TinyGS entries

https://claude.ai/code/session_01AgyyGJSXN21kGaSYF5ZMu9
Wires rf_sensor into the full frontend pipeline so 433MHz RF sensors
decoded by RTL_433 are visible on the map and inspectable.

- Add 'sensor' Track type to storeTypes; add signal_quality to Entity
- entityToTrack: recognize rf_sensor → type 'sensor'
- atlasIcons: add diamond icon (row 3, col 1) for rf_sensor
- buildEntityLayers: render sensor type with diamond icon + lime label at z10+
- colorUtils: lime-rf (#76DD00) color for sensor tracks
- tailwind.config.js: add lime-rf / lime-rf-muted design tokens
- RfSensorOverview: new detail panel showing readings grid (temp,
  humidity, pressure, wind, rain, UV, power, etc.), battery status,
  signal quality bar
- EntityDetail: wire in RfSensorOverview, add rf_sensor to TYPE_COLORS/TYPE_ICONS
- useWebSocket: add rf_sensor to FILTER_KEY_TO_ENTITY_TYPE
- Sidebar: add RF Sensors count + toggle button (compact and expanded)

https://claude.ai/code/session_01AgyyGJSXN21kGaSYF5ZMu9
@d3mocide d3mocide merged commit 6d1e730 into main May 22, 2026
6 checks passed
@d3mocide d3mocide deleted the claude/audit-tinygs-integration-fSkyf branch May 22, 2026 13:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants