Add local MQTT broker and IoT hardware ingress pipeline#75
Merged
Conversation
Adds Mosquitto as a core container (always-on, no profile flag) and a new
mqtt_subscriber poller that dispatches incoming messages to per-normalizer
handlers. Replaces the unstable TinyGS REST-only integration with a proper
local-node MQTT path while keeping the REST poller as an opt-in fallback.
Infrastructure:
- docker-compose.yml: mosquitto service (eclipse-mosquitto:2), mosquitto_data
volume, LAN port 1883 (MQTT_PORT), poller now depends on mosquitto health
- infra/mosquitto/mosquitto.conf: anonymous, plaintext, local-network-only
Data pipeline (poller):
- pollers/mqtt_subscriber.py: groups sources by broker, one aiomqtt connection
per broker, MQTT wildcard topic matching, reconnects on failure
- normalizers/tinygs_mqtt.py: tinygs_satellite entities + satellite_contact
events from local TinyGS node packets (SNR, RSSI, satPos, decoded payload)
- normalizers/rtl_433.py: rf_sensor entities at home coords from RTL_433 SDR
decoded frames (temperature, humidity, wind, power, etc.)
- normalizers/meshtastic_mqtt.py: mesh_node entities from Meshtastic JSON MQTT
uplink; text messages fenced from MeshCore by source_url prefix "mqtt:"
- normalizers/ais_mqtt.py: vessel entities via AIS-catcher MQTT output,
delegates to existing normalize_ais_catcher, alternative to WebSocket poller
- db.py: add write_mesh_message() for Meshtastic MQTT chat persistence
Config layer (both backend and poller):
- config_loader.py (x2): MqttSourceEntry model, added to SourcesConfig
- config_writer.py: add/remove/update_mqtt_entry keyed by name (not url)
- config_sync.py: _sync_mqtt_sources with full config/user diff semantics
- db/init/10_mqtt_sources.sql: mqtt_sources table (no credential columns)
API:
- routers/sources.py: GET/POST/PATCH/DELETE /sources/mqtt endpoints
- routers/admin.py: tinygs_satellite and rf_sensor added to _TYPE_TO_POLLER
- db/models.py: MqttSource ORM model
Auth: boolean auth_enabled gate per source; credentials from env vars
MQTT_{UPPER_SNAKE_NAME}_USERNAME/PASSWORD — never stored in DB or YAML
Frontend:
- storeTypes.ts: tinygs_satellite and rf_sensor added to EntityTypeFilter
- store.ts: defaults (true) and TTLs (3600s satellite, 900s sensor)
- MapOverlay.tsx: entityFilterRef initializer updated to include new types
Config docs:
- sources.example.yml: mqtt_sources section with all four source types
- .env.example: MQTT_PORT, auth credential naming convention, updated
TINYGS_ENABLED comment pointing users toward local MQTT path
TinyGS nodes can only connect to one MQTT broker at a time (local or mqtt.tinygs.com), making local integration impossible without losing cloud connectivity. Sunset the entire integration. - Delete poller/pollers/tinygs.py, poller/normalizers/tinygs_mqtt.py - Delete frontend TinyGS layer files (TinyGSLayer.tsx, buildTinyGSLayer.ts) - Remove tinygs_station / tinygs_satellite from entity type filters, store TTLs, sidebar counts, EntityDetail, EntitySearch, CommsPanel, MapOverlay, useWebSocket, SignalQualityChart - Remove tinygs_station from admin.py _TYPE_TO_POLLER - Remove TINYGS_ENABLED env var and all TinyGS config_loader/config_sync references - Update sources.example.yml and .env.example to remove TinyGS entries https://claude.ai/code/session_01AgyyGJSXN21kGaSYF5ZMu9
Wires rf_sensor into the full frontend pipeline so 433MHz RF sensors decoded by RTL_433 are visible on the map and inspectable. - Add 'sensor' Track type to storeTypes; add signal_quality to Entity - entityToTrack: recognize rf_sensor → type 'sensor' - atlasIcons: add diamond icon (row 3, col 1) for rf_sensor - buildEntityLayers: render sensor type with diamond icon + lime label at z10+ - colorUtils: lime-rf (#76DD00) color for sensor tracks - tailwind.config.js: add lime-rf / lime-rf-muted design tokens - RfSensorOverview: new detail panel showing readings grid (temp, humidity, pressure, wind, rain, UV, power, etc.), battery status, signal quality bar - EntityDetail: wire in RfSensorOverview, add rf_sensor to TYPE_COLORS/TYPE_ICONS - useWebSocket: add rf_sensor to FILTER_KEY_TO_ENTITY_TYPE - Sidebar: add RF Sensors count + toggle button (compact and expanded) https://claude.ai/code/session_01AgyyGJSXN21kGaSYF5ZMu9
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Mosquitto as a core container (always-on, no profile flag) and a new
mqtt_subscriber poller that dispatches incoming messages to per-normalizer
handlers. Replaces the unstable TinyGS REST-only integration with a proper
local-node MQTT path while keeping the REST poller as an opt-in fallback.
Infrastructure:
volume, LAN port 1883 (MQTT_PORT), poller now depends on mosquitto health
Data pipeline (poller):
per broker, MQTT wildcard topic matching, reconnects on failure
events from local TinyGS node packets (SNR, RSSI, satPos, decoded payload)
decoded frames (temperature, humidity, wind, power, etc.)
uplink; text messages fenced from MeshCore by source_url prefix "mqtt:"
delegates to existing normalize_ais_catcher, alternative to WebSocket poller
Config layer (both backend and poller):
API:
Auth: boolean auth_enabled gate per source; credentials from env vars
MQTT_{UPPER_SNAKE_NAME}_USERNAME/PASSWORD — never stored in DB or YAML
Frontend:
Config docs:
TINYGS_ENABLED comment pointing users toward local MQTT path