Skip to content

feat(ingestor): document and test ws:// / wss:// WebSocket MQTT broker support#902

Open
emuehlstein wants to merge 1 commit intoKpa-clawbot:masterfrom
emuehlstein:feat/ws-mqtt-source
Open

feat(ingestor): document and test ws:// / wss:// WebSocket MQTT broker support#902
emuehlstein wants to merge 1 commit intoKpa-clawbot:masterfrom
emuehlstein:feat/ws-mqtt-source

Conversation

@emuehlstein
Copy link
Copy Markdown

Summary

CoreScope's ingestor already supports WebSocket MQTT connections today — paho.mqtt.golang v1.5.0 handles ws:// and wss:// natively via gorilla/websocket. However this support was undocumented, untested, and had a TLS gap for wss:// connections.

This PR closes those gaps without any breaking changes.

Changes

cmd/ingestor/config.go

  • Added godoc comment to ResolvedSources() explaining all four supported schemes and which ones require translation vs. pass-through
  • ws:// and wss:// explicitly documented as native paho schemes requiring no mapping

cmd/ingestor/main.go

  • Extended TLS config to cover wss:// in addition to ssl://
  • Before: wss:// connections would use paho's default TLS (no explicit tls.Config set), which works for valid certs but doesn't apply the same predictable setup as ssl://
  • After: both ssl:// and wss:// get tls.Config{} (system CA pool), matching behavior; rejectUnauthorized: false still works for self-signed certs on both schemes

cmd/ingestor/config_test.go

Two new tests:

  • TestResolvedSourcesSchemeMapping: validates all six scheme variations (mqtt://, mqtts://, tcp://, ssl://, ws://, wss://) including paths like wss://host/mqtt
  • TestLoadConfigWSSource: full round-trip of a dual-source config (TCP + wss:// with username/password), verifies scheme unchanged through LoadConfig and ResolvedSources

config.example.json

  • Added wsmqtt example entry showing wss:// with username/password
  • Updated _comment_mqttSources to enumerate all supported schemes: mqtt://, mqtts://, ws://, wss://

Motivation

We run meshcore-mqtt-broker (a WebSocket MQTT bridge with JWT auth) alongside Mosquitto, and subscribe to both via mqttSources. The dual-source config works in production but nothing in the docs or example config made this discoverable for other operators.

Testing

cd cmd/ingestor && go test ./...
ok    github.com/corescope/ingestor  1.568s

All existing tests pass. Two new tests added.

No breaking changes

  • Existing configs: no change in behavior
  • ws:// / wss:// configs that were already working: same behavior + explicit TLS setup for wss://

…port

paho.mqtt.golang v1.5.0 already supports WebSocket MQTT connections natively
via gorilla/websocket. The ws:// and wss:// URL schemes pass through
ResolvedSources() unchanged — no scheme translation needed.

Changes:
- config.go: clarify ResolvedSources() scheme mapping in godoc comment;
  ws:// and wss:// explicitly noted as pass-through
- main.go: extend TLS config to cover wss:// (same system-CA-pool tls.Config{}
  already applied for ssl://); fixes missing TLS setup for WebSocket TLS
- config_test.go: add TestResolvedSourcesSchemeMapping (ws/wss/tcp/ssl/mqtt/mqtts
  all schemes) and TestLoadConfigWSSource (wss:// round-trip with auth)
- config.example.json: add wsmqtt example source showing ws:// / wss:// usage
  with username/password; update _comment_mqttSources to list all schemes

Tested with the meshcore-mqtt-broker WebSocket proxy running behind Caddy TLS.
@emuehlstein
Copy link
Copy Markdown
Author

testing at scope.chicagooffline and dev-scope.chicagooffline.com

@Kpa-clawbot
Copy link
Copy Markdown
Owner

Expert review on PR #902

Summary

Good PR — documents a real undocumented capability and closes a real TLS gap for wss:// connections. However, it needs a rebase before merge.

BLOCKER (2)

1. Merge conflict: cmd/ingestor/main.go
PR #949 (6345c6f) extracted inline MQTT options into buildMQTTOpts() (main.go:199-219). This PR's diff targets the old inline code that no longer exists. After rebase, the fix is straightforward: add || strings.HasPrefix(source.Broker, "wss://") to the TLS check at main.go:217.

2. Merge conflict: config.example.json
Master removed the db.vacuumOnStartup block and changed packetStore comments since this PR was branched.

Test assessment ✅

Both new tests are real logic tests, not tautological:

  • TestResolvedSourcesSchemeMapping (config_test.go:287): tests 8 scheme variants including ws:// and wss:// with paths — verifies pass-through behavior
  • TestLoadConfigWSSource (config_test.go:319): full round-trip through LoadConfig + ResolvedSources with wss:// + credentials

Tests exercise config/scheme handling (the scope of this PR), not paho's WebSocket transport — appropriate for a "document + test" PR.

Verification

  • go build ✅ — compiles clean on the PR branch
  • go test -run "SchemeMapping|WSSource" ✅ — both tests pass
  • Code reading confirms wss:// TLS gap is real on master: buildMQTTOpts() line 217 only checks ssl://

MINOR (1)

After rebase, the #nolint:gosec comment on the InsecureSkipVerify line should be retained — it's already on master's version.

NIT (1)

Consider adding the SetMaxReconnectInterval / SetConnectTimeout / SetWriteTimeout settings from #949 to the PR description's "no breaking changes" section, since those are now part of the function the fix targets.

Asks

  1. Rebase onto current master — resolve conflicts in main.go (move wss:// check into buildMQTTOpts) and config.example.json
  2. Re-run go test ./... after rebase to confirm

Verdict: merge after rebase 🟡

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants