fix(transport): detect a physically dropped serial/TCP connection so ConnectionStatus.Lost is reachable - #403
fix(transport): detect a physically dropped serial/TCP connection so ConnectionStatus.Lost is reachable#403tylerkron wants to merge 4 commits into
Conversation
…oses #382) A DAQiFi device kept reporting IsConnected == true after its USB cable was pulled, so ConnectionStatus.Lost — which DaqifiDevice is fully wired to raise — was unreachable on serial. SerialPort.IsOpen reflects the OS handle, which stays open after the device vanishes, and neither the consumer's read failures nor the producer's write failures were ever fed back to the transport. Two detectors now sit on top of the handle, sharing one TransportConnectionWatchdog so "declare the connection lost exactly once" lives in a single place: - Port-presence polling (serial), 1 s cadence, 2 consecutive misses required. Presence is SerialPort.GetPortNames() falling back to the Unix device node — no WMI, identical on Windows/macOS/Linux. Armed only when the port is visible to that probe at connect time, so a spelling the probe cannot see disables the check instead of reporting a false drop. Bounds unplug detection at ~3 s. - I/O fault escalation via the new ITransportHealthSink: the reader and writer loops report every read/write outcome, and 5 consecutive failures with no successful transfer between them are a drop. A blip that recovers resets the run and never disconnects. TCP had the same gap — nothing raised StatusChanged(false) on an unexpected drop — and is fixed alongside: the same escalation (including a zero-byte socket read, which means the peer closed), plus TCP keep-alive so a silently severed link is caught within ~20 s of idleness. An intentional Disconnect() disarms both detectors first, so it still reports Disconnected, never Lost. #381's "know this limit" note is removed from docs/DEVICE_INTERFACES.md and DaqifiDeviceRegistry's XML docs, replaced by a documented detection-bound table. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- A write TIMEOUT is no longer reported as an I/O fault. It means the device is not draining its receive buffer right now (busy or flow-controlled), not that the link is gone — escalating it could disconnect a healthy device. Mirrors how the reader loop already treats a read timeout as benign. - Public XML docs no longer cref members of the internal TransportConnectionWatchdog; the thresholds are stated in prose so the generated documentation resolves for consumers. - docs: document that Lost is raised on an internal reader/timer thread, with the pattern for tearing the device down off the callback. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
/agentic_review |
Code Review by Qodo
1.
|
… port Qodo review: IsPortEnumerated swallowed exceptions from GetPortNames() and the filesystem check and returned false, so a failure to OBSERVE was indistinguishable from the port actually being gone. Two transient enumeration failures would count as two consecutive misses and close a healthy connection — the watchdog's "a throwing probe is no evidence" guard could never engage for the default probe. A probe that cannot answer now throws: - The enumeration failure is swallowed only when the filesystem check can still answer for that port name (a Unix device-node path); otherwise it propagates. - StartDropDetection catches a throwing baseline probe and leaves presence polling off rather than guessing, and never lets it escape a successful connect. - The unplug harness's port watcher treats a throwing probe as "keep watching". Two regression tests: a probe that starts throwing after arming reports no drop and stays armed, and a probe that throws at connect time leaves the check unarmed while fault escalation still covers the port. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Finding 1 — "Probe errors trigger false loss": VALID, fixed in d5f2c6b. Correct, and it defeated the safeguard I thought I had. Took the shape of your Option B, with one refinement. Callers updated to match:
Two regression tests, per your suggestion: a probe that starts throwing after arming reports no drop and stays armed, and a probe that throws at connect time leaves the check unarmed while fault escalation still detects the drop. Full suite green on net9.0 and net10.0 (1975 Core + 23 MCP), 0 warnings. |
|
/agentic_review |
|
Code review by qodo was updated up to the latest commit d5f2c6b |
Qodo review, two findings. 1. PollPresence swallowed a probe exception but kept the miss run, so "absent, could-not-observe, absent" satisfied a two-miss threshold that never actually saw the port absent twice in a row. The threshold is about CONSECUTIVE observed absences, so the exception now clears the run rather than merely failing to extend it. Regression test drives that exact sequence. 2. The unplug harness's PortDisappearanceWatcher published its timestamp across threads unsynchronized, so the reader could see a stale value and print a wrong detection latency — undermining the one number that harness exists to produce. Now a long ticks field published with Interlocked.CompareExchange, which also makes "record the first absence, once" explicit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Both new findings: VALID, fixed in 5dd7d7d. 1 — "Misses survive probe exception". Right, and it's the same class of mistake as the one I just fixed one layer down. I made the exception stop incrementing the run but never stop preserving it, so 3 — "Unsynchronized timestamp publication". Valid. Test-only, but that watcher exists for one reason — to produce a trustworthy detection-latency number for the hardware validation — and a stale read would corrupt precisely that, including the "port disappeared" vs "never disappeared" branch that tells the operator whether the cable actually came out. Took your Full suite green on net9.0 and net10.0 (1976 Core + 23 MCP), 0 warnings. |
|
/agentic_review |
|
Code review by qodo was updated up to the latest commit 5dd7d7d |
Closes #382.
The problem, plainly
Pull the USB cable out of a DAQiFi box and Core carries on as if nothing happened.
IsConnectedstaystrue, no event fires, and the app keeps showing a live device that silently returns no data — indefinitely. That is the single most common way a DAQ connection ends, and it was the one Core could not see.ConnectionStatus.Lostalready existed andDaqifiDevicewas already fully wired to raise it. Nothing could ever trigger it on serial, because three layers each swallowed the evidence:SerialStreamTransport.IsConnectedisSerialPort.IsOpen— the OS handle, which stays open after the device vanishes.So
OnStatusChanged(false, …)was only ever raised byDisconnectAsync— an intentional disconnect — and theStatus = Lostbranch was dead code for serial.What a user experiences now
"I unplugged it." Within about three seconds the device reports
ConnectionStatus.LostandIsConnectedgoes false — measured at 1.6 s on the bench with a real cable pull. If data was flowing, it is reported in well under a second. The registry prunes it, so a later reconnect under the same key opens a real connection instead of handing back the dead handle."I clicked disconnect." Still
Disconnected, neverLost. The two remain cleanly distinguishable — that is what makesLostactionable for #379's auto-reconnect."I'm on Windows / Linux / macOS." Identical behavior. No WMI, no
Win32_DeviceChangeEventwatcher, no OS-specific code path — which is the point, since the desktop app had to write exactly that outside Core.How it works
Two independent detectors, sharing one internal
TransportConnectionWatchdogso that "declare this connection lost, exactly once, whichever signal wins" lives in a single place.Port presence is
SerialPort.GetPortNames()containing the port, falling back to the device node still existing on Unix. Requiring both to say "absent" means a transient enumeration hiccup cannot disconnect anything. It is armed only if the probe can see the port at connect time, so a platform or port-name spelling the probe does not recognize disables the check instead of reporting a false drop on every poll — fault escalation still covers that port.I/O fault escalation is the new public
ITransportHealthSink.StreamMessageConsumerandMessageProducertake one optionally and report every read/write outcome;DaqifiDevicepasses the transport when it implements it. A single failure escalates nothing — the threshold is a run of five with no successful transfer in between, and any successful read or write resets the run. Timeouts are not failures at all: an idle read timeout is what a quiet device looks like, and a write timeout means the device is not draining its buffer right now, not that it is gone.Everything is additive.
IStreamTransportis unchanged, so the five test doubles implementing it (and any consumer's) still compile; a consumer that passes no health sink gets exactly the old behavior.The TCP transport (acceptance item)
Checked, and it had the same gap — worse in one way.
TcpClient.Connecteddescribes the last completed operation, and nothing raisedStatusChanged(false)for an unexpected drop, soConnectionStatus.Lostwas unreachable over TCP too. Fixed alongside:NetworkStreamunambiguously means the peer closed, and which the reader loop previously treated as "no data yet" and spun on forever. OnlyNetworkStreamis escalated this way; other stream types legitimately return 0 for "nothing right now".Hardware validation
Bench Nyquist 1, fw 3.7.2, macOS,
/dev/cu.usbmodem1101, example CLI built against this branch.Lost, no spuriousStatusChanged— the only status event all session wasDisconnectedat teardownLostStatus: Disconnectedon every run, neverLost✅ Physical unplug — validated, cable actually pulled
Run with the committed
SerialUnplugValidationTestsharness on the bench Nyquist 1 (fw 3.7.2, macOS,/dev/cu.usbmodem1101, net9.0). The harness watches the device node itself, so the latency below is measured from the moment the port actually vanished — not from the operator prompt.1610 ms from the port disappearing to
ConnectionStatus.Lost— comfortably inside the documented ~3 s bound, and consistent with the mechanism (a 1 s poll cadence needing 2 consecutive misses, with the unplug landing part-way through an interval). It reportedLost, neverDisconnected; the trailingDisconnectedis test-teardown disposal, after the assertion. The 20 s warm-up immediately beforehand produced zero status changes, so the same run demonstrates both halves: it fires when it should and stays silent when it shouldn't.Second run, cable replugged and pulled again, re-verifying PR #381's stale-registration pruning against a real unplug:
PruneDisconnected()returned 1 and the registry emptied, withDeviceRemovalReason.Disconnected— the pruning that #381 documented as unreachable for an unplugged USB device now works. The 16.04 s in that line is not a detection latency: unlike run 1, that timer starts at the operator prompt, so it is dominated by human reaction time before the cable ever moved. Run 1's 1610 ms is the real measurement.To re-run either (each needs its own unplug — replug in between):
Without
DAQIFI_UNPLUG_PORTboth log why they did nothing and return, so CI is unaffected.Testing
42 new unit tests plus the 2 hardware-gated ones. Full suite green on net9.0 and net10.0 (1976 Core + 23 MCP), run three times to check the new background timer introduced no flakiness in the timing-sensitive tests. 0 warnings.
The escalation and the "recoverable blip" are covered at three levels: the decision logic directly (
TransportConnectionWatchdogTests), the reader/writer loops against a stream that starts failing and then recovers, and end to end over a real loopback socket where killing the server producesConnectionStatus.Loston a realDaqifiDevice— and an intentionalDisconnect()on that same setup producesDisconnectedwith noLost.Acceptance criteria
Lostwithin a bounded, documented time, raisingStatusChanged— measured at 1610 ms on real hardware against the documented ~3 s bound (docs/DEVICE_INTERFACES.mdand XML docs)GetPortNames()+ Unix device-node fallback, no OS-specific branchDisconnect()still reportsDisconnected, notLost— unit tests + every bench run; the unplug run reportedLostand neverDisconnectedDeviceRemovalReason.Disconnected. The "know this limit" note is removed fromdocs/DEVICE_INTERFACES.mdandDaqifiDeviceRegistry's XML docs, replaced by the detection-bound table — and that replacement text is now backed by the measurement rather than by intentScope
Detection only. Automatic reconnect after
Loststays with #379, which this unblocks. On a detected drop the transport closes its handle but the device's consumer/producer are left running — tearing them down from inside the event would re-enter the very reader thread that raised it; the docs say to callDisconnect()off the callback instead.Sibling PR note: #399 is also editing
SerialStreamTransport, boundingWriteTimeoutand adding an SD-download deadline. No semantic overlap — this PR owns drop detection and adds no SD-path timeouts — but expect a textual conflict in the constructor and field block.Not merging — opened for review.
🤖 Generated with Claude Code