IrohDNSPublished=True flips to DeferredToOwner mid-session on an actively-serving Connector

[drewr]

Summary

I had two cases with different semantics where an otherwise healthy tunnel stopped functioning after hours. After adding more logging and diagnosing other issues, the second of these cases revealed that there was a collision in resources across two different Datum Cloud accounts.

Much of the following is Opus 4.7 context that needed to save somewhere. But the two biggest needs I have from this issue are:

• Potentially merge connector/iroh-dns controllers fail to engage on 221 PCPs missing coordination.k8s.io discovery → Connector.Ready frozen indefinitely; 5s retry loop likely driving OOMKill #160, as it has a fix for part of the issue
• A PR which more fully fixes the issue (thanks @savme for fix: competing owner lease takeover #178!)

Observed behavior

On a Connector with IrohDNSPublished=True; Reason=Owner that has been serving traffic, the condition periodically flips to:

Type:    IrohDNSPublished
Status:  False
Reason:  DeferredToOwner
Message: iroh DNS record is owned by Connector /<project>/default/<other-connector>
         (uid <other-uid>)

The transition occurs on a server-side reconcile. No client action triggers it (no Create/Update/Delete on the affected Connector or its dependencies). After the flip, traffic to the corresponding tunnel returns 5xx at the edge until ownership is reclaimed.

Two cases of this behavior have been observed in the field. They share the same user-visible pattern but the cited "owner" Connector is in different states in each.

---

Case 1 — cited owner does not exist in the same project

• Active Connector: datum-connect-ltwr5, project drewr-y4nd1b, UID e43a8dd4-b10e-42de-b9ff-032788491a24.
• Cited owner: datum-connect-jttwh, same project drewr-y4nd1b, UID 226a90b6-3cad-4242-9eff-c2c71a335545.
• datumctl get connector datum-connect-jttwh --project drewr-y4nd1b -n default returns NotFound (not a permission error — same auth context that successfully resolves ltwr5).
• IrohDNSPublished on ltwr5 was True; Owner from 2026-06-07T14:44:39Z, flipped to False; DeferredToOwner at 2026-06-07T18:17:58Z — ~3.5 hours of active operation before the flip.
• Live HTTPProxy tied to the affected Connector (tunnel-kl6gj, hostname talk-known-phk4r.datumproxy.net) returned HTTP 503 from the moment the flip was visible, until recovery.

Case 2 — cited owner is a live Connector in a different user's project; ownership oscillates

• Active Connector: datum-connect-mhxj5, project drewr-y4nd1b, account a...s@gmail.com.
• Cited owner: datum-connect-dwq9z, project my-project-4h6v89, account d...s@datum.net. 87 days old. Confirmed to exist via datumctl get connector datum-connect-dwq9z --project my-project-4h6v89 -n default under d...s@datum.net's auth context.
• Both Connectors hold the same iroh public key (c1469d2f5d1547edbdc2dcbd007d97f92be25e1efbbd9bb728a1d160797fd2b8) in status.connectionDetails.publicKey.id.
• The flip from True; Owner to False; DeferredToOwner on mhxj5 occurred after the agent had been actively serving traffic for many hours.
• Approximately 13 seconds after the runtime watch reported the flip and the agent disconnected, IrohDNSPublished on mhxj5 transitioned back to True; Owner (Last Transition Time: 2026-06-08T15:44:51Z). The arbitration result inverted in seconds, on its own.
• At the time of the flip:
  • dwq9z's Lease had presumably been expired for an extended period (no agent active in my-project-4h6v89).
  • mhxj5's Lease was being renewed on schedule by an active heartbeat.

Common pattern across both cases

• The affected Connector has been actively serving traffic, with Ready=True and a fresh Lease, at the moment of the flip.
• The flip is unilateral from the operator side; no client request or resource mutation triggers it.
• The flip causes data-plane breakage (5xx at the edge) for as long as the Connector remains in DeferredToOwner.

---

Repro coordinates

Useful for an operator-side investigation:

Field	Case 1	Case 2
Affected Connector	drewr-y4nd1b/datum-connect-ltwr5	drewr-y4nd1b/datum-connect-mhxj5
Affected Connector UID	e43a8dd4-b10e-42de-b9ff-032788491a24	(not captured; can re-query)
Cited "owner"	drewr-y4nd1b/datum-connect-jttwh	my-project-4h6v89/datum-connect-dwq9z
"Owner" UID	226a90b6-3cad-4242-9eff-c2c71a335545	c0968142-c465-440a-ad40-fdb797ebf366
"Owner" current existence	Does not exist (NotFound)	Exists, 87d old, in different user's project
Flip detected at	2026-06-07T18:17:58Z	~2026-06-08T15:44:38Z
Subsequent flip back	Not observed (agent shut down via runtime watch)	2026-06-08T15:44:51Z (after agent shut down)
Live tunnel impact	HTTP 503 on talk-known-phk4r.datumproxy.net	Runtime watch caught the flip and exited the agent

---

Why this is filed

Investigation notes (proposed mechanisms, narrowing the cases) are in the comments below, where they belong as evidence is gathered — not in the issue body. The behavior described above is what's observed; the causes will be characterized as the comments accumulate.

The CLI-side mitigation that catches the flip and exits the agent cleanly is datum-cloud/app@6264818 (runtime watch, fail-fast on DeferredToOwner). The CLI-side recovery path is datum-cloud/app@76987b7 (per-project listen_key, prevents future cross-project collisions within a single user's accounts).

[drewr]

Investigation summary across both cases

What we learned tracing this from the agent side over the past two days. Recording it here so the issue body stays purely descriptive; treat the items below as evidence, not conclusions.

How the two cases were found

• Case 1 surfaced as a 5xx-after-Ready behavior: all six controller conditions on the agent-side HTTPProxy + Connector flipped True at ~0.1s on resume, but https://<hostname> returned 5xx for ~140s. The agent-side CLI verify phase (datum-cloud/app@1ed969e) is what exposed it as a measurable wait rather than "felt slow." Querying the Connector mid-wait showed IrohDNSPublished=False; DeferredToOwner citing jttwh. The same agent had been working fine for ~3.5 hours of clean operation before the flip.
• Case 2 surfaced after many hours of uninterrupted operation. Agent ran fine; runtime watch (datum-cloud/app@6264818) tripped on a DeferredToOwner transition and exited cleanly. The cited owner dwq9z was initially believed deleted (same as jttwh) because the agent's user couldn't resolve it. The actual reason was an auth-scope 404, not a not-found 404 — caught only after datumctl auth switch across the operator's other accounts.

What's shared across the two cases

• The agent has been actively serving traffic, with a freshly-renewed Lease, at the time of the flip.
• The agent's iroh public key is the same in both runs (c1469d2f...) — a single physical machine, originally a flat machine-wide listen_key (pre-datum-cloud/app@76987b7).
• The cited "owner" Connector in both cases also holds that same iroh public key in its status.connectionDetails.publicKey.id. So in both cases there are two records claiming the same iroh public key.
• In neither case did any client-side action trigger the flip.

What differs between the two cases (the part that's surprising)

	Case 1 (jttwh)	Case 2 (dwq9z)
Cited owner exists?	No — confirmed NotFound under the agent's same auth context that resolves the affected Connector	Yes — 87 days old, in a different user's project (d...s@datum.net)
Implication	The DNS ownership record outlasts the lifetime of the Connector that claimed it	An idle Connector (no live agent, expired Lease) is treated as a valid arbitration claimant against an active one

So the two cases produce the same user-visible failure mode but expose two different things about the arbitrator's state: (1) it considers Connectors that no longer exist, and (2) it doesn't weight liveness/Lease freshness when picking a winner among Connectors that do exist.

Recovery paths we've confirmed work

• Per-project iroh identity rotation (datum-cloud/app@76987b7): wiping the agent's listen_key for the affected project generates a new iroh public key, registers a new Connector with that key, and sidesteps the collision entirely. The hostname is preserved (lives on the HTTPProxy resource).
• Deletion of the idle Connector (Case 2): if the operator has access to the conflicting project, datumctl delete connector <name> --project <project> -n default removes the competing claim.
• Adoption-by-id (datum-cloud/app@ca4470f): resume the same HTTPProxy under a new iroh identity by id. Preserves user-visible URLs.

Diagnostic procedure that revealed Case 2's true nature

This is the part of the investigation worth recording, because the original Case 1 reasoning assumed it generalized to Case 2 and that turned out to be wrong:

1. Read the DeferredToOwner message verbatim. It names a project path and UID.
2. datumctl get connector <name> --project <project> -n default under the agent's auth.
3. If NotFound: also datumctl auth switch to every account the operator has and re-run the query under each. A 404 under one account does not mean the resource doesn't exist — it may be visible only under a different account's auth.
4. If still NotFound across all accessible accounts: the record genuinely doesn't exist anywhere the operator can see (Case 1 shape).
5. If found under another account: cross-account / cross-user collision (Case 2 shape).

The original investigation of Case 1 made step 3 only under one account and concluded "deleted" prematurely. Case 1 still turns out to be a real "deleted under any account I can see" once verified — but the procedure for confirming that wasn't followed rigorously enough on the first pass.

What's been ruled out (not the cause)

These are all things we initially suspected and then disproved:

• Not the operator's EnvoyPatchPolicy reconcile latency (tracked separately in #175). The CLI verify phase shows that proxy 5xx in this scenario correlates with the DeferredToOwner condition, not with Programmed lagging.
• Not the agent's spec patches. datum-cloud/app@0311960 made update_project idempotent at the lib boundary. Both observed cases had Tunnel already configured correctly. in the agent log — no update_active call, no HTTPProxy.spec patch, no metadata.generation bump.
• Not edge-side iroh dial latency (datum-cloud/iroh-gateway#12). That's a separate issue with its own signature (receive_relay arriving slowly without a DeferredToOwner flip on the agent's Connector). Both phenomena produce 5xx at the edge but they have distinct diagnostics.

Open questions for the operator team

Phrased as questions rather than claims so the answers can be confirmed against the actual implementation:

• What criterion does the iroh-DNS arbitrator use to select an owner among multiple Connectors holding the same status.connectionDetails.publicKey.id?
• Does that criterion consider Lease freshness / Ready condition status? If not, should it?
• In Case 1's shape: is there a path by which a DNSRecordSet or in-memory ownership-cache entry survives the deletion of the Connector that created it? Is owner-reference cascade firing correctly?
• In Case 2's shape: 13-second oscillation between two valid claimants implies a non-deterministic or fast-changing tiebreaker. What is it?
• Is there cluster-wide visibility into "which iroh public keys are claimed by multiple Connectors" that the operator team could use as a leading indicator?

[drewr]

Drew comment

Separate Opus session, this one run from the network-services-operator context.

Operator-side findings

Traced through internal/controller/iroh_dns_controller.go end-to-end. Posting what the code actually does and where each case's failure mode lives.

There is no arbitrator

Ownership is decided by a single Kubernetes Create race for one DNSRecordSet named iroh-<z32> in the downstream cluster. Whoever wins stamps four labels (iroh-dns-claimed-by-uid, …-connector-cluster, …-connector-namespace, …-connector-name). The only ownership check is byte-equality of the UID label at iroh_dns_controller.go:170-171:

currentClaim := existing.Labels[irohDNSClaimedByUIDLabel]
if currentClaim != string(connector.UID) { … defer … }

Direct answers to the open questions in the issue:

• Criterion for picking a winner among Connectors sharing a publicKey.id: none beyond who wins the apiserver Create. No creationTimestamp, UID compare, Lease freshness, or Ready check.
• Does it consider Lease freshness / Ready? No.
• Can the DNSRecordSet outlive its owning Connector? Yes, on specific failure modes (below). The happy path is correct.
• Cluster-wide visibility into duplicate-key claims: none today. No field-indexer on publicKey.id, no in-memory cache, no metric.

Case 1 (jttwh does not exist): orphan DNSRecordSet

No OwnerReference is set on the DNSRecordSet (intentional — cross-cluster). Cleanup is exclusively via the finalizer networking.datumapis.com/iroh-dns-cleanup → handleDeletion → releaseIfOwner.

releaseIfOwner (iroh_dns_controller.go:207-230) computes the record name from the Connector's own status.connectionDetails.publicKey.id. If that field is empty/nil at deletion time:

z32, err := connectorEndpointZ32(connector)
if err != nil || z32 == "" {
    return nil
}

…it returns nil immediately, handleDeletion removes the finalizer, and the DNSRecordSet is orphaned forever. From that point every sibling sharing the same public key gets permanent DeferredToOwner naming a Connector that no longer exists — exactly Case 1's shape.

Other ways the record can leak: delete --grace-period=0; finalizer manually removed during an operator outage; an error from EndpointHexToZ32 (line 209 swallows it).

Case 2 (dwq9z alive but idle; 13-second oscillation): no liveness + fragile handover

Two pieces:

1. Why an idle owner sticks. The operator never weighs Lease freshness or Ready. As long as the foreign label says claimed-by-uid=<dwq9z-UID>, mhxj5 will get DeferredToOwner even if dwq9z's agent has been dead for hours.

2. Why it oscillates. Sibling handover relies on the Lease watch at iroh_dns_controller.go:458-461. On main there is no periodic RequeueAfter in this reconciler (grep confirms). So when something does fire a reconcile, the sibling re-checks the labels once and returns to whatever they show. Owner and sibling reconcilers racing on independent Lease renewals — plus the SSA Patch at line 179 re-stamping labels when the owner reconciles — is consistent with the 13s flip-back.

The unmerged fix matters here

Commit 3578c30 on branch fix/optional-lease-watch ("Fixes #160") does two things that are directly relevant:

• Wraps the Lease watch in an OptionalKind source that degrades gracefully when a project control plane omits coordination.k8s.io from discovery. On main today, multicluster-runtime rejects the controller+cluster pair entirely for any PCP in that state — the Lease watch is silently dead and the sibling reconcile never fires from a heartbeat.
• Adds an unconditional RequeueAfter: leaseDurationSeconds to every iroh-routed reconcile, so handover converges within ≈ one lease duration even when the watch is disengaged.

Neither fully fixes #174, but without 3578c30 on main:

• The Case 2 oscillation has no upper bound on how long an idle owner can hold the claim past Lease expiry — a deferred sibling may not reconcile at all between sparse events.
• Case 1's orphan record is invisible to the system until something else (HTTPProxy change, ConnectorClass change, manual edit) triggers reconciles on every sibling.

Landing 3578c30 is a prerequisite for the real fixes; it gives the reconciler a guaranteed convergence cadence to do anything new on.

What an actual fix needs

1. Liveness-aware arbitration at line 171. When seeing a foreign claim, look up the labelled owner Connector across clusters (cluster label + ns + name); if NotFound or Lease expired, take over via SSA Patch with ForceOwnership and updated labels. Handles both Case 1 (owner gone) and Case 2 (owner idle).
2. DNSRecordSet GC independent of the owner Connector's status. Either make releaseIfOwner find the record by iroh-dns-claimed-by-uid=<UID> label (List, not z32 lookup) so a status-cleared Connector can still release; or run a periodic janitor over all records labelled app.kubernetes.io/managed-by=networking.datumapis.com and delete any whose labelled owner no longer exists.
3. Land 3578c30 (or replace with an equivalent deterministic periodic reconcile).
4. Field-indexer on Connector by status.connectionDetails.publicKey.id + a connector_duplicate_public_keys metric. Cheap, no API change, gives the cluster-wide visibility the issue asks for.

Ruled out by reading the code

• Not a stale in-memory cache — there isn't one. IrohDNSReconciler struct has only mgr, Config, Downstream.
• Not an OwnerReference cascade race — none is set on DNSRecordSet.
• Not a misencoded cluster label — encodeIrohClusterLabel/decodeIrohClusterLabel round-trip cleanly; the /<project>/default/<connector> shape in the message is exactly what decodeIrohClusterLabel produces (multicluster-runtime cluster names start with /).
• Not the SSA ForceOwnership Patch stealing records — it only runs after the currentClaim == UID check passes.

[drewr]

As noted on the PR, I think this was a red herring masking a simple case of trying to reuse an Iroh key across connections, which was a regression while developing the connect plugin where local state wasn't being properly accounted across projects.