ResourceClaims not garbage-collected when HTTPProxy owner is deleted

[drewr]

Problem

Been churning a lot on tunnels and was hitting quota limits even though I had deleted each tunnel (and associated Connector and HTTPProxy). When an HTTPProxy resource is deleted, the corresponding ResourceClaim (quota reservation) is not being garbage-collected by the quota-system component. This leaves orphaned ResourceClaims that continue to consume quota, eventually blocking new tunnel creation.

However, when an HTTPProxy resource is deleted, the corresponding ResourceClaim (quota reservation) doesn't seem to be garbage-collected by the quota-system component. This leaves orphaned ResourceClaims that continue to consume quota, eventually blocking new tunnel creation.

% datumctl get httpproxies --project drewr3-ceu4gt --namespace default
No resources found in default namespace.
% datumctl get resourceclaims --project drewr3-ceu4gt --namespace default | grep httpprox
httpproxy-tunnel-2bljz                           True      17h
httpproxy-tunnel-99qws                           True      87m
httpproxy-tunnel-brj5l                           True      17h
httpproxy-tunnel-dfxjk                           True      17h
httpproxy-tunnel-ghjjg                           True      17h
httpproxy-tunnel-l8q2j                           True      17h
httpproxy-tunnel-m6bws                           True      5d12h
httpproxy-tunnel-q7mmx                           True      17h
httpproxy-tunnel-tv25z                           True      21h
httpproxy-tunnel-wrm2v                           True      20h

Expected Behavior

ResourceClaims have an ownerReferences field pointing to their HTTPProxy:

ownerReferences:
- apiVersion: networking.datumapis.com/v1alpha
  kind: HTTPProxy
  name: tunnel-<id>
  uid: <uid>

When the HTTPProxy is deleted, the ResourceClaim should be automatically garbage-collected by the controller.

Actual Behavior

ResourceClaims remain in the cluster with stale ownerReferences after the HTTPProxy is deleted. The garbage collector does not clean them up.

Evidence

• ResourceClaim metadata shows quota.miloapis.com/auto-created: "true" and app.kubernetes.io/name: network-services-operator
• The controller watches HTTPProxy deletions but does not trigger ResourceClaim cleanup
• Orphaned claims persist indefinitely and consume project quota

Impact

Users who delete tunnels accumulate orphaned ResourceClaims. Once the project quota is exhausted (e.g., 10 HTTPProxy claims), no new tunnels can be created even when the project has no active HTTPProxies.

Additionally, users cannot manually delete orphaned claims — they get Forbidden errors:

resourceclaims.quota.miloapis.com "httpproxy-l8q2j" is forbidden: User "a...s@gmail.com" cannot delete resource "resourceclaims" in API group "quota.miloapis.com" in the namespace "default"

Only platform admins have permission to delete ResourceClaims, so affected users are completely blocked until the platform team manually cleans up.

Steps to Reproduce

1. Create a tunnel (creates HTTPProxy + ResourceClaim)
2. Delete the tunnel (HTTPProxy is deleted, ResourceClaim remains)
3. Repeat until quota is exhausted
4. New tunnel creation fails with "Quota limit exceeded for HTTPProxy resources"

Suggested Fix

1. Immediate: Add a mechanism for the platform to bulk-delete orphaned ResourceClaims for affected projects
2. Root cause: Ensure the quota-system controller properly watches for HTTPProxy deletions and deletes associated ResourceClaims. The ownerReference-based garbage collection may not be working because the ResourceClaim was created before the ownerReference was set, or the controller watch is incomplete.
3. RBAC: Consider whether end users should have permission to delete ResourceClaims tied to their own projects, as a recovery mechanism for this bug.

[drewr]

Deleted project drewr3-ceu4gt to get around it for now.