Skip to content

Broker tenancy + per-IP grant cap + fail-closed guards#91

Merged
Alexgodoroja merged 5 commits into
mainfrom
feat/broker-tenancy-hardening
Jul 16, 2026
Merged

Broker tenancy + per-IP grant cap + fail-closed guards#91
Alexgodoroja merged 5 commits into
mainfrom
feat/broker-tenancy-hardening

Conversation

@Alexgodoroja

Copy link
Copy Markdown
Collaborator

Summary

Makes the managed-key broker enforce per-caller resource isolation for apps whose partner account cannot be split per user (e.g. io.pilot.agentphone, an account bound to a provider campaign), and closes several defaults that resolved to "allow" when a field was unset.

Upstream, every pilot user of a shared-account app is the same customer, so the partner cannot tell them apart and nothing enforced isolation. Any caller could reference any resource id on the shared account. The broker is now the tenancy layer.

What changed

Tenancy (internal/broker/tenancy.go, owner.go, store*.go)

  • Every resource is claimed by its creator; a reference in the path, query, or body is checked against an ownership ledger before the request is forwarded; list responses are filtered to the caller's own rows; account summaries are recomputed from the caller's ledger.
  • Deny-by-default: a resource with no ledger row belongs to nobody and is refused to everybody (this is what retires shared/legacy resources).
  • Parser-differential hardening: bodies with duplicate keys or trailing content are refused (the broker validates with one parser but forwards raw bytes to another).
  • 404 (never 403) on a denied reference, so the broker isn't an oracle for enumerating other tenants' ids.

Per-IP grant cap (broker.go)

  • The (N+1)th distinct identity on a source IP is recorded with a zero grant rather than refused: it can still read, but every priced call hits 402. Bounds free budget minted per network without hard-locking a shared-NAT newcomer out.

Fail-closed defaults

  • allow entries are method-scoped (a registry can express reads-yes/writes-no).
  • creditEnabled keys off the credit block's presence, not a non-zero seed (so seed_credits: 0 means a zero budget, not "no metering").
  • max_identities_per_ip defaults to a cap; only an explicit negative disables it.
  • A classic managed app that forwards with header auth but no header name is rejected at load, not on first traffic.
  • Store/upstream errors return a correlation id instead of internal hostnames/addresses.
  • Access-key gate (accesskey.go) available for apps that grant something of value; broker refuses to boot if an app requires a key that isn't configured.

Deploy (deploy/setup-broker-tls.sh, deploy/agentphone-tenancy.py)

  • Generated nginx location / now sets X-Real-IP (the only source-IP signal the broker trusts; without it the per-IP cap collapses to one bucket).
  • Registry script for agentphone: enables spend routes (bound by tenancy + grant cap), withholds account-level webhooks and unownable reactions.

Tests

Tests are written as attacks: cross-tenant use via a body reference, reads of another tenant's resources, ownership takeover, type confusion, parser-differential duplicate keys, count fields that leak the unfiltered set, and the grant-cap contract (capped identity reads but cannot spend).

Live verification (pilot-publish)

Deployed and exercised via a real pilot node against all four managed apps:

  • agentphone: full funded flow (create agent → buy number → attach → real SMS from the owned number); cross-tenant send/read/delete all 404; lists and usage counts scoped to the caller; per-IP cap boundary (3 funded, 4th zero-grant reads but 402 on spend); balance debited exactly.
  • sixtyfour / bowmark / orthogonal: forward correctly (no tenancy — they return data, no per-user resources).

🤖 Generated with Claude Code

Alexgodoroja added 3 commits July 16, 2026 11:32
Managed apps whose partner account cannot be split per user (an account bound
to a provider campaign) had no isolation between pilot callers: upstream every
caller is the same customer, so the partner cannot tell them apart and nothing
else was doing it. Any caller could reference any resource id on the shared
account, and account-wide lists were returned verbatim.

Tenancy (internal/broker/tenancy.go, owner.go) makes the broker the isolation
layer. A resource is claimed by whoever creates it; a reference in the path,
query, or body is checked against the ledger before the request is forwarded;
list responses are filtered to the caller's rows, and account summaries are
recomputed from the caller's ledger. Deny-by-default: a resource with no ledger
row belongs to nobody and is refused to everybody.

Also closes several defaults that resolved to "allow" when unset:
- allow entries are method-scoped, so a registry can express reads-yes/writes-no
  instead of one verb implying all of them.
- creditEnabled keys off the credit block's presence, not a non-zero seed, so
  seed_credits:0 means a zero budget rather than no metering at all.
- max_identities_per_ip defaults to a cap; only an explicit negative disables it.
- access keys gate apps that grant something of value; the broker refuses to
  boot if an app requires a key that is not configured.
- store and upstream errors return a correlation id instead of internal
  hostnames, addresses, and driver state.

Tests are written as attacks: cross-tenant use via a body reference, reads of
another tenant's resources, ownership takeover, type confusion, parser-
differential duplicate keys, and count fields that leak the unfiltered set.
…guards

Turns the per-IP identity cap into a GRANT cap: the (N+1)th distinct identity on
a source IP is recorded with a zero budget instead of being refused. It can still
make free (read) calls; every priced call hits 402. This bounds how much free
budget one network can mint without hard-locking a legitimate user behind a
shared NAT out of the app — so the cap can be set aggressively low.

- deploy/setup-broker-tls.sh: the generated nginx `location /` now sets
  X-Real-IP. It is the only source-IP signal the broker trusts, and the per-IP
  cap collapses to a single 127.0.0.1 bucket without it. This was the origin of
  the real-IP regression; regenerating nginx on deploy no longer reintroduces it.
- registry: a classic managed app that forwards with header auth but no header
  name is now rejected at load instead of 502-ing on first traffic.
- deploy/agentphone-tenancy.py: re-enables the spend routes (messages/calls/
  numbers) now that tenancy binds each to an owned resource and the grant cap
  bounds exposure; keeps webhooks/reactions withheld (account-level / unownable).

Tests updated to assert the grant-cap contract: a capped identity reads (200)
but cannot spend (402), while a funded identity spends normally.
The broker emits one `ACCESS {json}` line per forwarded request to stdout →
journald, which the monitoring crawler ingests. This middleware was lost when
the broker was last rebuilt from a branch without it, so the pipeline went dark.
Restoring it here keeps it alongside the tenancy/security changes, so a redeploy
from this branch does not clobber one for the other.

- internal/broker/accesslog.go: WithAccessLog middleware, AccessEvent (stable
  JSON contract), StdoutSink, and the X-Pilot-Method (pilotctl command capture)
  and X-Pilot-Unrouted (skip unknown-app / scanner noise) header constants.
- internal/broker/broker.go: mark the two unrouted 404 paths with X-Pilot-Unrouted
  so scanner traffic never counts as broker usage.
- cmd/broker/main.go: wire the middleware with a -broker-name label (defaults to
  BROKER_NAME env or hostname); /gw/ health/usage paths pass through unlogged.

Reduced to the stdout→journald path (what the pilot-publish crawler reads); the
remote-push HTTPSink for the smol broker is intentionally not included here.
@Alexgodoroja

Copy link
Copy Markdown
Collaborator Author

Added two operational fixes surfaced during live deployment (both prod-verified on pilot-publish):

  1. X-Real-IP durabilitydeploy/setup-broker-tls.sh now emits proxy_set_header X-Real-IP in the catch-all location /. This was the root cause of orthogonal 429-ing every new user: the generated nginx vhost only set X-Real-IP on the signup sub-locations, so all managed-app callers collapsed to 127.0.0.1 and the per-IP identity cap saw one bucket. The live box was hotfixed; this makes it survive a regenerate. (smol-broker's template already sets it.)

  2. ACCESS log restore — re-added the WithAccessLog middleware so the broker emits structured ACCESS {json} per request to journald (monitoring crawler → Postgres). It had been lost on a prior rebuild; shipping it here keeps security + monitoring on the same binary so neither redeploy clobbers the other.

Ledger note: pruned unused-full (credits==seed) 127.0.0.1 provision rows so the cap math is clean; spent rows and active identities kept. Verified real IPs now recorded and orthogonal seeds fresh users at $5.

Alexgodoroja added 2 commits July 16, 2026 11:47
Restores internal/broker coverage above the security-critical gate: the new
tenancy, access-key, and ownership code was under-covered. Adds direct tests for
access-key parse/verify (incl. fail-closed with no keys and the require-key
gate), the SQLite ownership ledger (claim first-writer-wins, OwnerOf/OwnedSet/
Release, Owns) and credit ledger (debit/refund/settle/clamp), and account-summary
redaction (owned-count recompute + field redaction).
@Alexgodoroja
Alexgodoroja merged commit 641d477 into main Jul 16, 2026
6 checks passed
@Alexgodoroja
Alexgodoroja deleted the feat/broker-tenancy-hardening branch July 16, 2026 18:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant