Implement Milestone2 Networking (Ingress and Egress) by howardjohn · Pull Request #460 · agent-substrate/substrate

John Howard (howardjohn) · 2026-07-18T00:09:58Z

This implements most of the approved M2 networking milestone scope.

Changes

A new atunnel component is added within ateom. This is embedded in the same binary so the atunnel name is mostly internal. atunnel has 2 responsibilities: terminating incoming traffic from the ingress, and capturing and forwarding egress traffic.
- Actors no longer accept direct traffic. Traffic must flow through the ingress. Note the DNS already goes through the ingress, so existing usages should already be doing this.
  - This is currently enforce by lack of DNAT rules on the worker to direct to the actor (which we used to have; now removed). We could have NetworkPolicy additionally locking down to only 443 (where atunnel listens) or can leave that to users.
- atunnel will listener on port 443, terminate TLS, and forward to the actor's port 80. Additional ports are a TODO pending an API for this.
- atunnel additionally listens on port 15001 and accepts egress traffic from the application. A REDIRECT rule conditionally captures all traffic leaving the actor and forwards to the atunnel. The atunnel will establish a CONNECT+mTLS request to the configured egress proxy.
- Egress proxying is configured only if egress policies are set for the actor. This is currently impossible to set in this PR (in an effort to decouple API decisions from this PR; more on this below), so not actually triggered.
The Ext proc code is removed from atenet. This leaves only the DNS stuff in atenet; depending on the outcome of Consider removing custom DNS for actors #445 we can remove atenet entirely (if not, we should probably rename it or just merge it into ate-controller probably).
atenet-router is renamed to ateway-ingress and has a new implementation.
- Only the standalone mode is supported now. This matches the existing default. This means the ingress is its own standalone deployment.
- The high level flow remains the same: the ingress introspects the request to find the actor from the host, makes a ResumeActor call to the ate-api-server, and then dynamically forwards to that IP address. New in this PR:
  - mTLS from ingress to ateom/atunnel
  - atunnel verifies the actor the ingress send to is actually still the active actor
  - Actor IP assignment is cached in the ingress. Because atunnel can verify the actor, this allows us to actively evict+retry the cache if its stale.
- Ingress is substrate aware and adds the ate.* attributes to logs/traces/etc as defined in feat(otel): set ate.* actor telemetry identity on ateapi spans #412
ateway-egress is added. This terminates the CONNECT, and can enforce policies before dynamically forwarding the request.
- Currently, there is no policy, as there is not yet and API and this warrants a small PR that we can focus on the API. I had originally built out a simple stub API to verify this end to end; you can see the [Revert egress policy](https://github.com/agent-substrate/substrate/commit/21f854ef6a1d6d9cdc8cae9f506125c4432f9199) removing it. In parallel I will likely open a PR adding the API, but it doesn't need to block this
- The M2 scope policies are ACLs, static credential injection, and MITM. These have all been proved out as mentioned, just not included in this PR.
- Egress works similar to ingress; on each request a GetActor call is made to load the actor policies. This is cached. atunnel includes the actor id, atespace, and actor version. This ensures that if an actor policy is updated, we do not incorrectly apply stale policies from the cache.
- atunnel is configured to send to the egress IFF egress policy exists (note; this also means we should have an egress policy that says "allow anything", as a distinct thing from "no egress policy", to allowing sending to the egress. This is useful for custom egress, or just to get observablity. This is an API thing though).
- Redirection applies to TCP currently. All UDP is passed through for DNS; future PRs will lock down non-DNS UDP and/or proxy the DNS.
A new egress demo is added to showcase a worker that sends outbound requests.
Blocking merge: this PR currently uses an unofficial image for the ateways. I will swap it once we get high level buy in for this (just to avoid needing to do a bunch of releases if we need to iterate on this).

Testing

Some basic testing using the counter demo shows substantially reduced tail latencies on the ingress flow:

DEST            P50      P90      P99
ThisPR          0.171ms  0.254ms  0.517ms
Current         3.667ms  9.816ms  17.348ms

Example showing end to end flow with tracing:

Aside from unit testing this PR also adds some basic end to end tests of the ingress and egress flow

Extensibility

The M2 design is to be batteries included but swappable. Both ingress and egress can be swapped by replacing the ateway-ingress/ateway-egress deployments with their own. These components are identified by their hostname/service by the relevant components (atenet dns, ateom, etc) and can be overriden (or simply re-using the same name).

The ingress component is expected to identify an actor (through implementation-specific means), and forward to the appropriate worker on port 443 with mTLS. While the ingress could support custom hostnames on the incoming request flow it must send them in the standard my-counter.demo.actors.resources.substrate.ate.dev style.

The egress component is expected to accept CONNECT requests over mTLS and server the appropriate configuration for the actor identified by the X-Ate-Atespace, X-Ate-Actor, and X-Ate-Actor-Version headers. The :authority header contains the original destination of the traffic.

Note: I do not care if we keep the 'cute' names of ateway and atunnel. I am also fine to keep router though ingress aligns with egress which is why I shifted it.

For #430
For #126
For #326
Replaces #338
Replaces #393
Related to #340

Tests pass
- Note: egress policy tests are written but disabled as they are not yet included in this PR
Appropriate changes to documentation are included in the PR

Co-authored-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io> Co-authored-by: npolshakova <nina.polshakova@solo.io>

Bowei Du (bowei) · 2026-07-18T00:18:45Z

Did these files get moved? Where did this all go?

This was the old router logic for serving XDS/ext proc which is no longer there.

Looking at atstore specifically its used in two places:

For some reason we call it and ignore the result in the router controller

statusz page on the router.

I can add fetching the tempaltes and exposing them in atenet but it seems weird for atenet to expose these in its statusz given it doesn't care about them. I also think we can remove atenet (folding it into the controller) now that it does so little anyways

Bowei Du (bowei) · 2026-07-18T00:23:41Z

Probably want to split this up into:

egress
ingress
tunnel

in the real PRs

Bowei Du (bowei) · 2026-07-18T05:33:59Z

Taking a quick scan through the changes, I do think we have to discuss what it means for us to delete functionality out of the Substrate project and move it entirely into the agent-gateway repo, which is an independently evolving repo. This is not necessarily a good precedent to set for this project.

Specifically, I see that in order for this change to work, a changeset here (agentgateway/agentgateway#2586) also needs to go into Agent Gateway.

I would much prefer keeping as much of the core logic associated with Substrate including the networking as part of this project in a self-contained way. At the very least, one requirement in my mind is that code and implementation should be hosted here. It can be vendored outside to create plugable implementations, but at this point, we do need to keep things more contained than have a deep dependency on outside...

Lior Lieberman (LiorLieberman)

Thanks John Howard (@howardjohn), half way through it, still havent reviewed atunnel itself. Meanwhile, some nit comments in-line with a few questions here;

“Atunnel will listen on port 443.. and forward to the actor’s port 80”” – does it mean we only support 443 for ingress? can an actor receive on 443? (in-app tls)?
“A redirect rule conditionally captures all traffic leaving the actor” – what is the condition? is it only when a policy exist? I guess for the context of this PR is it when the req contains egressGateway address?
Can connect + mTLS be pluggable/configurable? If yes, how?
for “configured egress proxy” why not per atespace? I can see its on the actor now, how does this gets configured? And this means we cant change it while the actor is running, correct? I dont think we should be worried about supporting it now but would be better to document it if my assumption is correct.
"Egress proxying is configured only if egress policies are set for the actor”
I know API is TBD still but what would be the component that watches is and activates the egress proxy - the controller?
"The Egress component is expected to accept CONNECT requests over mTLS" – I think there were prior discussions to make that pluggable as well - how can we support different tunnels that are not only CONNECT?

Lior Lieberman (LiorLieberman) · 2026-07-18T01:15:24Z

+	if req.GetActorVersion() < 1 {
+		errs = append(errs, field.Invalid(field.NewPath("actor_version"), req.GetActorVersion(), "must be positive"))
+	}


nit: question: whats the reason for that? why would it be less than 1?

Lior Lieberman (LiorLieberman) · 2026-07-18T01:19:04Z

+	atunnelCredentialBundle    = pflag.String("atunnel-credential-bundle", "/run/podidentity.podcert.ate.dev/credential-bundle.pem", "PEM credential bundle for actor ingress HTTPS")
+	atunnelTrustBundle         = pflag.String("atunnel-trust-bundle", "/run/podidentity.podcert.ate.dev/trust-bundle.pem", "PEM trust bundle for actor ingress clients")
+	atunnelClientIdentity      = pflag.String("atunnel-client-identity", "spiffe://cluster.local/ns/ate-system/sa/ateway-ingress", "SPIFFE identity allowed to call actor ingress HTTPS")
+	atunnelEgressListenAddress = pflag.String("atunnel-egress-listen-address", "0.0.0.0:15001", "Address for transparently intercepted actor egress TCP")


nit: should 15001 be configured as a shared const somewhere to ensure consistency with workload_apply.go?

Lior Lieberman (LiorLieberman) · 2026-07-18T05:15:30Z


 	interiorNetNS netns.NsHandle
 	actorLogger   *actorlog.ActorLogger
+	atunnel       *atunnel.Server


nit: atunnelIngress ?

Lior Lieberman (LiorLieberman) · 2026-07-18T05:20:29Z

+	if egressClient != nil {
+		if err := s.atunnelEgress.Activate(egressClient, atespace, actorName, actorVersion, ""); err != nil {
+			if s.atunnel != nil {
+				_ = s.atunnel.Deactivate(context.Background())


why deactivating ingress if egress had a problem?

Lior Lieberman (LiorLieberman) · 2026-07-18T05:25:37Z

-	// should replace the broad masquerade path with transparent TCP capture and
-	// default-deny rules.
+	// TODO: Restrict the compatibility masquerade to DNS traffic sent to the
+	// configured cluster resolver and drop all other non-tunneled actor egress.


whats cluster resolver? you mean dns resolver? So that only DNS traffic can flow without a tunnel?

Lior Lieberman (LiorLieberman) · 2026-07-18T05:28:38Z

 }

-func tcpDestinationPortEqual(port uint16) []expr.Any {
+func tcpProtocol() []expr.Any {


function name sounds weird. maybe tcpProtocolMatcher()?

Lior Lieberman (LiorLieberman) · 2026-07-18T05:29:44Z

 	// logWriter with the runtime logger so the two streams to os.Stdout are
 	// serialized through one SyncedWriter and never interleave-corrupt lines.
 	actorLogger := actorlog.NewActorLogger(logWriter, metadata.OnGCE())
+	upstream, err := url.Parse("http://169.254.17.2:80")


nit: you defined actorHTTPUpstream somewhere, would be better to use that constant

Lior Lieberman (LiorLieberman) · 2026-07-18T05:31:52Z

majority of changes and content of this file is the same as ateom-gvisor, why not using shared funcs? is it by purpose?

John Howard (howardjohn) and others added 6 commits July 17, 2026 17:11

Remove the original atenet

9139589

Add ateway-ingress, egress deployment and atunnel component

cb7fcd1

Co-authored-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io> Co-authored-by: npolshakova <nina.polshakova@solo.io>

Revert egress policy

020e723

misc

dcc497d

Use semi-official image

aa97d2c

lint

0049002

John Howard (howardjohn) force-pushed the net/m2-no-egress branch from a1492ed to 0049002 Compare July 18, 2026 00:13

Bowei Du (bowei) reviewed Jul 18, 2026

View reviewed changes

John Howard (howardjohn) mentioned this pull request Jul 18, 2026

Implement Milestone2 Networking (Ingress ONLY) #461

Draft

Lior Lieberman (LiorLieberman) reviewed Jul 18, 2026

View reviewed changes

Uh oh!

Conversation

John Howard (howardjohn) commented Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Testing

Extensibility

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Bowei Du (bowei) commented Jul 18, 2026

Uh oh!

Bowei Du (bowei) commented Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Lior Lieberman (LiorLieberman) left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

John Howard (howardjohn) commented Jul 18, 2026 •

edited

Loading

Bowei Du (bowei) commented Jul 18, 2026 •

edited

Loading

Lior Lieberman (LiorLieberman) left a comment •

edited

Loading