tests(proto): Implement real routing infrastructure for proto tests by flub · Pull Request #675 · n0-computer/noq

flub · 2026-05-28T16:30:15Z

Description

This is a whole bunch of new routing infrastructure that aims to allow us to assemble all relevant network configurations for QNT. It currently implements a directly connected interface and an "easy NAT".

The aim is to add a "hard NAT" to this so that this can also be tested.

If this works this should be sufficiently advanced to remove all the previous Routing enum variants and replace them with just the TwoHopRouting.

Breaking Changes

None

Notes & open questions

Opening this as draft, I really need to add a hard NAT to prove that this whole thing works.

I'm pretty unhappy about the AimAdfNat needing an inner that's wrapped in a mutex. It's already given me a deadlock. But that kind of footgun should be limited to developing the SubNetRouters, which should only be a handful.

The cause of this is really that SubNetRouter::assign_ip is needed. I opted for that instead of allowing to pre-configure it because otherwise you'd need more knowledge to create the right networks up front. Now the system does as much as it can.

This also adds the ConnPairBuilder... It's yet another test infra migration stated in this PR, which isn't exactly nice. Because some day we need to finish such migrations. The main issue was that all the existing methods created the Pair and immediately connected them before you could customise the Routing. Before we kind of skirted around this by using the same SocketAddrs in BasicRouting and others, but that's really fragile and disconnected as well.

Change checklist

Self-review.
Documentation updates following the style guide, if relevant.
Tests if relevant.
All breaking changes documented.

This is a whole bunch of new routing infrastructure that aims to allow us to assemble all relevant network configurations for QNT. It currently implements a directly connected interface and an "easy NAT". The aim is to add a "hard NAT" to this so that this can also be tested. If this works this should be sufficiently advanced to remove all the previous Routing enum variants and replace them with just the TwoHopRouting.

github-actions · 2026-05-28T16:32:57Z

Documentation for this PR has been generated and is available at: https://n0-computer.github.io/noq/pr/675/docs/noq/

Last updated: 2026-05-29T09:07:32Z

matheus23 · 2026-05-28T16:35:56Z

If this works this should be sufficiently advanced to remove all the previous Routing enum variants and replace them with just the TwoHopRouting.

Hmmmmm I don't think you'll be able to remove ManyToManyRouting unless you implement proptest shrinking for TwoHopRouting (good luck! :P)

github-actions · 2026-05-28T16:42:26Z

Performance Comparison Report

a5fe9a37c9118002d54d54dfc66ca50ec116d33c - artifacts

Raw Benchmarks (localhost)

Scenario	noq	upstream	Delta	CPU (avg/max)
large-single	5654.7 Mbps	7822.7 Mbps	-27.7%	97.8% / 150.0%
medium-concurrent	5512.5 Mbps	7949.5 Mbps	-30.7%	97.9% / 148.0%
medium-single	3895.2 Mbps	4643.5 Mbps	-16.1%	91.4% / 101.0%
small-concurrent	3811.7 Mbps	5115.0 Mbps	-25.5%	95.7% / 103.0%
small-single	3533.5 Mbps	4646.1 Mbps	-23.9%	96.1% / 149.0%

Netsim Benchmarks (network simulation)

Condition	noq	upstream	Delta
ideal	3087.0 Mbps	4028.7 Mbps	-23.4%
lan	796.4 Mbps	797.3 Mbps	~0%
lossy	69.8 Mbps	55.9 Mbps	+25.0%
wan	83.8 Mbps	83.8 Mbps	~0%

Summary

noq is 24.8% slower on average

---

a5fe9a37c9118002d54d54dfc66ca50ec116d33c - artifacts

Raw Benchmarks (localhost)

Scenario	noq	upstream	Delta	CPU (avg/max)
large-single	5360.2 Mbps	7867.9 Mbps	-31.9%	95.6% / 100.0%
medium-concurrent	5517.5 Mbps	7633.3 Mbps	-27.7%	95.8% / 102.0%
medium-single	3807.3 Mbps	4749.2 Mbps	-19.8%	98.2% / 152.0%
small-concurrent	3793.7 Mbps	5214.9 Mbps	-27.3%	92.6% / 104.0%
small-single	3511.5 Mbps	4732.8 Mbps	-25.8%	94.5% / 104.0%

Netsim Benchmarks (network simulation)

Condition	noq	upstream	Delta
ideal	2946.7 Mbps	4065.0 Mbps	-27.5%
lan	782.4 Mbps	810.4 Mbps	-3.4%
lossy	69.8 Mbps	55.9 Mbps	+25.0%
wan	83.8 Mbps	83.8 Mbps	~0%

Summary

noq is 26.5% slower on average

---

cc66f4415146284c14f2143a0398f119c2ab8e0e - artifacts

Raw Benchmarks (localhost)

Scenario	noq	upstream	Delta	CPU (avg/max)
large-single	5369.8 Mbps	7949.9 Mbps	-32.5%	97.3% / 146.0%
medium-concurrent	5499.0 Mbps	7883.2 Mbps	-30.2%	91.8% / 96.9%
medium-single	3614.3 Mbps	4654.6 Mbps	-22.3%	89.6% / 97.7%
small-concurrent	3771.9 Mbps	5413.7 Mbps	-30.3%	95.6% / 102.0%
small-single	3520.1 Mbps	4788.3 Mbps	-26.5%	94.7% / 102.0%

Netsim Benchmarks (network simulation)

Condition	noq	upstream	Delta
ideal	3097.3 Mbps	4083.5 Mbps	-24.2%
lan	782.4 Mbps	810.3 Mbps	-3.4%
lossy	69.8 Mbps	55.9 Mbps	+25.0%
wan	83.8 Mbps	83.8 Mbps	~0%

Summary

noq is 27.8% slower on average

divagant-martian · 2026-05-29T03:02:13Z

+    /// to be sent on the network that contains said src_ip or not at all. If there is only
+    /// a destination IP it needs to be sent on the network that contains said destination
+    /// IP or not at all.
+    fn route_transmit(&mut self, source: Side, transmit: &Transmit) -> RoutingDecision {


I'm probably lacking context because of my time away, but is the purpose here to mimic routing tables selection? my understanding is that we first find the routing tableS that match the source if given (which might be more than one), and find one with a route to the destination, and the first might not be the one where this match is found. So the code above doesn't really try to match both, which is surprising to me. If this is intended, what's the purpose?

Coming back to this after reading more parts of the pr. This might happen if we are part of two nats simultaneously, but this is a case far too convoluted for now. Leaving the comment up anyway if there's any interesting discussion around it

I think you are right and this implementation is overly naive. It should indeed keep looking for other source interfaces that might match and try if those have a route to the destination.

Funny that you mention the two NATs. I do want this to be able to simulate all environments we want to deal with. I ended up with the simplification that we can always emulate what is needed with only two hops. E.g. if you have a CG-NAT (hard NAT) followed by a home NAT (easy NAT) you only need to emulate the hard NAT really.

But one thing that is definitely missing (and was there in an earlier attempt) was that you need to be able to have E.g. several NAT interfaces on the client side and server side that all route to each other, to emulate e.g. having a wifi and mobile data connection. And than being able to dynamically add and remove them and break connections to make for good proptests.

divagant-martian · 2026-05-29T04:13:45Z

+    }
+}
+
+/// A single network with no firewalls.


From what I'm gathering this is something you want SubnetRouters to do, but this structure itself can't guarantee that

Maybe I need to improve this description. I mean to say that this network itself will forward any packet sent by a SubNetRouter to any recipient SubNetRouter as long as both are attached to this network. So the network itself does not have any firewalls. What the SubNetRouter internally implements is up to itself. It can pretend to be anything.

Co-authored-by: Diva Martínez <26765164+divagant-martian@users.noreply.github.com> Co-authored-by: Philipp Krüger <philipp.krueger1@gmail.com>

matheus23 · 2026-05-29T14:19:14Z

+    let server_nat = EimAdfNat::new_v4_server();
+    let client_nat = EimAdfNat::new_v4_client();
+    let net_nat = TwoHopNetwork::new("2::/64".parse()?, server_nat.clone(), client_nat.clone());
+    let routes = TwoHopRouting::from_iter([net_public, net_nat]);
+
+    let mut pair = ConnPairBuilder::default()
+        .with_transport_cfg(transport_cfg)
+        .with_routes(routes.into())
+        .connect();
+
+    info!("adding addrs");
+    pair.add_nat_traversal_address(Server, server_nat.qad_addr())?;
+    pair.add_nat_traversal_address(Client, client_nat.qad_addr())?;


So this is the reason you have a Mutex in EimAdfNat.
I went ahead and autocoded a refactor that removes the Mutex. Necessarily it makes this piece of the code slightly less beautiful, but perhaps worth discussing, would love to hear your thoughts: #679

flub

oh look, github has been hiding my comments behind pending again

flub · 2026-05-29T09:02:20Z

+    }
+}
+
+/// A single network with no firewalls.


Maybe I need to improve this description. I mean to say that this network itself will forward any packet sent by a SubNetRouter to any recipient SubNetRouter as long as both are attached to this network. So the network itself does not have any firewalls. What the SubNetRouter internally implements is up to itself. It can pretend to be anything.

flub · 2026-05-29T09:06:55Z

+        server.assign_ip(server_router);
+        client.assign_ip(client_router);


I don't understand this comment, you mean in the assing_ip implementations? The AimAdfNat::assign_ip uses this.

flub requested review from divagant-martian and matheus23 May 28, 2026 16:30

matheus23 reviewed May 28, 2026

View reviewed changes

Comment thread noq-proto/src/tests/util.rs Outdated

n0bot Bot added this to iroh May 28, 2026

github-project-automation Bot moved this to 🚑 Needs Triage in iroh May 28, 2026

divagant-martian reviewed May 29, 2026

View reviewed changes

flub closed this May 29, 2026

github-project-automation Bot moved this from 🚑 Needs Triage to ✅ Done in iroh May 29, 2026

flub reopened this May 29, 2026

github-project-automation Bot moved this from ✅ Done to 🚑 Needs Triage in iroh May 29, 2026

Apply suggestions from code review

cc66f44

Co-authored-by: Diva Martínez <26765164+divagant-martian@users.noreply.github.com> Co-authored-by: Philipp Krüger <philipp.krueger1@gmail.com>

matheus23 reviewed May 29, 2026

View reviewed changes

Comment thread noq-proto/src/tests/util.rs

matheus23 reviewed May 29, 2026

View reviewed changes

matheus23 moved this from 🚑 Needs Triage to 🏗 In progress in iroh May 29, 2026

flub mentioned this pull request May 29, 2026

refactor(tests): Introduce ConnPairBuilder #680

Open

2 tasks

flub commented May 29, 2026

View reviewed changes

		server.assign_ip(server_router);
		client.assign_ip(client_router);

Conversation

flub commented May 28, 2026

Description

Breaking Changes

Notes & open questions

Change checklist

Uh oh!

github-actions Bot commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

matheus23 commented May 28, 2026

Uh oh!

Uh oh!

github-actions Bot commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Performance Comparison Report

Raw Benchmarks (localhost)

Netsim Benchmarks (network simulation)

Summary

Raw Benchmarks (localhost)

Netsim Benchmarks (network simulation)

Summary

Raw Benchmarks (localhost)

Netsim Benchmarks (network simulation)

Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

divagant-martian May 29, 2026

Choose a reason for hiding this comment

Uh oh!

flub May 29, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

divagant-martian May 29, 2026

Choose a reason for hiding this comment

Uh oh!

flub May 29, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

matheus23 May 29, 2026

Choose a reason for hiding this comment

Uh oh!

flub left a comment

Choose a reason for hiding this comment

Uh oh!

flub May 29, 2026

Choose a reason for hiding this comment

Uh oh!

flub May 29, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented May 28, 2026 •

edited

Loading

github-actions Bot commented May 28, 2026 •

edited

Loading