stale-routes-check removes non-cached route table entries

[domdom82]

Describe the bug

• When using the stale-routes-check option, OpenVPN deletes all stale entries from the route table, including permanent routes that were added using iroute entries, e.g. from a client-config-dir file.
• Once all permanent routes have been removed, new IP addresses from any of the iroute routes will no longer be forwarded through the VPN, creating an outage.

To Reproduce

• Set up OpenVPN with mode tun, client-config-dir and stale-routes-check directive
• Let a client connect and put some traffic using one of the iroute routes in the client-config through the VPN
• Wait for the "clean" interval of the stale-routes-check to pass
• Retry the same traffic from above
• Traffic to the IP within the iroute does not work

Expected behavior

• Only cached entries (with a C suffix) should be deleted
• Traffic to the IP within the iroute network should work for all IPs, cached or not

Version information (please complete the following information):

• OS: Ubuntu 26.04 LTS
• OpenVPN version: 2.7 (2.7.0 up to 2.7.3)

Additional context

• Sample openvpn server config:

mode server
tls-server
topology subnet

ca /cacert.pem
cert /cert.pem
key /key.pem
cipher AES-128-GCM
dh /dh2048.pem

max-routes-per-client 256

# Stale after 300s, Remove after 600s
stale-routes-check 300 600

client-config-dir /client-config-dir

dev tun0
server-ipv6 fd8f:6d53:b97a:1::100:0/112

keepalive 10 60
verb 3
proto tcp6-server

status /var/run/vpn-server.status 15
status-version 2

• Sample client-config-dir contents:

# DEFAULT file in /client-config-dir 
iroute 242.0.0.0 255.0.0.0
iroute 243.0.0.0 255.0.0.0
iroute 244.0.0.0 255.0.0.0

• Sample openvpn client config:

client
tls-client
pull

ca /cacert.pem
cert /vpn-client-0/cert.pem
key /vpn-client-0/key.pem

dev tun0
remote 192.168.100.100

keepalive 10 60
verb 3

• Test procedure:

# INITIAL STATUS FILE

root@vpn-server-0:/# cat /var/run/vpn-server.status
TITLE,OpenVPN 2.7.0 aarch64-unknown-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] [DCO]
TIME,2026-06-22 15:16:54,1782141414
HEADER,CLIENT_LIST,Common Name,Real Address,Virtual Address,Virtual IPv6 Address,Bytes Received,Bytes Sent,Connected Since,Connected Since (time_t),Username,Client ID,Peer ID,Data Channel Cipher
CLIENT_LIST,vpn-client-0,tcp6-server:192.168.100.2:59016,,fd8f:6d53:b97a:1::100:2,4547,4805,2026-06-22 15:16:39,1782141399,UNDEF,0,0,AES-256-GCM
HEADER,ROUTING_TABLE,Virtual Address,Common Name,Real Address,Last Ref,Last Ref (time_t)
ROUTING_TABLE,244.0.0.0/8,vpn-client-0,tcp6-server:192.168.100.2:59016,2026-06-22 15:16:39,1782141399
ROUTING_TABLE,243.0.0.0/8,vpn-client-0,tcp6-server:192.168.100.2:59016,2026-06-22 15:16:39,1782141399
ROUTING_TABLE,fd8f:6d53:b97a:1::100:2,vpn-client-0,tcp6-server:192.168.100.2:59016,2026-06-22 15:16:39,1782141399
ROUTING_TABLE,242.0.0.0/8,vpn-client-0,tcp6-server:192.168.100.2:59016,2026-06-22 15:16:39,1782141399
GLOBAL_STATS,Max bcast/mcast queue length,1
GLOBAL_STATS,dco_enabled,0
END


# PING TARGET IN 244.0.0.0/8

root@vpn-server-0:/# ping 244.11.0.2
PING 244.11.0.2 (244.11.0.2) 56(84) bytes of data.
64 bytes from 244.11.0.2: icmp_seq=1 ttl=63 time=6.83 ms
64 bytes from 244.11.0.2: icmp_seq=2 ttl=63 time=0.405 ms
64 bytes from 244.11.0.2: icmp_seq=3 ttl=63 time=0.662 ms
64 bytes from 244.11.0.2: icmp_seq=4 ttl=63 time=0.293 ms
64 bytes from 244.11.0.2: icmp_seq=5 ttl=63 time=0.240 ms
^C
--- 244.11.0.2 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4055ms
rtt min/avg/max/mdev = 0.240/1.685/6.826/2.574 ms

# STATUS FILE NOW CONTAINS A CACHED ENTRY

root@vpn-server-0:/# cat /var/run/vpn-server.status
TITLE,OpenVPN 2.7.0 aarch64-unknown-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] [DCO]
TIME,2026-06-22 15:17:40,1782141460
HEADER,CLIENT_LIST,Common Name,Real Address,Virtual Address,Virtual IPv6 Address,Bytes Received,Bytes Sent,Connected Since,Connected Since (time_t),Username,Client ID,Peer ID,Data Channel Cipher
CLIENT_LIST,vpn-client-0,tcp6-server:192.168.100.2:59016,,fd8f:6d53:b97a:1::100:2,4755,5715,2026-06-22 15:16:39,1782141399,UNDEF,0,0,AES-256-GCM
HEADER,ROUTING_TABLE,Virtual Address,Common Name,Real Address,Last Ref,Last Ref (time_t)
ROUTING_TABLE,244.0.0.0/8,vpn-client-0,tcp6-server:192.168.100.2:59016,2026-06-22 15:16:39,1782141399
ROUTING_TABLE,243.0.0.0/8,vpn-client-0,tcp6-server:192.168.100.2:59016,2026-06-22 15:16:39,1782141399
ROUTING_TABLE,fd8f:6d53:b97a:1::100:2,vpn-client-0,tcp6-server:192.168.100.2:59016,2026-06-22 15:16:39,1782141399
ROUTING_TABLE,244.11.0.2C,vpn-client-0,tcp6-server:192.168.100.2:59016,2026-06-22 15:17:38,1782141458
ROUTING_TABLE,242.0.0.0/8,vpn-client-0,tcp6-server:192.168.100.2:59016,2026-06-22 15:16:39,1782141399
GLOBAL_STATS,Max bcast/mcast queue length,1
GLOBAL_STATS,dco_enabled,0
END

# see ROUTING_TABLE,244.11.0.2C,vpn-client-0,tcp6-server:192.168.100.2:59016,2026-06-22 15:17:38,1782141458


# NOW WAIT FOR 600s AS PER CONFIG FILE

sleep 10m

# RE-CHECK STATUS FILE

cat /var/run/vpn-server.status

TITLE,OpenVPN 2.7.0 aarch64-unknown-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] [DCO]
TIME,2026-06-22 15:28:13,1782142093
HEADER,CLIENT_LIST,Common Name,Real Address,Virtual Address,Virtual IPv6 Address,Bytes Received,Bytes Sent,Connected Since,Connected Since (time_t),Username,Client ID,Peer ID,Data Channel Cipher
CLIENT_LIST,vpn-client-0,tcp6-server:192.168.100.2:59016,,fd8f:6d53:b97a:1::100:2,7699,8787,2026-06-22 15:16:39,1782141399,UNDEF,0,0,AES-256-GCM
HEADER,ROUTING_TABLE,Virtual Address,Common Name,Real Address,Last Ref,Last Ref (time_t)
GLOBAL_STATS,Max bcast/mcast queue length,1
GLOBAL_STATS,dco_enabled,0
END

# ALL ENTRIES ARE GONE - NOW RETRY TO CONNECT VIA PING

root@vpn-server-0:/# ping 244.11.0.2
PING 244.11.0.2 (244.11.0.2) 56(84) bytes of data.
^C
--- 244.11.0.2 ping statistics ---
8 packets transmitted, 0 received, 100% packet loss, time 7146ms


# OUTAGE UNTIL RECONNECT

[schwabe]

Reading the man page it sounds like stale-routes-check and max-routes-per-client was only designed for tap.

[domdom82]

@schwabe thanks for the response. Ironically, we had to increase the max-routes-per-client setting because we got the route quota (256) exceeded message on a tun server.

Regardless of the intention it was designed for, it is used in both tap and tun scenarios and it produces outages because it also deletes static iroute entries.

So IMO there are two options here:

• Fix the static route removal (which I would prefer tbh)
• Ignore max-routes-per-client and stale-routes-check in tun mode and print a warning if used so

[cron2]

Some background. OpenVPN is a truly old project, which has changed hands a couple of times - and some of the options, like --stale-routes-check, are "from back then", and knowledge on "what should this option do, when should it be used, and what are the consequences" sometimes gets lost. Plus, special-case options sometimes do not see the testing they need.

So it's well possible that we broke the interaction between DCO, iroutes and --stale-routes-check (does it behave correctly if you use --disable-dco on the server?), or that it never worked correctly in the first place for tun + iroute setups.

We are currently not exactly well staffed in the project, and while this sounds like an interesting problem to dive into, it might take quite a while to get to it :-(

[domdom82]

@cron2 Thank you for your quick response.

I've tested with disable-dco - same issue still persists.

I will disable the stale-routes-check for now until this is resolved. Without the option, will the route table grow indefinitely until max-routes-per-client is hit? Or is there another algorithm that may remove routes besides the stale check?

[cron2]

Technically, route size is bounded by the size of the networks you route towards the clients - so if you have an iroute for a /24, the route table can not grow beyond 256 (maybe 257?) entries. In your case using /8 iroutes, it could indeed get very big.

Thanks for testing with disable-dco. I assumed that it would be unrelated, but since iroutes are handled very differently when DCO is active it might have been a bug there (and I did not have enough coffee yet).

Thinking more through this, I actually wonder if your server is using DCO at all, and if "use DCO" wouldn't be the correct answer to fix your problem for the time being :-) - without DCO, all packets go through the OpenVPN server process, who does an iroute lookup, populates the internal ip-to-client hash (which is the one that overflows), and then sends the packet to that client. With DCO, data packets never(!) hit the userland server, because everything is set up with kernel routing - that is, iroute config ends up being real ip route entries, and the kernel does not see and would not care about max-routes-per-client anyway.

I see nothing in your server config that would be incompatible with DCO, so maybe you're just missing the necessary kernel module. Copying in our "DCO Linux version number experts" so they can tell you what to do on Ubuntu 26.04 to figure this out - @ralflici @ordex can you say more? Is a special package needed?

[schwabe]

Ubuntu 26.04 has ovpn in the normal kernel. So it should just work. Maybe check your log why dco gets disabled without disable-dco

[domdom82]

The ubuntu version is only used in a Docker setup for this issue. In production we are using a different kernel from GardenLinux

The Docker Ubuntu Linux produces this log entry at the beginning, disabling DCO:

2026-06-22 15:08:26 net_iface_type: type of tun0: tun
2026-06-22 15:08:26 Interface tun0 exists and is non-DCO. Disabling data channel offload
2026-06-22 15:08:26 OpenVPN 2.7.0 aarch64-unknown-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] [DCO]
2026-06-22 15:08:26 library versions: OpenSSL 3.5.5 27 Jan 2026, LZO 2.10
2026-06-22 15:08:26 DCO version: N/A

The tun0 device was created before OpenVPN was started using these commands:

mkdir -p /dev/net && mknod /dev/net/tun c 10 200
openvpn --mktun --dev tun0

In our production workloads, the kernel does not seem to support DCO (we also still use OpenVPN 2.6 there but want to upgrade):

2026-06-23 08:58:55 OpenVPN 2.6.20 x86_64-alpine-linux-musl [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD]

the [DCO] feature is missing there.

[ralflici]

Hi,

There are a few separate requirements here.

On Linux, DCO is provided by a kernel driver. For openvpn 2.7 this is the ovpn driver, and it must be available in the host kernel. This can be from a recent enough kernel, for example Linux 6.16+ or via the backport package (see https://github.com/OpenVPN/ovpn-backports). The openvpn userspace binary also needs to be built with DCO support. Your Ubuntu container log shows [DCO], so that part is present there. The production openvpn 2.6 log does not, so that binary cannot use DCO regardless of the kernel module.

The way tun0 is created in the Docker setup is also not compatible with DCO: mknod only exposes the classic tun character device, and openvpn --mktun --dev tun0 creates a regular TUN netdev. For DCO, openvpn normally creates the interface itself as an ovpn netdev.

If you pre-create an ovpn device manually, that currently does not work reliably either. There is an "already existing DCO interface" branch in userspace, but it appears broken: it does not adopt the existing interface state correctly, and shutdown still deletes the interface despite saying it will not. Also, an ovpn interface needs an extra netlink mode parameter, MP for server mode or P2P for p2p/client mode. Current iproute2 does not expose an ovpn link plugin/argument for that, so a plain ip link add tun0 type ovpn will not create the same device openvpn would create for server mode.

So if the reason for pre-creating tun0 is to keep it alive across openvpn process restarts, I do not think that is currently compatible with Linux DCO. The supported path is to let openvpn create and own the DCO interface.

[domdom82]

Hi @ralflici thanks for your detailed explanation.

Yes, we are precreating the tun devices, because we want them to survive restarts. We also add routes for them, i.e.

# ip route
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0 scope link
242.0.0.0/8 dev tun0
243.0.0.0/8 dev tun0
244.0.0.0/8 dev tun0

As you pointed out, DCO does not seem to be an option for us right now with the GL kernel not supporting it.

IMO OpenVPN should handle route expiration correctly regardless and only remove cached IP / MAC entries, never iroutes.

It may be worth mentioning that we also run OpenVPN in tap mode in different setups, IIRC this would also not support DCO.
The problem is not as pronounced in tap mode because there only the MAC addresses of the VPN clients are cached (which are a fixed number in our case), but not the IPs that are used as targets in the tunnel. (which can become quite a lot in larger clusters with /12 or /16 networks, hence we needed to bump the max-routes-per-client value in tun mode but not tap.)

[cron2]

Of course OpenVPN should not have bugs and do everything correctly :-) - but as I said, we might not find time any time soon to diagnose why userland route expiration does not work properly, and even then, with a /8, you'll end up bumping into max-routes-per-client limits the moment someone fires up a network scan... so following that path, you need someone with time at their hands to delve into old code and figure out what it should do, when (or of) it worked last time, what broke it and how to fix it...

The path "go for DCO" might be quicker - installing the backport ovpn module in the host system is a bit fumbly, but should give the docker environment what it needs... but yes, interfaces need to be created on-demand and routes need to be added using route 242.0.0.0 255.0.0.0 in the openvpn config.

Also, performance with DCO is way better, of course - in terms of throughput, multi-core scalability, etc.

For TAP mode, indeed, DCO is no option (sorry).

[ordex]

I had a deep dive into the stale routes logic and it indeed feels that this was added for TAP, but then other routes were sucked in by accident.
Most notably: iroutes and ifconfig related routes were accidentally added to the stale check processing, while they were never supposed to end up there.
I believe this can be fixed by clearly marking those routes as "permanent" and ensure they are not touched by any caching logic.

[ordex]

I posted here the smallest patch I could come up with (hopefully @cron2 appreciates that ;P):

http://gerrit.openvpn.net/c/openvpn/+/1729

@domdom82 would you be able to test it? (I developed it on master but I expect it to apply on 2.7 too)