Building the world's fastest VPN using what you have

Written by Leon Dang

Hardware is no longer cheap, and for most developers it has stopped getting faster. That is a real loss: we can no longer validate as thoroughly or run as many simultaneous tests as we used to. A large amount of effort to develop software for Cosinium is therefore tied to the limited equipment that we and others have. Why it matters?

  • Ignoring GPU prices, DRAM has risen more than 450% this year alone (↗), NVMe more than doubled in price (↗) and even hard drives rose more than 100% (↗).
  • Older equipment held on longer with over 20% reduction in new computer units shipped.
  • Software needs to improve because of the lack of available faster hardware.

Overview

Our QuicN VPN protocol, even with all its security features enabled, is measurably the fastest (footnote) and created with computer equipment that we hadn’t upgraded since long before the price rises. In this article, I provide some background on how QuicN evolved to become what it is and the creation of an alternative data channel offload driver (udco) for OpenVPN, to demonstrate why Cosinium’s software suite is so focussed on being small and performant so that it can run on what you already have.

Results

In a container based connection on our Ryzen 5 5600 testbed, QuicN throughput hovers around 18.5Gbps, which is above every other VPN tested.

Tunnel Gbit/s
QuicN 18.5
OpenVPN + udco 17
WireGuard (Go) 12.4
WireGuard (kernel) 2.28

In a small virtual machine setup with hardware network interfaces, hosted by ESXi on a Ryzen 9 5900X, it stays at the top alongside our udco code:

Tunnel Gbit/s
OpenVPN + udco 12.0 – 12.8
QuicN 11.85 – 12.25
WireGuard (kernel) 7.4 – 9.4
OpenVPN 2.7.7 DCO 4.2 – 4.4
OpenVPN userspace 0.1 – 1.3
IPsec ESP 3.2

Test setup

The testbed system is an AMD Ryzen 5 5600 with 6 cores and each has a limit of around 30Gbps AES crypto when operating at 1500 byte packets - that number is only reached if you’re doing nothing else other than directly passing memory into the encryption/decryption stream; that doesn’t happen with networking workflows.

Why efficiency matters

Now that we’re hit with a global crisis of limited hardware choices due to astonishingly high inflation never seen before, we have to start thinking more carefully about how we can make the best use of what we have. I remember the limitations that were put on programmers during the early years of personal computing. What they achieved with what they had was nothing short of amazing. The Amiga A500 had a minuscule 512KB of memory and yet had the ability to run a full-fledged graphical Workbench user interface, colours and sound. id Software brought 3D gaming to life with Wolfenstein 3D running on 640KB PCs.

After seeing PCs loaded with 4GB then 8GB, we started seeing so much more software, but the “how did you do so much with so little?” wow moments became rarer. Applications got huge but didn’t seem to do much more, AAA games started requiring an abundance of everything… it just seemed like an unending series of overloading. Flashy but forgetful. There are exceptions - video compression algorithms, retro style game creators, demo scenes, and Nintendo first-party titles.

I’ve been developing programs and operating systems for a while now, and in all that time, my primary motivation is how much more can I pull out of the system without needing more memory, CPU, or storage. Not everyone can afford the best hardware, so making what they already have workable lets them keep it longer.

Designing QuicN

When it came time to create QuicN, I wanted it to be the fastest implementation that could be done whether it runs on bare metal, as a virtual machine or as a container on an operating system environment that cannot be changed. Two benchmarks it must perform well in: low memory, high throughput. It is written in C# to help with memory safety and its mature native AOT compiler provides the needed performance.

Where QuicN fits

Apart from using QuicN in the Cosineq mesh VPN for direct peer-to-peer access, it is part of Cosinium’s architecture to provide security enforcement for AI workloads efficiently, especially when they use externally hosted LLM API providers. From the user’s perspective, any driver that gives them access should be out of the way as much as possible, and therefore efficiency is necessary since the hardware hosting their workspaces might not be the latest or fastest.

Staff devices and Cosinium nodes each connect over QuicN to the Cosineq mesh at the centre. Each Cosinium node runs Agent Gate in front of its agent workspaces, either several to one node or one node per workspace. In the node shown in detail, the agent's LLM and MCP calls leave the workspace only through Agent Gate.
Cosinium Agent Gate in front of each agent workspace, shared or dedicated, as a Cosineq mesh node running QuicN

QuicN has the following security controls:

  • Post-quantum key negotiation: while quantum computers are not here yet, having PQ cryptography protects captured packets that are sent across the network from being decrypted in the future.
  • Fine grained access control that tracks who and what gets to connect to another system, and what they can access not only on that system but anything else networked to it.
  • Data plane implemented in userspace so that you can run it on existing hosts without having to recompile or install a new kernel, and the only update needed is with the QuicN programs. At the same time every packet has to go through the kernel’s firewall intact, just like every other traffic that is sent.
  • Rootless tunnel support, which starts without administrator privileges so it acts as a proxy running as the user who started the VPN service. Every received packet must therefore be inspected against its sender’s entitlements before being delivered.

Tuning QuicN

QuicN’s performance on Linux today of touching 19Gbps with a single crypto worker for each transmission direction surpasses anything I’ve benchmarked on my testbed. The process took a few months to reach this point and required a lot of profiling and algorithm tuning. QuicN started as a proof of concept to ensure that post-quantum QUIC keying for the data channel worked and that traffic would pass correctly - it got 1.5Gbps. I then applied basic sendmmsg/recvmmsg calls which achieved 3Gbps. After changing memory usage behaviour and applying dedicated worker threads, it reached 6Gbps, but it became a hard barrier to break after that. Questions came as to why it’s being blocked from doing more. It uses libmsquic, which chooses to own the socket and I needed to remove that limitation because while MS-QUIC is one of the top performing libraries, it’s built for many short HTTP connections and not long-lived VPN tunnels. Prying it out of MS-QUIC and making the C# side own the socket allowed me to optimise the code loops and packet pipelines into the crypto workers. That moved the needle to 10Gbps. Adding io_uring pushed it past 12Gbps. I was ok with that number given that it only used 90MB of memory and didn’t overconsume its CPU budget.

OpenVPN and udco

A few months later…

Because Cosinium supports OpenVPN, I decided to see what its real numbers were - since I have OpenVPN serving on my NanoPi R5C as a backup connection. My disappointment was pretty clear; only 60Mbps or thereabouts with userspace data plane and 170Mbps with kernel DCO. That was too weak compared to WireGuard (200s Mbps up, 500s Mbps down) and QuicN at 450-500 Mbps both directions.

The first experiment was getting QuicN to use Intel’s IPsec-MB for the AES data plane crypto… 14Gbps. That was my first “oh cool” in a while.

From there, I decided to create a pure userspace DCO tunnel (udco) which implements the DCO-like RPCs over a Unix socket to keep it separate from the openvpn daemon. I had worked on DCO performance improvements in the past: in FreeBSD’s kernel and for OpenVPN’s official Windows DCO driver. I wrote udco in Odin-lang; it’s a small modern language which interfaces with C-libraries easily, and uses LLVM build tools. Using QuicN packet processing workflow as a model, udco immediately pumped through 16Gbps, with regular 17Gbps peaks; it was my WOW moment… 30MB, 1 RX and 1 TX crypto thread and the essential socket readers/writers. It felt pure and without excess.

OpenVPN with the standard kernel DCO: the daemon in userspace controls the DCO driver in the kernel through system calls, and the driver moves data between the network and the ovpn interface.
OpenVPN with the standard kernel DCO
OpenVPN with udco: the daemon talks to udco over a Unix socket, and udco, running in userspace, moves data between the network and the ovpn-tun interface.
OpenVPN with udco, the userspace DCO

So why does the userspace version beat the kernel one? On the same ESXi virtual machine the kernel DCO managed 4.4Gbps while udco reached 12.8Gbps. Linux’s ovpn module encrypts and decrypts one packet at a time and the whole path relies on a perfectly streamlined flow from a packet arriving on the tunnel or network interface to it being scheduled for work in the kernel’s stack. The idea of keeping the data path out of the daemon is sound, especially when crypto hardware is available; however, this implementation of it is inefficient. udco takes the opposite approach, built from what QuicN had already proven using batched socket calls, dedicated crypto threads, integrating Intel’s IPsec-MB, and no dynamic memory allocation. Doing it in userspace benefits the development cycle: quick to try, profile and tune.

Back to QuicN

My algorithms were working, but I wanted QuicN to be faster. So a day later, I worked on flamegraph profiling and refactoring the code. QuicN does a lot more than the other protocols: it needs to enforce mesh ACLs, handle inline STUN and relay requests, address masquerading, local DNS and also support root-less servicing. Even with those things active, it should be able to do better. Initially I thought it was a limitation of C#.NET, but it wasn’t because I had reduced all dynamic memory allocation to only small control plane messages. The refactoring eliminated two threads… but it was stuck at 15Gbps with TCP retransmissions. Something was still blocking it, so onto more profiling. The final moment was realising that due to my too restrictive use of memory, the job processing buffer ring was insufficient for dealing with the volume of packets. When I raised it to 4096 entries from 1024, memory usage increased to 110MB from 90MB, but that pushed throughput to 19Gbps and with reduced CPU usage to boot; an extra 20MB was a small compromise I am willing to take.

Conclusion

I’m glad I got QuicN to this stage with what I had, and as a bonus created the fastest DCO implementation (it is now more practical to use on my NanoPi R5C with the ability to drive close to 800Mbps download and 550Mbps upload). I don’t have the latest CPU or the fastest ARM SBC, but this limitation is what drove me to find alternative means. My priority with Cosinium is making sure that anyone who uses our products is getting something that is well tuned, irrespective of what system they run it on.

Footnote

By “fastest” I mean a VPN that runs alongside the kernel network stack, the way all ordinary Linux traffic does. There are VPN implementations that use polled-mode stacks such as DPDK, which are meant for routing. Their problem is that host programs cannot directly interact with them and the kernel’s firewall is not involved in their data plane, because they have their own tight-looped network stack and own the network interfaces instead of the host OS. They might have a socket path that system programs can send traffic over, but it is generally very slow and only meant as a control plane.