Proxy benchmarks

​

Results

1. Envoy didn’t resume TLS sessions in the harness’s check, so its new connections are full handshakes; the other five resumed.

• nginx and HAProxy forward keep-alive HTTP about 2.3 times as fast as dnz-proxy and use less than half its CPU per request.
• dnz-proxy comes third on keep-alive HTTP and HTTPS, ahead of Traefik, Envoy and Caddy. It sets up new TLS connections faster than any of the others (rustls with ring, against OpenSSL in nginx and HAProxy), and keeps up with nginx on 64 KiB answers.
• Memory: dnz-proxy holds the least, 2 MiB idle and 8 MiB under load, against 21 to 39 MiB for the others.
• Latency: nginx and HAProxy answer in about 1 ms, dnz-proxy in 3.5 ms (p99 6.4 ms); the Go proxies (Caddy, Traefik) have p99s of 16 to 17 ms.

​

After 1.7.0: one write per answer, fewer allocations

• Each answer’s header and body went out in two writes. Pingora gives a connection a write buffer of one TCP segment only in its own listeners; connections from dnz-proxy’s listener had none, so a 1 KiB answer took two packets, each a full trip through the TCP stack. Now they leave in one. Event streams and WebSockets aren’t held back: each chunk and each message on an upgraded connection is still sent right away.
• Every visit built a whole request log line (a dozen allocations, a GeoIP lookup), though only Logged per second, per node of them are kept. The traffic counters now take what they need from the visit, and a line is made only when it is kept.

​

Method

• Each proxy runs in its own container on one Docker network, started on its own, and forwards to the same origin: nginx serving two static files (1 KiB and 64 KiB) on its own CPU.
• CPUs are pinned (cpuset), not limited by a quota, so each proxy sizes its worker threads to the one CPU it may use, as it would on a one-CPU server. Envoy, which counts the host’s CPUs, gets --concurrency 1.
• dnz-proxy joins a DNZ that pushes it the routers, the service and the certificate over its configuration stream, as in production. It runs with its defaults: analytics and request logs on. The DNZ control plane shares the origin’s CPU; no traffic goes through it.
• The others run the official images with a minimal configuration (in proxybench/conf/): one route to the origin, upstream keep-alive, the usual X-Forwarded-* headers, nginx’s access log off (the others don’t log requests by default).
• TLS is TLS 1.3 with the same ECDSA P-256 certificate everywhere. Each proxy keeps its default session resumption; wrk resumes sessions on new connections, so the new-connection column is mostly TCP setup and resumed handshakes.
• Load: wrk with 2 threads and 64 connections for the requests per second (HTTP/1.1; Connection: close for the new-connection case), and hey for the fixed rate (50 workers of 100 requests a second each).
• CPU per request is the proxy’s CPU time (its cgroup’s cpu.stat) during the fixed-rate run divided by the requests, so it includes whatever the proxy does in the background. Memory is the cgroup’s anonymous memory, idle after start and right after the load; it leaves out the binary itself.

​

Run it yourself

curl -fsSL https://github.com/terrific-ci/dnz/archive/refs/heads/main.tar.gz \
  | tar -xz --strip-components=1 dnz-main/proxybench
cd proxybench
./run.sh

BENCH_DURATION=60 BENCH_RUNS=5 BENCH_PROXIES="dnzproxy nginx" ./run.sh