Skip to main content
An upstream is where a route sends its visits: a set of servers, how visits are shared between them, and how their health is checked. Proxy → Upstreams holds one list for every proxy. A route on several proxies uses the same upstream on each, so a change to an upstream reaches every route that uses it, on every proxy.

The list

Each upstream shows its servers, its balancing (and whether visits are sticky), its health check and what uses it: routes and the proxies they are on, other upstreams that send visits to it, and error pages that ask it for their pages. Servers taken from a DNS name say so, for example 3 servers from swarm-nodes.example.local, 2 min ago, or that they are stale while lookups fail (see Servers from a DNS name).
  • Filter by kind of traffic: HTTP, TCP or UDP.
  • Hide routes’ own leaves out the upstreams made from a route’s servers (see below).
  • Not used keeps only the upstreams nothing uses.
  • Search by name, server or route.
  • The list comes a page at a time (20 by default). The filters and the search run on the server, so they look through every upstream, and the counts beside the filters are of all of them. The dialogs that pick an upstream still offer every one.

Server health

Each server has a dot showing its health as the DNZ Proxy nodes see it, from their reports of the last 30 seconds: Hover a server to see which nodes find it down and why (connection refused, for example). An upstream with servers down says how many, such as 2 of 9 servers down, or 1 of 3 servers down on some nodes when other nodes still reach it. Servers that are down come first, so the two servers the list shows always include a failing one. + N more (or, for an upstream of two servers, its servers down line) opens a list of every server with its dot, and under each server that is down, which node finds it down and why. Long lists scroll inside it. The upstream’s dialog shows the same dot after each server’s address; tap it for the nodes and the reason. For servers from a DNS name, the dialog lists the servers found now, each with its dot. A server no DNZ Proxy node checks (an upstream used only by Traefik or Caddy, or not used at all) has no dot: only DNZ Proxy nodes report health. A node’s own page shows the node only; the health of the servers is here.

Health check times

Next to each server’s dot is how long its last health check took, such as 42 ms: the time from the check’s start to the server’s answer, connecting included. With several nodes it is the slowest of the nodes that find the server up. It turns amber when slow: 500 ms or more, or a quarter of the check’s timeout when that is shorter (250 ms for a 1 s timeout). The + N more list and the dot’s popover in the dialog add a line per node with its last time and the range of its last 20 passing checks (node-1 (Edge): 42 ms · last 20: 38–55 ms, avg 44 ms). A server that is down says how long its check took to fail, such as timed out after 5 s or connection refused after 0.6 ms. Nodes older than DNZ v2.1 report no times, and their servers show none.
Prometheus has the time as dnz_proxy_upstream_check_seconds{upstream,server} (see Prometheus).

Adding and editing

Add upstream opens the upstream dialog:
  1. Kind of traffic: HTTP, TCP or UDP. An upstream keeps its kind.
  2. What it is (HTTP):
    • Load balancer: servers you list, sharing the visits.
    • Split: other upstreams by weight, for example 9 and 1 for a canary release.
    • Mirror: one upstream answers; a share of the visits is copied to others, whose answers are dropped.
    • Fallback: another upstream while this one is down (or answers with the status codes you list).
  3. Name and description. The name is the service name every proxy knows the upstream by.
  4. Servers from (a load balancer):
    • Listed here: servers with a weight (HTTP), or host:port addresses (TCP, optionally with TLS to the server; UDP). For HTTP, type each server as address:port (10.0.0.10:8080): its scheme badge (http:// at first) picks http or https for that server, and with https whether its certificate is checked (Skip TLS verification, per server on DNZ Proxy only). Its heart button, before the weight, sets its health check. A full URL typed or pasted sets the row’s scheme, and Enter adds the next server. See Typing the servers, Certificate checks per server and Health checks per server.
    • Every address of a DNS name: DNZ looks the name up with its own DNS (its zones first) and keeps one server per address, such as every Docker Swarm node behind its routing mesh.
    • Docker Swarm tasks (HTTP): tasks.<service>, which each DNZ Proxy node and Caddy look up themselves inside the service’s overlay network. Refused on Traefik.
    Test looks the name up now, and the guides write the record set or the Swarm network set-up for you. See Servers from a DNS name.
  5. Balancing: weighted round robin, power of two choices, highest random weight or least time, and sticky sessions (a cookie keeps each visitor on the server it first reached).
  6. Health check: a path and how often to ask it; a server that doesn’t answer well gets no visits. On DNZ Proxy a server can have a path of its own, or none.
  7. Connection (folded): pass the visitor’s Host header on, keep the server URLs’ paths, and how every proxy connects to the servers: server name, trusted CAs, a client certificate, timeouts and, for TCP, the PROXY protocol. See Connection settings.
Editing an upstream that routes use says how many routes on which proxies the change reaches. Rename follows the new name everywhere the upstream is named, on every proxy. Delete is refused while anything uses the upstream, and says what does.

A route’s own upstream

In the route dialog, Upstream is either Servers of its own or An upstream from the list. Servers typed there make the route’s own upstream, listed as For app.example.com: From the upstream dialog, a route’s own upstream can be made shared: it then stays when its route goes, and other routes can pick it.

Merging

Merge… points every route, upstream and error page that uses the upstreams picked at the one kept, and deletes the others. When two upstreams have the same servers but other settings (one with a health check, one without, for example), the list suggests merging them and keeps the name someone typed rather than one made from a hostname.

Connection settings

Since DNZ v2 an upstream holds how the proxies connect to its servers, the same on every proxy that uses it: the upstream dialog’s Connection (folded). Settings only Traefik runs are marked Traefik only. HTTP upstreams:
  • The server name (SNI) to send and check the certificate for.
  • Trusted CAs: PEM text, or paths that must exist on every proxy’s machine. Once set, they replace the public CAs.
  • Client certificates for mTLS: the certificate and its key as PEM. DNZ keeps the key and never shows it again; saving without a key keeps the one DNZ has.
  • The expected certificate URI and SPIFFE, TLS versions and ciphers.
  • Idle connections, HTTP/2 and timeouts.
TCP upstreams: connect timeout, keep-alive, termination delay, the PROXY protocol (so the servers see the client’s address), and TLS to the servers (server name, CAs, client certificates). What each proxy kind does with them:

Certificate checks per server

Whether an HTTPS server’s certificate is checked is each server’s own setting, its scheme badge’s Skip TLS verification (skip_tls_verify on the server in the API and in proxy YAML). Until v2 one accept any certificate switch covered all of an upstream’s servers, so an upstream couldn’t mix a server with a certificate worth checking and one without. On upgrade, every HTTPS server of an upstream that accepted any certificate (the default) keeps doing so. Skipping the check per server is DNZ Proxy only. Traefik and Caddy have one certificate setting per upstream, and the servers decide it: So on Traefik and Caddy, a server that skips the check in a mixed upstream needs a valid certificate. The server editor marks the switch DNZ Proxy and says so when Traefik or Caddy routes use the upstream. From the command line, write such a server https+insecure://10.0.0.5:8443; --scheme https+insecure does the same for the servers of a DNS name.

Health checks per server

The upstream’s health check (path and interval) applies to all its servers. On DNZ Proxy a server can have its own: the heart button of its row in Servers (between the address and the weight) picks Upstream’s path (the default), A path of its own, or Not checked (it then counts as up). The heart shows the choice: muted at the upstream’s path, highlighted with a path of its own, struck through when not checked. It is the same in the upstream dialog, the route dialog and the record dialog’s route, which also set the route’s own upstream’s path and interval under the servers. A path of its own is asked at the upstream’s interval, with the upstream’s other health-check settings. The upstream dialog’s Health check → Per server (marked DNZ Proxy) lists every server’s check as an overview and can change it too. While no server differs, it is folded behind Different path per server…; when some do, it opens by itself, and folded it still names them (10.0.5.12:8080 · /status). In the API and proxy YAML it is health_check_path on the server (off for no check). From the command line, put it after the URL and weight: http://10.0.0.5:8080=3?health=/status, or ?health=off. Uptime service monitors and proxy alerts follow each server’s own check, as the nodes report it.

From per-proxy transports

Before v2 an upstream named a servers transport, and each proxy had its own transport by that name under its Settings. Upgrading to v2 moves each transport into the upstream that names it, client certificates copied in as PEM:
  • Where proxies connected to an upstream differently (another client certificate on another proxy, or no transport there), the upstream is split so each proxy keeps connecting as it did. The way most routes use keeps the upstream; each other way gets a copy named <upstream>-<proxy>, which those proxies’ routes use.
  • Transports nothing named did nothing and are dropped (the audit log and the backup keep them).
  • Caddy proxies start applying an upstream’s connection settings, which they ignored before.
dnz proxy advanced-check lists the splits and the Caddy proxies affected; run it on a copy of the database before upgrading (see Checking a proxy setup and From per-proxy middlewares).

TCP and UDP routes

A route can be TCP or UDP instead of HTTP: pick it with the option cards at the top of Add route (not on Caddy, which routes HTTP only). It listens on an entry point of the proxy (DNZ Proxy’s TCP and UDP entry points, or a Traefik’s: UDP ones have /udp addresses) and sends connections to a TCP or UDP upstream, from the list or made of server addresses typed in the dialog.
  • TLS (TCP): none (every connection on the port), ended by the proxy (it holds the certificate, the servers get plain TCP) or passed through (the servers end TLS; the proxy reads only the hostname). With TLS, a route can take only some hostnames (SNI); their records can go through the proxy like an HTTP route’s.
  • Only from these addresses (TCP): client addresses or ranges.
  • Priority (TCP): a higher one wins when routes overlap.
  • Middlewares (TCP): address lists and connection limits (at most this many open connections per client address) from Proxy → Middlewares, shared by every proxy’s TCP routes. New TCP middleware in the dialog makes one and picks it.
They are listed with the HTTP routes, marked TCP pg or UDP dns (protocol and entry point), with a TCP route’s middlewares. TCP and UDP routes are in the proxy’s YAML and in route history too.

Coming from per-proxy services

Before upstreams, each proxy kept services of its own, and a route on two proxies had a copy on each. Upgrading moves them into upstreams once:
  • Services identical in content (servers and weights, balancing, sticky sessions, health check and the rest) become one upstream, whatever their names. It takes the name someone typed rather than one made from a hostname, else the one most routes use.
  • The same name with other servers stays apart: the later one is named after its proxy, for example swarm-service-edge2.
  • A redirect route’s empty service becomes no upstream. Services nothing used become upstreams shown as Not used; none is deleted.
  • What visitors get doesn’t change. Sticky sessions keep the cookie name of the service whose name and settings the upstream took.
Proxy → Upstreams says once what the move did, and the audit log keeps it.
DNZ copies a SQLite database (and the Redis backend’s local copy) to backups/before-upstreams-<time>.db next to it before the move. With Postgres or MySQL, take a dump before upgrading. From v2.0.0 on, the old per-proxy services are gone (DNZ makes another copy, backups/before-cleanup-<time>.db, before dropping them), so going back to a version before v1.27 needs that backup. See Upgrading.

Checking a proxy setup

The proxy page has no Advanced tab any more: everything it held has a form (routes with their Matching, upstreams with their connection settings, middlewares and TCP middlewares under Proxy → Middlewares, and TLS under the proxy’s Settings). A few things are changed elsewhere: a route with TLS or observability options of its router (in the proxy’s YAML), and a TCP route with a rule of Traefik’s own (through the API). dnz proxy advanced-check lists them, per proxy; each proxy’s Settings shows its own. Its v2 part says what upgrading to DNZ v2 does to the database: see From per-proxy middlewares. The check only reads, and it works on a copy of the database too: run it with the new version against a copy of the current one before upgrading, without touching the running server.
Copy the database out of the running container (with its -wal file, which holds the newest changes), then check it with the new image:
After the upgrade, the same check runs inside the container:
The copy is opened read-only and never migrated. On a read-only folder, SQLite can’t read the -wal file and the check reads the main file alone (it says so); copy into a folder you can write to, or use a backup from Settings → Backups.

From the command line

See Proxy commands for every option.