> ## Documentation Index > Fetch the complete documentation index at: https://docs.dnzwithazee.com/llms.txt > Use this file to discover all available pages before exploring further. # Upstreams > Where proxies send visits: one list for every proxy, with servers, balancing, sticky sessions, health checks and how proxies connect to the servers. An **upstream** is where a route sends its visits: a set of servers, how visits are shared between them, and how their health is checked. **Proxy → Upstreams** holds one list for every proxy. A route on several proxies uses the same upstream on each, so a change to an upstream reaches every route that uses it, on every proxy. ## The list Each upstream shows its servers, its balancing (and whether visits are sticky), its health check and what uses it: routes and the proxies they are on, other upstreams that send visits to it, and error pages that ask it for their pages. Servers taken from a DNS name say so, for example *3 servers from swarm-nodes.example.local, 2 min ago*, or that they are stale while lookups fail (see [Servers from a DNS name](https://docs.dnzwithazee.com/proxy/upstream-discovery.md)). * Filter by kind of traffic: **HTTP**, **TCP** or **UDP**. * **Hide routes' own** leaves out the upstreams made from a route's servers (see below). * **Not used** keeps only the upstreams nothing uses. * Search by name, server or route. * The list comes a page at a time (20 by default). The filters and the search run on the server, so they look through every upstream, and the counts beside the filters are of all of them. The dialogs that pick an upstream still offer every one. ### Server health Each server has a dot showing its health as the DNZ Proxy nodes see it, from their reports of the last 30 seconds: | Dot | Meaning | | - | - | | Green | Every node finds the server up | | Amber | Some nodes find it down | | Red | No node finds it up | Hover a server to see which nodes find it down and why (*connection refused*, for example). An upstream with servers down says how many, such as *2 of 9 servers down*, or *1 of 3 servers down on some nodes* when other nodes still reach it. Servers that are down come first, so the two servers the list shows always include a failing one. **+ N more** (or, for an upstream of two servers, its *servers down* line) opens a list of every server with its dot, and under each server that is down, which node finds it down and why. Long lists scroll inside it. The upstream's dialog shows the same dot after each server's address; tap it for the nodes and the reason. For servers from a DNS name, the dialog lists the servers found now, each with its dot. A server no DNZ Proxy node checks (an upstream used only by Traefik or Caddy, or not used at all) has no dot: only DNZ Proxy nodes report health. A node's own page shows the node only; the health of the servers is here. ### Health check times Next to each server's dot is how long its last health check took, such as *42 ms*: the time from the check's start to the server's answer, connecting included. With several nodes it is the slowest of the nodes that find the server up. It turns **amber when slow**: 500 ms or more, or a quarter of the check's timeout when that is shorter (250 ms for a 1 s timeout). The **+ N more** list and the dot's popover in the dialog add a line per node with its last time and the range of its last 20 passing checks (*node-1 (Edge): 42 ms · last 20: 38–55 ms, avg 44 ms*). A server that is down says how long its check took to fail, such as *timed out after 5 s* or *connection refused after 0.6 ms*. Nodes older than DNZ v2.1 report no times, and their servers show none. ```bash theme={null} dnz proxy upstreams show swarm-service # each server: up on 3 of 3 nodes and the check's time, a line per node, # or down on which node and why ("timed out after 5 s") dnz proxy routes list # servers marked "(42 ms)", "(slow: 812 ms)" or "(down)" ``` Prometheus has the time as `dnz_proxy_upstream_check_seconds{upstream,server}` (see [Prometheus](https://docs.dnzwithazee.com/dashboard/prometheus.md)). ## Adding and editing **Add upstream** opens the upstream dialog: 1. **Kind of traffic**: HTTP, TCP or UDP. An upstream keeps its kind. 2. **What it is** (HTTP): * **Load balancer**: servers you list, sharing the visits. * **Split**: other upstreams by weight, for example 9 and 1 for a canary release. * **Mirror**: one upstream answers; a share of the visits is copied to others, whose answers are dropped. * **Fallback**: another upstream while this one is down (or answers with the status codes you list). 3. **Name** and **description**. The name is the service name every proxy knows the upstream by. 4. **Servers from** (a load balancer): * **Listed here**: servers with a weight (HTTP), or `host:port` addresses (TCP, optionally with TLS to the server; UDP). For HTTP, type each server as `address:port` (`10.0.0.10:8080`): its **scheme badge** (`http://` at first) picks http or https for that server, and with https whether its certificate is checked (**Skip TLS verification**, per server on DNZ Proxy only). Its **heart button**, before the weight, sets its health check. A full URL typed or pasted sets the row's scheme, and **Enter** adds the next server. See [Typing the servers](https://docs.dnzwithazee.com/proxy/overview.md#typing-the-servers), [Certificate checks per server](#certificate-checks-per-server) and [Health checks per server](#health-checks-per-server). * **Every address of a DNS name**: DNZ looks the name up with its own DNS (its zones first) and keeps one server per address, such as every Docker Swarm node behind its routing mesh. * **Docker Swarm tasks** (HTTP): `tasks.`, which each DNZ Proxy node and Caddy look up themselves inside the service's overlay network. Refused on Traefik. **Test** looks the name up now, and the guides write the record set or the Swarm network set-up for you. See [Servers from a DNS name](https://docs.dnzwithazee.com/proxy/upstream-discovery.md). 5. **Balancing**: weighted round robin, power of two choices, highest random weight or least time, and **sticky sessions** (a cookie keeps each visitor on the server it first reached). 6. **Health check**: a path and how often to ask it; a server that doesn't answer well gets no visits. On DNZ Proxy a server can have a path of its own, or none. 7. **Connection** (folded): pass the visitor's `Host` header on, keep the server URLs' paths, and how every proxy connects to the servers: server name, trusted CAs, a client certificate, timeouts and, for TCP, the PROXY protocol. See [Connection settings](#connection-settings). Editing an upstream that routes use says how many routes on which proxies the change reaches. **Rename** follows the new name everywhere the upstream is named, on every proxy. **Delete** is refused while anything uses the upstream, and says what does. ## A route's own upstream In the route dialog, **Upstream** is either **Servers of its own** or **An upstream from the list**. Servers typed there make the route's **own upstream**, listed as *For app.example.com*: | What happens | The route's own upstream | | - | - | | The route's servers change in the dialog | It changes, on every proxy the route is on | | The same route is added to another proxy | That proxy uses it too: no copy | | The route picks an upstream from the list instead | It is deleted when nothing uses it any more | | The route is deleted | It is deleted when nothing uses it any more | From the upstream dialog, a route's own upstream can be made **shared**: it then stays when its route goes, and other routes can pick it. ## Merging **Merge…** points every route, upstream and error page that uses the upstreams picked at the one kept, and deletes the others. When two upstreams have the same servers but other settings (one with a health check, one without, for example), the list suggests merging them and keeps the name someone typed rather than one made from a hostname. ## Connection settings Since DNZ v2 an upstream holds how the proxies connect to its servers, the same on every proxy that uses it: the upstream dialog's **Connection** (folded). Settings only Traefik runs are marked *Traefik only*. **HTTP upstreams:** * The **server name** (SNI) to send and check the certificate for. * **Trusted CAs**: PEM text, or paths that must exist on every proxy's machine. Once set, they replace the public CAs. * **Client certificates** for mTLS: the certificate and its key as PEM. DNZ keeps the key and never shows it again; saving without a key keeps the one DNZ has. * The expected certificate URI and SPIFFE, TLS versions and ciphers. * Idle connections, HTTP/2 and timeouts. **TCP upstreams:** connect timeout, keep-alive, termination delay, the **PROXY protocol** (so the servers see the client's address), and TLS to the servers (server name, CAs, client certificates). What each proxy kind does with them: | Proxy kind | Gets | | - | - | | Traefik | A servers transport named `dnz-`, which the upstream's load balancer names. An upstream with no settings whose HTTPS servers all skip the certificate check keeps the shared insecure transport | | DNZ Proxy | The same. It uses the server name, certificate checks, trusted CAs, the client certificate, timeouts and HTTP/2 (HTTP and TCP), for visits and health checks | | Caddy | Its `reverse_proxy` transport: server name, CAs (inline or files), timeouts, idle connections, HTTP/1.1 only. Client certificates aren't sent: Caddy reads them from files only, and the proxy page says so | ### Certificate checks per server Whether an HTTPS server's certificate is checked is each server's own setting, its scheme badge's **Skip TLS verification** (`skip_tls_verify` on the server in the API and in proxy YAML). Until v2 one *accept any certificate* switch covered all of an upstream's servers, so an upstream couldn't mix a server with a certificate worth checking and one without. On upgrade, every HTTPS server of an upstream that accepted any certificate (the default) keeps doing so. Skipping the check per server is **DNZ Proxy only**. Traefik and Caddy have one certificate setting per upstream, and the servers decide it: | Proxy kind | Every HTTPS server skips | None skips | Some skip, some don't | | - | - | - | - | | DNZ Proxy | Skipped | Checked | Each server as set, for visits and health checks | | Traefik | Skipped | Checked | **Every certificate checked**, and the proxy page warns that skipping per server is DNZ Proxy only | | Caddy | Skipped | Checked | **Every certificate checked**, with the same warning | So on Traefik and Caddy, a server that skips the check in a mixed upstream needs a valid certificate. The server editor marks the switch *DNZ Proxy* and says so when Traefik or Caddy routes use the upstream. From the command line, write such a server `https+insecure://10.0.0.5:8443`; `--scheme https+insecure` does the same for the servers of a DNS name. ### Health checks per server The upstream's health check (path and interval) applies to all its servers. On **DNZ Proxy** a server can have its own: the **heart button** of its row in **Servers** (between the address and the weight) picks **Upstream's path** (the default), **A path of its own**, or **Not checked** (it then counts as up). The heart shows the choice: muted at the upstream's path, highlighted with a path of its own, struck through when not checked. It is the same in the upstream dialog, the [route dialog](https://docs.dnzwithazee.com/proxy/overview.md#typing-the-servers) and the record dialog's route, which also set the route's own upstream's path and interval under the servers. A path of its own is asked at the upstream's interval, with the upstream's other health-check settings. The upstream dialog's **Health check → Per server** (marked *DNZ Proxy*) lists every server's check as an overview and can change it too. While no server differs, it is folded behind **Different path per server…**; when some do, it opens by itself, and folded it still names them (`10.0.5.12:8080 · /status`). In the API and proxy YAML it is `health_check_path` on the server (`off` for no check). | Proxy kind | A server's own health check path | | - | - | | DNZ Proxy | Used for that server. Nodes older than v2.0 check it at the upstream's path | | Traefik | Not used: every server is checked at the upstream's path, and the proxy page warns | | Caddy | Not used, the same, with a warning | From the command line, put it after the URL and weight: `http://10.0.0.5:8080=3?health=/status`, or `?health=off`. Uptime service monitors and proxy alerts follow each server's own check, as the nodes report it. ### From per-proxy transports Before v2 an upstream *named* a servers transport, and each proxy had its own transport by that name under its **Settings**. Upgrading to v2 moves each transport into the upstream that names it, client certificates copied in as PEM: * Where proxies connected to an upstream differently (another client certificate on another proxy, or no transport there), the upstream is **split** so each proxy keeps connecting as it did. The way most routes use keeps the upstream; each other way gets a copy named `-`, which those proxies' routes use. * Transports nothing named did nothing and are dropped (the audit log and the backup keep them). * **Caddy proxies start applying** an upstream's connection settings, which they ignored before. `dnz proxy advanced-check` lists the splits and the Caddy proxies affected; run it on a copy of the database before upgrading (see [Checking a proxy setup](#checking-a-proxy-setup) and [From per-proxy middlewares](https://docs.dnzwithazee.com/proxy/middlewares.md#from-per-proxy-middlewares)). ## TCP and UDP routes A route can be **TCP** or **UDP** instead of HTTP: pick it with the option cards at the top of **Add route** (not on Caddy, which routes HTTP only). It listens on an **entry point** of the proxy (DNZ Proxy's TCP and UDP entry points, or a Traefik's: UDP ones have `/udp` addresses) and sends connections to a TCP or UDP upstream, from the list or made of server addresses typed in the dialog. * **TLS** (TCP): none (every connection on the port), **ended by the proxy** (it holds the certificate, the servers get plain TCP) or **passed through** (the servers end TLS; the proxy reads only the hostname). With TLS, a route can take **only some hostnames** (SNI); their records can go through the proxy like an HTTP route's. * **Only from these addresses** (TCP): client addresses or ranges. * **Priority** (TCP): a higher one wins when routes overlap. * **Middlewares** (TCP): address lists and connection limits (at most this many open connections per client address) from [Proxy → Middlewares](https://docs.dnzwithazee.com/proxy/middlewares.md#tcp-middlewares), shared by every proxy's TCP routes. **New TCP middleware** in the dialog makes one and picks it. They are listed with the HTTP routes, marked *TCP pg* or *UDP dns* (protocol and entry point), with a TCP route's middlewares. TCP and UDP routes are in the proxy's [YAML](https://docs.dnzwithazee.com/proxy/config-as-code.md#tcp-and-udp-routes) and in [route history](https://docs.dnzwithazee.com/proxy/route-history.md) too. ## Coming from per-proxy services Before upstreams, each proxy kept services of its own, and a route on two proxies had a copy on each. Upgrading moves them into upstreams once: * Services identical in content (servers and weights, balancing, sticky sessions, health check and the rest) become **one upstream**, whatever their names. It takes the name someone typed rather than one made from a hostname, else the one most routes use. * The **same name with other servers** stays apart: the later one is named after its proxy, for example `swarm-service-edge2`. * A redirect route's empty service becomes no upstream. Services nothing used become upstreams shown as **Not used**; none is deleted. * What visitors get doesn't change. Sticky sessions keep the cookie name of the service whose name and settings the upstream took. **Proxy → Upstreams** says once what the move did, and the audit log keeps it. > [!WARNING] > > DNZ copies a SQLite database (and the Redis backend's local copy) to > `backups/before-upstreams-