Skip to main content
High availability → Drills. A failover you have never tried is a guess. On a schedule, DNZ takes something out on purpose and reports whether the failover worked and how long it took.
A drill touches production on purpose, so it is off by default. Switching drills on starts nothing by itself: a drill runs only if you set one up, and by itself only if you say so.

What keeps it safe

  • Nothing is killed. A node leaves DNS and stops answering, or one node counts a server as down. The process stays, open connections finish, and the origin isn’t touched.
  • It is refused when it would leave nothing. No drill takes out the last healthy node of a proxy, or the last healthy server of an upstream, or a server without a health check (nothing would notice).
  • It runs only in your window, on its schedule or when someone starts it, and only when it can finish inside it.
  • It stops by itself: at its time limit, and at the first sign of visitors getting errors.
  • An undone drill leaves no change behind, even if DNZ restarts in the middle. What a drill changes is written down before it changes it, and every change has an end time. A node that lost DNZ comes back by itself, in its own clock.
  • One at a time. None starts while another drill, a node update, or an open incident on what it touches is under way.
  • People start drills. An assistant only reads them.

The stages

Start with the first; go on when it passes. The first two are for DNZ Proxy only: DNZ can’t take anything out on a Traefik or Caddy it doesn’t run. Nodes must run this version or newer; an older one is refused by name.

How a run goes

1

Reading

For 30 seconds nothing changes: DNZ reads what errors look like now, so a site that always answers some 500s doesn’t count against the drill.
2

Asking again

Everything that could refuse it is asked again, right before anything changes.
3

Out

A server is counted as down, or a node leaves DNS and then stops answering.
4

Noticed

For a server, the node’s next health check finds it down, as it would a real failure. For a node, DNZ notices the way it notices a real failure: its reports stop, or its check of the node’s port fails.
5

Held

For 30 seconds once traffic moved, to see visitors aren’t hurt.
6

Back

Everything goes back, and DNZ waits up to 3 minutes until it is as it was.

When it stops by itself

Whichever comes first, and everything goes back at once: A node that doesn’t come back within 3 minutes after a drill fails it, by name. If it is still gone, its usual alert goes out too.

The report

Each run is kept with its drill’s history, a page at a time, and stays when the drill goes:
  • Time to notice: from taking it out until its health check found the server down, or DNZ found the node gone.
  • Time until traffic moved: for a server, until nothing was in progress to it on that node; for a node, until its requests fell to a tenth of what they were. For a node this is mostly your DNS TTL, and the number to look at.
  • Errors visitors saw: 5xx answers on the proxy’s nodes during the drill, against the share before it. Nodes that can’t tell (older ones, or with analytics off) are named: their errors aren’t in the count. A connection refused by a stopped node shows on no node: that is why a node’s drill waits for the TTL first, and why a monitor’s failed check counts.
  • What didn’t go as set: noticing or moving slower than the seconds you expect (120 by default), visitors still arriving after the node’s time in DNS, a take over’s check that failed.
A run ends as passed, failed (it found something), stopped (someone or something ended it, no finding), or refused (a schedule’s chance that wasn’t safe, said once per window, with why).
A passed drill says the failover worked that day, under that day’s traffic. It isn’t proof of the next one.

Quiet on purpose

A drill uses scheduled maintenance’s way of keeping things quiet, only for what it covers:
  • Monitors of the hostnames it covers keep checking, but a failed check counts nothing: no incident, nothing in the day’s uptime. Their row says In maintenance until ….
  • Alerts about its node and its server wait. If it is still down after the drill, they go out then.
  • Its incident is marked A failover drill, and is never on the public status page.
A real failure elsewhere alerts as ever. On the same proxy that also stops the drill.

Secondaries

A node’s address leaving DNS reaches a DNZ secondary as a real failure does: the primary’s answers change, the zone’s serial moves, and the transfer no longer holds it (and again on the way back). Only the cluster’s leader runs drills, never a DNZ secondary or a standby.

Alerts

Add the event A failover drill failed (failover_drill) to a notification provider (Settings → Notifications). New providers get it; existing ones keep the events they had. It goes out once per failed run.