Upgrading

vulnboxctl upgrade puts a new build in place and the running daemon takes it up without closing a single port. Nothing is refused, nothing is restarted, and the containers are not touched.

# A release archive, a directory, or the binaries themselves.
sudo vulnboxctl upgrade ./vulnboxd-x86_64-unknown-linux-gnu.tar.gz
sudo vulnboxctl upgrade ./target/release/vulnboxd ./target/release/vulnboxctl
sudo vulnboxctl upgrade          # the latest release from GitHub

Where a build can come from

what you give it	what it does
nothing	fetches a release from GitHub, checks it against its .sha256
a .tar.gz, .tgz, .tar.xz, .tar	unpacks it; a .sha256 next to it is checked if there is one
a directory	takes the binaries out of it
one or more files	installs those, whatever they are called

Files are identified by asking them: vulnboxd-0.8.0-rc2, or a binary straight out of cargo build under any name, is recognised by what it answers to --version. Anything that will not name itself as one of vulnboxd's binaries is refused rather than installed over something that works, which is what catches a half-finished copy or a build for the wrong architecture.

Installing only vulnboxd or only vulnboxctl warns and carries on. The two are built together and mixed versions are not supported, so do it only when you know what you changed.

When it happens

By default the daemon takes the new build up at the next end-of-tick window — the same few seconds in which it restarts services anyway, when the game is least likely to be looking. vulnboxctl upgrade says so and waits until the daemon has acknowledged it:

==> found vulnboxd 0.8.0
==> found vulnboxctl 0.8.0
==> installed vulnboxd and vulnboxctl to /usr/local/bin
==> asked vulnboxd to take the new build up at the end of the next tick
==> vulnboxd acknowledged; watch it switch with journalctl -fu vulnboxd

--now takes it up as soon as nothing is being built or deployed, usually within a second. Either way, the switch never lands in the middle of a deploy.

If the daemon is not running there is nothing to hand over: the new build is what systemd starts next.

If the new build cannot run at all — a config it will not read, a binary for the wrong architecture — the old daemon notices that it stopped before handing it anything, keeps managing and keeps its connections, and says so in an alert. Nothing changes until you fix it.

What makes it seamless

A stop-and-start leaves every public port unbound in between. It is only milliseconds, but a connection that arrives in them is refused, and a checker that is refused scores the service down for the whole tick. Connections already being served are dropped as well.

So the daemon is not restarted. It starts the new build beside itself and hands its listening sockets across, so the ports are never closed, and then steps back and stays alive to finish the connections it already had open:

	stop and start	taking the new build up
public ports	closed, then reopened	never closed
connections arriving during it	refused	wait in the kernel's backlog, then served
connections being served	dropped	finished by the old daemon
containers	untouched	untouched
systemd	counts a restart	told which process to watch; no restart

Measured on one service, hammering the port across the switch: a plain restart refused 4 connections, the handover refused none of 3,599. A connection opened before the handover kept exchanging data after it, and the old daemon exited the moment that connection closed.

Why the old daemon has to stay

A connection being served is more than a socket. Where the client is in its request, how far the rules have scanned its body, which rules have already been decided, which upstream socket it is paired with — all of that lives in the memory of the daemon serving it. Handing the socket to the new build would hand over none of it.

Relaying the bytes blindly instead would keep such a connection alive, but the rules would stop applying to it: on an HTTP keep-alive connection the next request would go through unfiltered, so anyone holding a connection open across an upgrade would escape filtering. That is the opposite of what the proxy is for, so the old daemon keeps serving them itself, rules and all, until they end.

Only one daemon manages

For those few seconds two daemons are alive, and exactly one of them may deploy, commit, push or ask the gameserver anything. That is not left to careful ordering: the right to manage is an exclusive lock on manager.lock in state_dir. The old daemon lets it go as it steps back, the new one does nothing until it holds it, and a daemon that cannot get it stops rather than working alongside another. The kernel releases the lock if a daemon is killed, so a crash cannot leave the box with nothing in charge.

The daemon that has stepped back does nothing else at all: no deploys, no git, no gameserver, no port checks, no alerts. It finishes its connections and exits.

Settings

Full entries in the configuration reference and vulnboxctl upgrade.

upgrade:
  repo: aarondewes/vulnboxd   # where a release is fetched from
  token: ""                   # for a private repository, or VULNBOXD_UPGRADE_TOKEN
  drain: 30s                  # 0 replaces the daemon outright, closing what was open

Whatever is still open when the drain is up is closed. A service whose connections are long-lived wants a longer one; 0 gives the simpler behaviour of replacing the process in place.

From a private repository

A team that keeps its own fork private points upgrade at it, with a token that can read it. A fine-grained token with read access to that one repository's contents is enough, and a config holding one is written readable by root only. VULNBOXD_UPGRADE_TOKEN in the environment wins over the file.

Going back

A release that turns out to be worse goes back the same way it came:

sudo vulnboxctl upgrade --version v0.6.0

Nothing about an upgrade is one-way — it is only ever binaries being replaced — and the old build is still a release, so a downgrade is an upgrade to an earlier tag. Your config and state are untouched by either.

Your config after an upgrade

vulnboxd is a thing you run for a weekend, so settings that are renamed are renamed, with no older spelling kept working alongside. An unknown key is an error rather than a warning, so a config written for an older build can stop a new one starting, and the release notes say what moved.

The handover is built for that: the new build loads the config before it takes over, so a config it cannot read means it exits straight away, and the old daemon keeps managing, keeps its connections and alerts. Nothing is lost — the upgrade simply does not happen until you fix the file.

Checking which build is running

vulnboxd --version
curl -s 127.0.0.1:9330/metrics | grep vulnboxd_info

vulnboxd_info{version="..."} is what the running daemon says about itself, which is the one that counts after a handover — vulnboxd --version only reports the binary on disk.

The capture is left alone

The capture broker is a separate service and a separate binary, so an upgrade does not stop it and a reader of its stream never notices. --pcap replaces it too, which does restart it and does interrupt the capture:

sudo vulnboxctl upgrade ./release.tar.gz --pcap

The unit has to allow it

Staying behind means two processes for a moment, and systemd watches one process per service: if it is not told which, the old one exiting looks like the service stopping and it starts a second daemon next to the draining one. So the unit carries

Type=notify
NotifyAccess=main

and the old daemon moves systemd's MAINPID to the new one before it steps back.

A daemon that finds it cannot say this — an older unit, Type=simple — replaces itself instead and closes the connections that were open, which is the behaviour without a drain. It says so in the log. That is deliberate: a dropped connection costs a tick, while two daemons deploying and committing to the same repositories costs the game.

After upgrading to a version that drains

A running service keeps the Type= it was started with, even after a daemon-reload, so the first upgrade that installs this unit cannot use it yet. If systemctl show -p Type vulnboxd says simple, this applies to you: vulnboxctl upgrade writes the new unit, reloads systemd and tells you that this particular handover still closes connections. One systemctl restart vulnboxd, at a moment you choose, and every upgrade after it keeps them.

Doing it by hand

The request is a file, so anything can ask:

echo window > <state_dir>/upgrade-pending   # or: now

state_dir is /var/lib/vulnboxd unless you changed it.

The daemon reads it within a second and removes it. Without one, systemctl restart vulnboxd works as always; it is just the version that refuses a few connections.

Replacing /usr/local/bin/vulnboxd with cp or scp while the daemon runs fails with text file busy: a running binary cannot be written into. vulnboxctl upgrade writes the new file next to the old one and renames it over the top, which is atomic and never hits that.