Rollback

When a change breaks a service, vulnboxd can put it back as it was. Every rollback is a commit that is pushed, so the team sees it in git, and the daemon deploys it like any other change.

What vulnboxd remembers

For each service, vulnboxd keeps track of two things: its code, as a git commit, and its rules, as the text of vulnboxd-rules.yml. For each of them it knows:

• what runs now;
• what ran before the latest change;
• the latest version known to work.

"Known to work" means the checker said the service was up after that version went live. Without a game mode to ask, the code counts as working once a deploy passes its health check. What runs when vulnboxd first sees a service is taken to work. This lives in <state_dir>/<service>/history.json, so it survives a restart.

There is no command for it, so to see what vulnboxd currently believes — and therefore what it would put back if you pushed something broken now — read that file:

jq . /var/lib/vulnboxd/web/history.json

code.good is the commit it would go back to, code.current what runs, and pinned whether the running commit asked to be left alone.

When it happens

A new version does not come up. After every deploy, vulnboxd waits up to deploy.health_timeout (30 seconds) for the containers to run and every tcp port to answer. If a newly built version fails that, it commits the code back to the last version known to work, pushes it, builds it and deploys it right away. It does not wait for the window: the service is down already.

The checker says down after a change. With a game mode, a verdict of down rolls back the change that came before it. Only one change is undone at a time, the most recent: the rules if they changed last, otherwise the code. If that was not the cause and the service is still down a tick later, the next one goes back too, so it works backwards rather than throwing away everything at once.

A change is only undone when it could be the cause:

• it was live for a whole tick before the verdict arrived, so a change made halfway through a tick is judged by the next tick's verdict;
• it went live before the service stopped working. A change made while it was already down is not what broke it, and may well be the fix;
• the checker has not already called the service working since.

You ask for it.

sudo vulnboxctl rollback web

This undoes the latest change to the service, its code or its rules, whichever came last. Running it again undoes the rollback.

A rollback vulnboxd decides on is alerted and counted in vulnboxd_rollbacks_total, labelled health or sla by what asked for it. One you run yourself is neither: vulnboxctl is a separate process from the daemon, so it has no webhook and no metrics of its own. You see it in git, and in the daemon's log when it deploys the revert.

What a rollback commit does

A code rollback puts the service's directory back as it was at the commit being returned to, in one new commit on top of the current one. It is a revert of everything since that commit, so it cannot conflict:

7e60548 Revert web to 0642173b4f25

The new version did not come up after the deploy: app exited with 1, port 18080 does not answer.

Committed by vulnboxd.

• Only this service. With several services in one repository, only the service's own directory changes.
• The rules stay. The service's rules file keeps its current state; the rules have a rollback of their own.
• Uncommitted work is overwritten. Uncommitted changes to the service's other tracked files are replaced, since the containers are built from the working tree and it has to match the commit. Commit your work.
• It is pushed. The commit is pushed, catching up with the remote first if a teammate pushed in the meantime.

A rules rollback writes the rules file back and commits and pushes it, the same way vulnboxd commits the rules file it creates.

After a rollback

The bad change is still in git, one commit back. To try it again, push a fixed version, or revert the revert. A later push is built and deployed as usual.

Settings

rollback:
  enabled: true
  consecutive: 1   # ticks the checker must call it down before anything is undone
  when: now        # or window
  services:
    notes: { consecutive: 3 }
    legacy: { enabled: false }

setting	default
enabled	true	false only alerts, and never undoes anything. vulnboxctl rollback still works
consecutive	1	how many ticks in a row the checker has to call the service down first. Higher is more patient: more SLA lost before it acts, but more evidence that the change really is the cause
when	now	now builds and deploys the revert as soon as it is ready; window waits for the end of the tick, like any other deploy. The service is usually down by then, so waiting is more downtime, not less
services	{}	the same three per service, by the name of its directory. Anything left out is taken from above

A service that is slow to come up, or whose checker is flaky, is a good candidate for a higher consecutive. A service you are in the middle of working on by hand is a good candidate for enabled: false.

Keeping a commit

A commit whose message carries Vulnboxd-Skip-Rollback is never rolled back away from, whatever the checker says and whatever the settings are:

Fix the auth bypass properly

Vulnboxd-Skip-Rollback: true

It is for a fix you are sure about, where vulnboxd's guess would be the wrong one: a patch that deliberately changes what the service does, say, or one you have just tested by hand. Only the commit that is running counts, so the next commit without it is treated normally again. The marker is matched anywhere in the message, in any case.

It will not loop

A revert is a new commit, so without care it would look like one more change to undo, and vulnboxd would revert its own revert for as long as the service stayed down. Two things stop that:

• a revert holds what the last working version held, so vulnboxd records it as being that version rather than a change away from it, and there is nothing left to undo;
• the same rollback is never made twice. If vulnboxd has already put a version back and the service is still down, something else is wrong, and it says so instead of trying again.