Alerts and checks

vulnboxd can post to a webhook when something needs a human: a service that stopped answering, a failed build or deploy, a rollback, a broken rules file. The format is Discord's, so a Discord channel webhook works as it is:

webhook_url: https://discord.com/api/webhooks/.../...

See webhook_url and monitor.

Alerts are sent in the order they happen, and each starts with the box's name: its address from init.vulnbox_ip or the game mode, or its hostname otherwise.

An alert that cannot be delivered — Discord rate-limiting you, the network gone — is logged and dropped, not retried, and nothing else waits for it. The webhook is a convenience, so treat the metrics and the scoreboard as what you actually watch.

The capture broker is its own service and alerts for itself, so a capture that stops says so even though the daemon knows nothing about it.

What is alerted

alert	when
🔴 port does not answer	a tcp port missed monitor.failures checks in a row
🟢 port is back	the port answers again, with how long it was gone
🔴 the checker says down	the gameserver's verdict turned to down (game modes)
🟡 recovering / 🟢 up again	the verdict improved
❌ building failed	a new version did not build; the running one stays
❌ deploying failed	docker compose up failed
🔴 not working after the deploy	a deploy failed its health check and there was nothing to roll back to
⏪ rolled back	vulnboxd rolled a change back, and why
⚠️ rules do not load	a rules file is broken; the previous rules stay
⬆️ taking a new build up	an upgrade is installed and about to be taken up
❌ a new vulnboxd cannot be run	the installed build would not start; the running one keeps going
⚠️ the traffic capture stopped	the capture broker's command died, so nothing is being captured

A setup run posts to the same webhook when it starts and finishes, unless init.webhook_url names another one.

Port checks

Every monitor.interval (10 seconds), vulnboxd checks that every tcp port of every service accepts a connection. For a port behind the proxy it checks the service itself, on its loopback port, since the proxy always takes the connection. A port vulnboxd leaves alone is checked where it is published.

monitor:
  interval: 10s
  failures: 3   # checks in a row before a port counts as down

A port that stops answering is alerted once, and again when it is back. Checks pause during the restart window and while a service is being deployed, when it is expected to be away for a moment. A port that is down shows as DOWN in vulnboxctl top, and as vulnboxd_port_up 0 in the metrics.

Udp ports have no generic check, so only tcp ports are watched.

Health checks after a deploy

After every deploy, vulnboxd waits up to deploy.health_timeout (30 seconds) for the service to work:

• every container is running, or has finished cleanly if it is a one-off job;
• no container reports itself unhealthy or is still starting;
• every tcp port answers.

A new version that fails this is rolled back.