Metrics
Served in the Prometheus text format on http://127.0.0.1:9330/metrics (see metrics). For how to use them during a game, see Watching with metrics.
Counters survive a rules reload and reset when vulnboxd restarts.
Traffic
| metric | type | labels | |
|---|---|---|---|
vulnboxd_connections_total | counter | service, port | client connections accepted |
vulnboxd_connections_open | gauge | service, port | client connections currently open |
vulnboxd_bytes_total | counter | service, port, direction | payload bytes forwarded; direction is in or out |
vulnboxd_http_responses_total | counter | service, port, class | responses by status class: 2xx, 4xx, 5xx, ... |
vulnboxd_upstream_errors_total | counter | service, port | failures to reach the service at all |
Flags
| metric | type | labels | |
|---|---|---|---|
vulnboxd_flags_out_total | counter | service, port | flags in responses sent to clients |
vulnboxd_flags_out_last_tick | gauge | service, port | flags that went out during the last whole tick |
A flag out is one of the game's flags, by its flag format, in something vulnboxd sent back to a client. That is what an attacker is stealing, so a jump here is an exploit working. It counts what is actually sent, after rules have run: a flag a rule rewrote away is not counted, and one a rule decrypts on the way out is.
It needs the flag format, from the config, the game mode or the gameserver. Without one nothing is counted. Responses that are compressed by the service are not read, and neither are the data connections of ftp. Every flag in a response counts, including your own checker's reads, so expect a steady baseline and watch for what rises above it.
vulnboxd_flags_out_last_tick is updated once a tick ends. It is not there for a tick that vulnboxd did not see from start to end, such as the one it was started in.
Rules
| metric | type | labels | |
|---|---|---|---|
vulnboxd_rule_hits_total | counter | service, port, rule, action | times a rule fired; action is blocked, stalled, rewritten or matched |
vulnboxd_connections_stalled | gauge | service, port | connections held open by a stall rule, out of limits.stalled; at the limit, further stalls block instead |
vulnboxd_bypassed_total | counter | service, port | requests that skipped every rule with a bypass header |
vulnboxd_mirror_total | counter | service, port, rule, result | mirrored requests; result is agrees, body differs, status differs or unreachable |
rule is the rule's name, or #1, #2, ... by position for rules without one. Name your rules.
A rule set to action: stall counts as stalled whenever it fires, including when the port was already holding limits.stalled connections and it blocked instead. vulnboxd_connections_stalled is what says how many are really being held.
Health
| metric | type | labels | |
|---|---|---|---|
vulnboxd_port_up | gauge | service, port | whether the port answered its last check, 1 or 0 |
vulnboxd_sla_up | gauge | service | whether the game's checker found the service working in the latest tick, 1 or 0; needs a game mode |
Deploys
| metric | type | labels | |
|---|---|---|---|
vulnboxd_deploys_total | counter | service, result | deploys, ok or failed |
vulnboxd_deploy_ms | gauge | service | how long the last deploy took, in milliseconds |
vulnboxd_builds_total | counter | service, result | builds of a new version, ok or failed |
vulnboxd_git_polls_total | counter | result | git fetches, ok or failed |
vulnboxd_rollbacks_total | counter | service, trigger | rollbacks vulnboxd made, health or sla |
vulnboxd_services | gauge | services being managed | |
vulnboxd_tick | gauge | the tick it is now, counted from the start time; follows the gameserver's timing when it gives one | |
vulnboxd_game_tick | gauge | the latest tick the gameserver has verdicts for | |
vulnboxd_gameserver_up | gauge | whether the gameserver answered the last time it was asked, 1 or 0 | |
vulnboxd_info | gauge | version | build information, always 1 |
Useful queries
# Flags that went out in the last tick, per service
sum by (service) (vulnboxd_flags_out_last_tick)
# Flags going out right now, per tick (here 180 s ticks)
sum by (service) (rate(vulnboxd_flags_out_total[1m])) * 180
# Rules firing right now, per minute
sum by (service, rule, action) (rate(vulnboxd_rule_hits_total[1m])) * 60
# Share of 5xx answers per service
sum by (service) (rate(vulnboxd_http_responses_total{class="5xx"}[5m]))
/ sum by (service) (rate(vulnboxd_http_responses_total[5m]))
# Deploys that came close to the window (here 5 s)
vulnboxd_deploy_ms > 4000