Node monitoring

Watch your nodes live, and get told when one goes wrong.

When an alert rule stays broken for long enough, the panel opens an incident.

Where to find it

Monitoring is not in the admin sidebar. Open it from the Nodes page.

1. Navigate to Admin > Nodes.
2. Click the Monitoring tab at the top of the page. The tab strip has three tabs: Nodes, Monitoring, and Incidents.

To look at one node only, open that node from Admin > Nodes and click its Monitoring tab.

The monitoring page has two tabs of its own: Alert rules and Live nodes. Alert rules opens first.

Live nodes

Click Live nodes to see the metrics. The node list is on the left. Click a node to see its metrics on the right. The view shows one node at a time.

The metrics refresh every 15 seconds. The panel keeps 14 days of history.

Time window

The Window buttons set the period of the charts: 15m, 1h, 6h, and 24h. The default is 1h.

Move the pointer along a chart to read the value at one moment in time.

Metrics

Card	What it shows
CPU %	Processor use of the whole node.
Memory %	Memory in use, as a percentage of the total memory of the node.
Disk used	Disk in use, as a percentage of the total disk of the node.
1m load	The one-minute load average.
Net in	Inbound network rate.
Net out	Outbound network rate.
Dropped pkts	Packets that the node dropped, per second.
DDoS Mitigation	Whether protection is active on this node, and the number of active rules.

Busiest servers

This card ranks the game servers on the node. The columns are Server, CPU, Memory, and Disk. The Sort by buttons change the order between CPU and Disk.

Use this card to find the server that causes high load on a node.

Busiest processes

This card lists the 20 processes with the highest processor use on the node. The columns are Process, CPU, and RSS. Each row shows the process name, the process ID, and the user that owns it.

Filesystems

This card lists each mounted partition with the space in use and a usage bar. The bar changes colour with the usage:

Usage	Colour
Less than 75%	Green
75% to 89%	Amber
90% or more	Red

Stopping a process

WARNING

A process that you stop here is a host process, not a game server. Stop a game server from the server page.

Each row in Top Processes has two links:

• term sends a stop signal and lets the process close itself.
• kill ends the process immediately.

Click the link, then confirm in the dialog. Use term first. If the process does not stop, use kill.

Alert rules

An alert rule watches one metric on your nodes. If the metric breaks the rule for the full sustain time, the panel opens an incident. When the metric returns to normal, the panel resolves that incident.

The Alert rules table shows these columns: Rule, Severity, Condition, For, and Status. Click the Enabled or Disabled pill to turn a rule on or off.

The rules you start with

A new panel has four rules. All four arrive Disabled. Turn on the rules that you want.

Rule	Condition	Sustain	Severity
Memory > 90%	memory_pct > 90	300 seconds	warning
CPU > 90%	cpu_pct > 90	300 seconds	warning
Disk > 95%	disk_pct > 95	60 seconds	error
Node offline >2m	offline_seconds > 120	0 seconds	error

Adding a rule

1. On the Alert rules tab, click New rule.
2. Enter a Name. The name is required.
3. Select a Severity: info, warning, or error. The default is warning.
4. Build the condition. Read the next section for the two modes.
5. Set Sustain for (seconds). The default is 300. The metric must break the rule for this full time before the panel opens an incident.
6. Leave Enabled on to make the rule live.
7. Click Save.

To change a rule, click Edit on its row. To remove a rule, click Delete and confirm.

Builder or expression

Builder mode gives you one metric, one operator, and one threshold. This mode covers most rules.

Expression mode lets you combine metrics with AND, OR, and NOT. For example:

(memory_pct > 75 OR cpu_pct > 90) AND server_count > 0

These are the metrics that you can use in both modes:

Metric	Meaning
memory_pct	Memory use, in percent.
cpu_pct	Processor use, in percent.
disk_pct	Disk use, in percent.
memory_used_mb	Memory in use, in MB.
disk_used_mb	Disk in use, in MB.
load_avg_1m	One-minute load average.
offline_seconds	Seconds since the last contact from the node.
server_count	Number of game servers on the node.

The operators are >, <, >=, <=, ==, and !=.

What the alert says

Title template and Summary template set the text of the incident. You can put these placeholders in either field:

{node}, {rule}, {metric}, {value}, {threshold}, {duration}

The default title is {node}: {rule} at {value}%. The default summary is Sustained for {duration}s. Threshold {threshold}.

A rule that you make here applies to every node.

Next steps

• Incidents - the incidents that these rules open
• Linux Nodes - add and manage a Linux node
• Node Drives - storage on a Windows node