3.2 KiB
GMetric Scripts
A collection of bash scripts designed to collect various system, hardware, and cluster metrics and send them to Ganglia using the gmetric tool.
Overview
Each script in this repository focuses on a specific area of system monitoring. They are intended to be run periodically (e.g., via cron) to provide real-time monitoring of cluster health and performance.
Available Scripts
| Script | Description | Metrics Collected |
|---|---|---|
systemd2ganglia |
Systemd Health | Number of failed systemd units. |
slurmnode2ganglia |
Slurm Node State | Node state, down reason, partition, and details of currently running jobs. |
slurmqueue2ganglia |
Slurm Queue State | Global queue statistics (running/pending jobs), node allocation per partition, and oldest pending job time. |
gpu2ganglia |
NVIDIA GPU Metrics | Detailed GPU stats including driver version, memory usage, utilization, temperature, power draw, and clocks. Supports multiple GPUs with per-GPU and aggregate metrics. |
ipmisensors2ganglia |
IPMI Hardware | Hardware sensors read via ipmi-sensors. |
sensors2ganglia |
General Hardware | System sensors from sensors and zen-rapl (temp, voltage, fan speed, etc.). |
infiniband2ganglia |
InfiniBand Network | Bandwidth and packet statistics using perfquery. |
ecc2ganglia |
ECC Memory | Correctable and uncorrectable ECC memory errors via EDAC. |
smart2ganglia |
Disk Health | HDD/SSD temperatures using smartctl. |
psi2ganglia |
Pressure Stall Info | Linux PSI metrics for CPU, Memory, and IO pressure. |
ups2ganglia |
APC UPS Metrics | UPS status including load, battery charge, and runtime using apcaccess. |
hw2ganglia |
Environment & Network | UPS power consumption, ambient temperature, and switch uplink bandwidth via SNMP. |
Prerequisites
Depending on the scripts you use, you may need the following tools installed:
- Core:
gmetric(part of Ganglia) - GPU:
nvidia-smi(NVIDIA drivers) - Slurm:
sinfo,squeue - Hardware:
ipmi-sensors,lm-sensors,smartmontools(smartctl) - Networking:
ibstat,perfquery(InfiniBand),snmpget(Net-SNMP) - UPS:
apcaccess(APC UPS tools) - Other:
bc,awk,sed,grep
Usage
Running a Single Script
Make the script executable and run it:
chmod +x gpu2ganglia
./gpu2ganglia
Automation
To monitor your system continuously, add the desired scripts to your crontab. For example, to run gpu2ganglia every minute:
* * * * * /home/matt/code/gmetric-scripts/gpu2ganglia
Combining Scripts
If you have many metrics to collect, it is more efficient to combine them into a single wrapper script to reduce the number of process spawns:
#!/bin/bash
# combined_metrics.sh
/home/matt/code/gmetric-scripts/gpu2ganglia
/home/matt/code/gmetric-scripts/systemd2ganglia
/home/matt/code/gmetric-scripts/sensors2ganglia
# ... add others as needed
Contribution
Feel free to add new scripts or improve existing ones. Ensure that your metrics are logically grouped using the --group flag in gmetric for better organization in the Ganglia web interface.