74 lines
3.2 KiB
Markdown
74 lines
3.2 KiB
Markdown
# GMetric Scripts
|
|
|
|
A collection of bash scripts designed to collect various system, hardware, and cluster metrics and send them to [Ganglia](https://github.com/ganglia/) using the `gmetric` tool.
|
|
|
|
## Overview
|
|
|
|
Each script in this repository focuses on a specific area of system monitoring. They are intended to be run periodically (e.g., via cron) to provide real-time monitoring of cluster health and performance.
|
|
|
|
### Available Scripts
|
|
|
|
| Script | Description | Metrics Collected |
|
|
| :--- | :--- | :--- |
|
|
| `systemd2ganglia` | Systemd Health | Number of failed systemd units. |
|
|
| `slurmnode2ganglia` | Slurm Node State | Node state, down reason, partition, and details of currently running jobs. |
|
|
| `slurmqueue2ganglia` | Slurm Queue State | Global queue statistics (running/pending jobs), node allocation per partition, and oldest pending job time. |
|
|
| `gpu2ganglia` | NVIDIA GPU Metrics | Detailed GPU stats including driver version, memory usage, utilization, temperature, power draw, and clocks. Supports multiple GPUs with per-GPU and aggregate metrics. |
|
|
| `ipmisensors2ganglia` | IPMI Hardware | Hardware sensors read via `ipmi-sensors`. |
|
|
| `sensors2ganglia` | General Hardware | System sensors from `sensors` and `zen-rapl` (temp, voltage, fan speed, etc.). |
|
|
| `infiniband2ganglia` | InfiniBand Network | Bandwidth and packet statistics using `perfquery`. |
|
|
| `ecc2ganglia` | ECC Memory | Correctable and uncorrectable ECC memory errors via EDAC. |
|
|
| `smart2ganglia` | Disk Health | HDD/SSD temperatures using `smartctl`. |
|
|
| `psi2ganglia` | Pressure Stall Info | Linux PSI metrics for CPU, Memory, and IO pressure. |
|
|
| `ups2ganglia` | APC UPS Metrics | UPS status including load, battery charge, and runtime using `apcaccess`. |
|
|
| `hw2ganglia` | Environment & Network | UPS power consumption, ambient temperature, and switch uplink bandwidth via SNMP. |
|
|
|
|
## Prerequisites
|
|
|
|
Depending on the scripts you use, you may need the following tools installed:
|
|
|
|
- **Core**: `gmetric` (part of Ganglia)
|
|
- **GPU**: `nvidia-smi` (NVIDIA drivers)
|
|
- **Slurm**: `sinfo`, `squeue`
|
|
- **Hardware**: `ipmi-sensors`, `lm-sensors`, `smartmontools` (`smartctl`)
|
|
- **Networking**: `ibstat`, `perfquery` (InfiniBand), `snmpget` (Net-SNMP)
|
|
- **UPS**: `apcaccess` (APC UPS tools)
|
|
- **Other**: `bc`, `awk`, `sed`, `grep`
|
|
|
|
## Usage
|
|
|
|
### Running a Single Script
|
|
|
|
Make the script executable and run it:
|
|
|
|
```bash
|
|
chmod +x gpu2ganglia
|
|
./gpu2ganglia
|
|
```
|
|
|
|
### Automation
|
|
|
|
To monitor your system continuously, add the desired scripts to your crontab. For example, to run `gpu2ganglia` every minute:
|
|
|
|
```cron
|
|
* * * * * /home/matt/code/gmetric-scripts/gpu2ganglia
|
|
```
|
|
|
|
### Combining Scripts
|
|
|
|
If you have many metrics to collect, it is more efficient to combine them into a single wrapper script to reduce the number of process spawns:
|
|
|
|
```bash
|
|
#!/bin/bash
|
|
# combined_metrics.sh
|
|
|
|
/home/matt/code/gmetric-scripts/gpu2ganglia
|
|
/home/matt/code/gmetric-scripts/systemd2ganglia
|
|
/home/matt/code/gmetric-scripts/sensors2ganglia
|
|
# ... add others as needed
|
|
```
|
|
|
|
## Contribution
|
|
|
|
Feel free to add new scripts or improve existing ones. Ensure that your metrics are logically grouped using the `--group` flag in `gmetric` for better organization in the Ganglia web interface.
|