# GMetric Scripts

A collection of bash scripts designed to collect various system, hardware, and cluster metrics and send them to [Ganglia](https://github.com/ganglia/) using the `gmetric` tool.

## Overview

Each script in this repository focuses on a specific area of system monitoring. They are intended to be run periodically (e.g., via cron) to provide real-time monitoring of cluster health and performance.

### Available Scripts

| Script | Description | Metrics Collected |
| :--- | :--- | :--- |
| `systemd2ganglia` | Systemd Health | Number of failed systemd units. |
| `slurmnode2ganglia` | Slurm Node State | Node state, down reason, partition, and details of currently running jobs. |
| `slurmqueue2ganglia` | Slurm Queue State | Global queue statistics (running/pending jobs), node allocation per partition, and oldest pending job time. |
| `gpu2ganglia` | NVIDIA GPU Metrics | Detailed GPU stats including driver version, memory usage, utilization, temperature, power draw, and clocks. Supports multiple GPUs with per-GPU and aggregate metrics. |
| `ipmisensors2ganglia` | IPMI Hardware | Hardware sensors read via `ipmi-sensors`. |
| `sensors2ganglia` | General Hardware | System sensors from `sensors` and `zen-rapl` (temp, voltage, fan speed, etc.). |
| `infiniband2ganglia` | InfiniBand Network | Bandwidth and packet statistics using `perfquery`. |
| `ecc2ganglia` | ECC Memory | Correctable and uncorrectable ECC memory errors via EDAC. |
| `smart2ganglia` | Disk Health | HDD/SSD temperatures using `smartctl`. |
| `psi2ganglia` | Pressure Stall Info | Linux PSI metrics for CPU, Memory, and IO pressure. |
| `ups2ganglia` | APC UPS Metrics | UPS status including load, battery charge, and runtime using `apcaccess`. |
| `hw2ganglia` | Environment & Network | UPS power consumption, ambient temperature, and switch uplink bandwidth via SNMP. |

## Prerequisites

Depending on the scripts you use, you may need the following tools installed:

- **Core**: `gmetric` (part of Ganglia)
- **GPU**: `nvidia-smi` (NVIDIA drivers)
- **Slurm**: `sinfo`, `squeue`
- **Hardware**: `ipmi-sensors`, `lm-sensors`, `smartmontools` (`smartctl`)
- **Networking**: `ibstat`, `perfquery` (InfiniBand), `snmpget` (Net-SNMP)
- **UPS**: `apcaccess` (APC UPS tools)
- **Other**: `bc`, `awk`, `sed`, `grep`

## Usage

### Running a Single Script

Make the script executable and run it:

```bash
chmod +x gpu2ganglia
./gpu2ganglia
```

### Automation

To monitor your system continuously, add the desired scripts to your crontab. For example, to run `gpu2ganglia` every minute:

```cron
* * * * * /home/matt/code/gmetric-scripts/gpu2ganglia
```

### Combining Scripts

If you have many metrics to collect, it is more efficient to combine them into a single wrapper script to reduce the number of process spawns:

```bash
#!/bin/bash
# combined_metrics.sh

/home/matt/code/gmetric-scripts/gpu2ganglia
/home/matt/code/gmetric-scripts/systemd2ganglia
/home/matt/code/gmetric-scripts/sensors2ganglia
# ... add others as needed
```

## Contribution

Feel free to add new scripts or improve existing ones. Ensure that your metrics are logically grouped using the `--group` flag in `gmetric` for better organization in the Ganglia web interface.