From 206cdb50a8ac886f12de24fb6a295b23742a6af6 Mon Sep 17 00:00:00 2001
From: moosecrap <admin@long-cat.net>
Date: Mon, 2 Nov 2026 19:10:38 -0800
Subject: [PATCH] Readme

---
 README.md | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 72 insertions(+), 2 deletions(-)

diff --git a/README.md b/README.md
index 7817d72..c65d2fb 100644
--- a/README.md
+++ b/README.md
@@ -1,3 +1,73 @@
-A collection of various scripts that use `gmetric` to send statistics to [Ganglia](https://github.com/ganglia/).
+# GMetric Scripts
 
-You may find it useful to combine these into one script for the metrics you would like
\ No newline at end of file
+A collection of bash scripts designed to collect various system, hardware, and cluster metrics and send them to [Ganglia](https://github.com/ganglia/) using the `gmetric` tool.
+
+## Overview
+
+Each script in this repository focuses on a specific area of system monitoring. They are intended to be run periodically (e.g., via cron) to provide real-time monitoring of cluster health and performance.
+
+### Available Scripts
+
+| Script | Description | Metrics Collected |
+| :--- | :--- | :--- |
+| `systemd2ganglia` | Systemd Health | Number of failed systemd units. |
+| `slurmnode2ganglia` | Slurm Node State | Node state, down reason, partition, and details of currently running jobs. |
+| `slurmqueue2ganglia` | Slurm Queue State | Global queue statistics (running/pending jobs), node allocation per partition, and oldest pending job time. |
+| `gpu2ganglia` | NVIDIA GPU Metrics | Detailed GPU stats including driver version, memory usage, utilization, temperature, power draw, and clocks. Supports multiple GPUs with per-GPU and aggregate metrics. |
+| `ipmisensors2ganglia` | IPMI Hardware | Hardware sensors read via `ipmi-sensors`. |
+| `sensors2ganglia` | General Hardware | System sensors from `sensors` and `zen-rapl` (temp, voltage, fan speed, etc.). |
+| `infiniband2ganglia` | InfiniBand Network | Bandwidth and packet statistics using `perfquery`. |
+| `ecc2ganglia` | ECC Memory | Correctable and uncorrectable ECC memory errors via EDAC. |
+| `smart2ganglia` | Disk Health | HDD/SSD temperatures using `smartctl`. |
+| `psi2ganglia` | Pressure Stall Info | Linux PSI metrics for CPU, Memory, and IO pressure. |
+| `ups2ganglia` | APC UPS Metrics | UPS status including load, battery charge, and runtime using `apcaccess`. |
+| `hw2ganglia` | Environment & Network | UPS power consumption, ambient temperature, and switch uplink bandwidth via SNMP. |
+
+## Prerequisites
+
+Depending on the scripts you use, you may need the following tools installed:
+
+- **Core**: `gmetric` (part of Ganglia)
+- **GPU**: `nvidia-smi` (NVIDIA drivers)
+- **Slurm**: `sinfo`, `squeue`
+- **Hardware**: `ipmi-sensors`, `lm-sensors`, `smartmontools` (`smartctl`)
+- **Networking**: `ibstat`, `perfquery` (InfiniBand), `snmpget` (Net-SNMP)
+- **UPS**: `apcaccess` (APC UPS tools)
+- **Other**: `bc`, `awk`, `sed`, `grep`
+
+## Usage
+
+### Running a Single Script
+
+Make the script executable and run it:
+
+```bash
+chmod +x gpu2ganglia
+./gpu2ganglia
+```
+
+### Automation
+
+To monitor your system continuously, add the desired scripts to your crontab. For example, to run `gpu2ganglia` every minute:
+
+```cron
+* * * * * /home/matt/code/gmetric-scripts/gpu2ganglia
+```
+
+### Combining Scripts
+
+If you have many metrics to collect, it is more efficient to combine them into a single wrapper script to reduce the number of process spawns:
+
+```bash
+#!/bin/bash
+# combined_metrics.sh
+
+/home/matt/code/gmetric-scripts/gpu2ganglia
+/home/matt/code/gmetric-scripts/systemd2ganglia
+/home/matt/code/gmetric-scripts/sensors2ganglia
+# ... add others as needed
+```
+
+## Contribution
+
+Feel free to add new scripts or improve existing ones. Ensure that your metrics are logically grouped using the `--group` flag in `gmetric` for better organization in the Ganglia web interface.