-
Notifications
You must be signed in to change notification settings - Fork 8
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #293 from barrucadu/alert-diskspacelow
Add an alert and a runbook for low disk space
- Loading branch information
Showing
3 changed files
with
50 additions
and
10 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,35 @@ | ||
DiskSpaceLow | ||
============ | ||
|
||
This alert fires when a partition has under 10% free space remaining. | ||
|
||
The alert will say which partitions are affected, `df -h` also has the | ||
information: | ||
|
||
``` | ||
$ df -h | ||
Filesystem Size Used Avail Use% Mounted on | ||
devtmpfs 1.6G 0 1.6G 0% /dev | ||
tmpfs 16G 112K 16G 1% /dev/shm | ||
tmpfs 7.8G 9.8M 7.8G 1% /run | ||
tmpfs 16G 1.1M 16G 1% /run/wrappers | ||
local/volatile/root 1.7T 1.8G 1.7T 1% / | ||
local/persistent/nix 1.7T 5.1G 1.7T 1% /nix | ||
local/persistent/persist 1.7T 2.0G 1.7T 1% /persist | ||
local/persistent/var-log 1.7T 540M 1.7T 1% /var/log | ||
efivarfs 128K 40K 84K 33% /sys/firmware/efi/efivars | ||
local/persistent/home 1.7T 32G 1.7T 2% /home | ||
/dev/nvme0n1p2 487M 56M 431M 12% /boot | ||
data/nas 33T 22T 11T 68% /mnt/nas | ||
tmpfs 3.2G 12K 3.2G 1% /run/user/1000 | ||
``` | ||
|
||
Note all ZFS datasets in the same pool (`local/*` and `data/*` in the example | ||
above) share the underlying storage. | ||
|
||
Debugging steps: | ||
|
||
- See the `node_filesystem_avail_bytes` metric for how quickly disk space is | ||
being consumed | ||
- Use `ncdu -x` to work out where the space is going | ||
- Buy more storage if need be |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters