Have a look into the retention of backups on the server as well as off-site - bellow ran out of disk space causing a service outage on 7th April 2021 due to an excessive over retention of monthly backups on-site. It would appear monthly backups were not being cycled out but daily and weekly backups are being retained and cycled properly on site.
Description
Description
Event Timeline
Comment Actions
A monthly cron is now run on the second day of every month to delete monthly backups older than a specified amount, currently set to 120 days. This regained approximately 180 GB of disk space on the manual run. This is currently called "monthlyBackupCleanup" pending commit to the repository and rework as a ... more robust script.
Comment Actions
So this cropped up a couple more times - primarily around monthly archiving before the cleanup. Ran into it again as dailies finally started being big enough to push us over that edge.
Backups now live on a 1TB NFS store, this should be more than suitable for the foreseeable future, especially as we're doing some server moves when I can free up a bit more time.