Just wanted to share with the community something I noticed last week while I was backing up my Ghost content’s folder with rsync. rsync would hang as it moved the logs directory; looking further into it, my Ghost access log had grown to several gigabytes and was continuing to grow.
Ultimately the cause is the lack of a logrotate configuration for the Ghost logs. After troubleshooting and testing my config this weekend, I am comfortable enough releasing it to the community. Hopefully this helps someone else out:
Note that this is only a concern for self-hosted instances of Ghost.
Glad to hear that! I don’t really see it as a problem with Ghost per se; logrotate is a standard utility that most Linux sysadmins would be aware of that might need configuration with the installation of a new application. The only reason I brought it up to the community is that others may not be aware.
Looks like I’m not the first person to notice this, someone else reported it here back in October:
Install process was about 1.5-2 yrs ago using the following guide for Ubuntu:
All defaults used, running Ubuntu 18.04 LTS with regular, weekly Ghost updates.
head -1 indicates the access log in /var/www/ghost/content/logs/ goes back to June 2020.
Looking further at the other report of this issue:
Now this is interesting. We have almost the exact same date (maybe I just updated a day later). Seems like an oddly strange coincidence, doesn’t it? Within a day of each other in June almost a year ago:
I have experienced something similar, but relating to insufficient permissions erroring out in logrotate.
I’ve added the su directive to the logrotate.d/ghost configuration file that now looks like this due to errors that were being output when testing logrotate manually with debug logging.
considering log /var/www/<domain>/content/logs/https___<domain>_production.log
error: skipping "/var/www/<domain>/content/logs/https___<domain>_production.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
more /etc/logrotate.d/ghost
/var/www/*/content/logs/*.log {
daily
maxsize 400M
notifempty
rotate 7
compress
copytruncate
su ghost ghost
}
I added `notifempty`, `su ghost ghost` and modified the number of log files to keep around and to rotate daily or at a specific `maxsize` (as file system sizes aren't a function of days/weeks/months but rather size).
I just wanted to chime in because my team experienced this exact same issue recently. During a massive traffic spike on our Ghost site, the log files grew drastically high over a short period. Since standard rotation is purely time-based (e.g., waiting 1 day to rotate), the file had plenty of time to indiscriminately consume our infrastructure until the site crashed entirely with a no space left on device error.
It was frustrating enough that we actually dug into the Ghost framework’s core codebase to see why we couldn’t just restrict the file size limit inside config.production.json.
Here’s what we found to fix it: Ghost does actually possess an incredibly robust log rotating engine natively (bunyan-rotating-filestream) that supports strict size-limits and gzip compression! However, it was completely hidden in the background, making it impossible for server admins to natively access these capabilities via regular defaults.
To solve this for all of us, we just created a Pull Request that formally exposes these advanced parameters to the default file structure. Once merged, any admin can securely cap their logs by size and enable compression, like this:
With this, Ghost will actively monitor the log size and instantly compress and rotate it the moment it hits 50m, fully preventing the disk-space bloat crashes.
If this has been a major pain point for you as well, it would be awesome if you could support/review the PR over on GitHub to help us get it merged into core faster!
You can check out the Pull Request here: