I am running 22.04.1-Ubuntu with Ghost 5.59.2 and noticing that my Google Cloud server freezes up and needs to be rebooted every few days. it’s a (free) micro instance with 2 vCPU + 1 GB memory + 30 GB standard persistent disk. I assumed this would be sufficient for Ghost cuz I didn’t think it requires a lot of CPU or memory.
Was just looking at my syslogs and it appears there are multiple issues:
snapd.service
is timing out and failing to start.systemd-journald.service
is also timing out and has corruption issues.- A node job (
email-analytics-fetch-latest
) is already running and causing warnings. - The logs suggest that snapd is struggling to either acquire resources or finish its tasks within the watchdog’s time limit and this is probably what is causing the system to freeze.
Does anyone else have server freezes like these?
Aug 26 15:09:13 ghost-blog systemd[1]: Starting Daily apt download activities…
Aug 26 15:09:16 ghost-blog systemd[1]: Starting Update APT News…
Aug 26 15:09:17 ghost-blog systemd[1]: Starting Update the local ESM caches…
Aug 26 15:09:20 ghost-blog systemd[1]: apt-news.service: Deactivated successfully.
Aug 26 15:09:20 ghost-blog systemd[1]: Finished Update APT News.
Aug 26 15:09:25 ghost-blog systemd[1]: esm-cache.service: Deactivated successfully.
Aug 26 15:09:26 ghost-blog systemd[1]: Finished Update the local ESM caches.
Aug 26 15:13:39 ghost-blog systemd[1]: Starting GCE Workload Certificate refresh…
Aug 26 15:24:35 ghost-blog CRON[12257]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Aug 26 15:29:13 ghost-blog systemd[1]: snapd.service: Watchdog timeout (limit 5min)!
Aug 26 15:29:55 ghost-blog systemd[1]: snapd.service: Killing process 433 (snapd) with signal SIGABRT.
Aug 26 15:30:33 ghost-blog snapd[433]: SIGABRT: abort
Aug 26 15:31:48 ghost-blog snapd[433]: PC=0x562000ae89a1 m=0 sigcode=0
Aug 26 15:33:38 ghost-blog snapd[433]: goroutine 0 [idle]:
Aug 26 15:35:27 ghost-blog node[891]: [2023-08-26 15:27:53] #033[35mWARN#033[39m Job “email-analytics-fetch-latest” is already running
Aug 26 15:37:26 ghost-blog node[891]: #033[35m
Aug 26 15:38:46 ghost-blog node[891]: #033[35mJob “email-analytics-fetch-latest” is already running#033[39m
Aug 26 15:39:30 ghost-blog node[891]: #033[90m----------------------------------------#033[39m
Aug 26 15:41:07 ghost-blog kernel: [213989.210823] systemd[1]: snapd.service: start operation timed out. Terminating.
Aug 26 15:41:19 ghost-blog kernel: [214000.980789] systemd[1]: snapd.service: Failed with result ‘timeout’.
Aug 26 15:41:20 ghost-blog kernel: [214002.860936] systemd[1]: Failed to start Snap Daemon.
Aug 26 15:41:33 ghost-blog kernel: [214015.540420] systemd[1]: systemd-journald.service: State ‘stop-watchdog’ timed out. Killing.
Aug 26 15:41:34 ghost-blog kernel: [214016.473929] systemd[1]: systemd-journald.service: Killing process 117 (systemd-journal) with signal SIGKILL.
Aug 26 15:41:36 ghost-blog kernel: [214018.600673] systemd[1]: systemd-journald.service: Main process exited, code=killed, status=9/KILL
Aug 26 15:41:36 ghost-blog kernel: [214018.600881] systemd[1]: systemd-journald.service: Failed with result ‘watchdog’.
Aug 26 15:41:39 ghost-blog kernel: [214021.507273] systemd[1]: systemd-journald.service: Consumed 10.720s CPU time.
Aug 26 15:41:59 ghost-blog kernel: [214040.880879] systemd[1]: snapd.service: Scheduled restart job, restart counter is at 2.
Aug 26 15:42:08 ghost-blog kernel: [214050.097606] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1.
Aug 26 15:42:08 ghost-blog kernel: [214050.856890] systemd[1]: Stopped Journal Service.
Aug 26 15:42:08 ghost-blog kernel: [214051.067851] systemd[1]: systemd-journald.service: Consumed 10.720s CPU time.
Aug 26 15:42:11 ghost-blog kernel: [214053.941325] systemd[1]: Starting Journal Service…
Aug 26 15:42:11 ghost-blog kernel: [214053.941567] systemd[1]: Stopped Snap Daemon.
Aug 26 15:42:12 ghost-blog kernel: [214054.572416] systemd[1]: Starting Snap Daemon…
Aug 26 15:43:43 ghost-blog kernel: [214145.230023] systemd[1]: snapd.service: start operation timed out. Terminating.
Aug 26 15:43:51 ghost-blog kernel: [214153.598756] systemd[1]: snapd.service: Failed with result ‘timeout’.
Aug 26 15:43:52 ghost-blog kernel: [214154.440926] systemd-journald[12275]: File /var/log/journal/f66856b0b1c17b0b98a48c18290b07fc/system.journal corrupted or uncleanly shut down, renaming and replacing.
Aug 26 15:43:52 ghost-blog kernel: [214155.445509] systemd[1]: Failed to start Snap Daemon.
Aug 26 15:44:07 ghost-blog kernel: [214169.115247] systemd[1]: systemd-journald.service: start operation timed out. Terminating.
Aug 26 15:45:08 ghost-blog kernel: [214230.918304] systemd[1]: snapd.service: Scheduled restart job, restart counter is at 3.
Aug 26 15:45:23 ghost-blog kernel: [214245.157136] systemd[1]: Stopped Snap Daemon.
Aug 26 15:48:58 ghost-blog kernel: [214459.266112] systemd[1]: Starting Snap Daemon…
Aug 26 15:49:41 ghost-blog node[891]: [2023-08-26 15:42:11] #033[35mWARN#033[39m Job “email-analytics-fetch-latest” is already running
Aug 26 15:49:59 ghost-blog node[891]: #033[35m
Aug 26 15:50:15 ghost-blog node[891]: #033[35mJob “email-analytics-fetch-latest” is already running#033[39m
Aug 26 15:50:35 ghost-blog node[891]: #033[90m----------------------------------------#033[39m
Aug 26 15:50:53 ghost-blog node[891]: #033[90mError: Job “email-analytics-fetch-latest” is already running
Aug 26 15:51:09 ghost-blog node[891]: at Bree.run (/var/www/ghost/versions/5.59.2/node_modules/bree/lib/index.js:324:42)
Aug 26 15:51:30 ghost-blog node[891]: at /var/www/ghost/versions/5.59.2/node_modules/bree/lib/index.js:546:27
Aug 26 15:51:50 ghost-blog node[891]: at Timeout.scheduleTimeout [as _onTimeout] (/var/www/ghost/versions/5.59.2/node_modules/@breejs/later/lib/index.js:1038:7)
Aug 26 15:52:07 ghost-blog node[891]: at listOnTimeout (node:internal/timers:559:17)
looking at journalctl -u snapd.service:
Aug 24 04:14:47 ghost-blog snapd[433]: backends.go:58: AppArmor status: apparmor is enabled and all features>
Aug 24 04:14:47 ghost-blog systemd[1]: Started Snap Daemon.
Aug 24 10:24:50 ghost-blog snapd[433]: storehelpers.go:773: cannot refresh: snap has no updates available: ">
Aug 24 16:09:48 ghost-blog snapd[433]: storehelpers.go:773: cannot refresh: snap has no updates available: ">
Aug 24 22:39:49 ghost-blog snapd[433]: storehelpers.go:773: cannot refresh: snap has no updates available: ">
Aug 25 01:19:48 ghost-blog snapd[433]: storehelpers.go:773: cannot refresh: snap has no updates available: ">
Aug 25 10:39:49 ghost-blog snapd[433]: storehelpers.go:773: cannot refresh: snap has no updates available: ">
Aug 25 10:40:43 ghost-blog snapd[433]: storehelpers.go:773: cannot refresh snap “google-cloud-cli”: snap has>
Aug 25 13:19:49 ghost-blog snapd[433]: storehelpers.go:773: cannot refresh: snap has no updates available: ">
Aug 25 23:39:49 ghost-blog snapd[433]: storehelpers.go:773: cannot refresh: snap has no updates available: ">
Aug 26 02:19:48 ghost-blog snapd[433]: storehelpers.go:773: cannot refresh: snap has no updates available: ">
Aug 26 09:54:50 ghost-blog snapd[433]: storehelpers.go:773: cannot refresh: snap has no updates available: ">
Aug 26 14:49:50 ghost-blog snapd[433]: storehelpers.go:773: cannot refresh: snap has no updates available: ">
Aug 26 15:29:11 ghost-blog systemd[1]: snapd.service: Watchdog timeout (limit 5min)!
Aug 26 15:29:13 ghost-blog systemd[1]: snapd.service: Killing process 433 (snapd) with signal SIGABRT.
Aug 26 15:30:33 ghost-blog snapd[433]: SIGABRT: abort
Aug 26 15:31:39 ghost-blog snapd[433]: PC=0x562000ae89a1 m=0 sigcode=0
Aug 26 15:33:36 ghost-blog snapd[433]: goroutine 0 [idle]:
– Boot 04cd692e1a2c4b1db5d96469477a6167 –
Aug 26 16:00:32 ghost-blog systemd[1]: Starting Snap Daemon…
Aug 26 16:00:41 ghost-blog snapd[447]: overlord.go:272: Acquiring state lock file
Aug 26 16:00:41 ghost-blog snapd[447]: overlord.go:277: Acquired state lock file
Aug 26 16:00:41 ghost-blog snapd[447]: daemon.go:247: started snapd/2.60.2 (series 16; classic) ubuntu/22.04>
Aug 26 16:00:41 ghost-blog snapd[447]: daemon.go:340: adjusting startup timeout by 50s (pessimistic estimate>
Aug 26 16:00:42 ghost-blog snapd[447]: backends.go:58: AppArmor status: apparmor is enabled and all features>
Aug 26 16:00:42 ghost-blog systemd[1]: Started Snap Daemon.