I’ve tried to update Ghost from 3.29.1 to 3.30 and above in the past days and I ran into an issue with the server running out of space and the Ghost updates failing to restart the instance.
I’ve tried to rollback to previous versions and the process failed. Tried to force updates and the process succeeds, but the instance is still not starting. If I restart the server, the instance shows as running, but trying to reach the site in the browser gives 502 gateway error.
In the logs:
Debug Information:
OS: Ubuntu, v18.04.5 LTS
Node Version: v10.22.0
Ghost Version: 3.30.1
Ghost-CLI Version: 1.14.1
Environment: production
Command: 'ghost update'
Message: Could not communicate with Ghost
Suggestion: journalctl -u ghost_learn-t4ngo-io -n 50
Stack: Error: Could not communicate with Ghost
at Server.server.close (/usr/lib/node_modules/ghost-cli/lib/utils/port-polling.js:56:20)
at Object.onceWrapper (events.js:286:20)
at Server.emit (events.js:198:13)
at emitCloseNT (net.js:1619:8)
at process._tickCallback (internal/process/next_tick.js:63:19)
Is there a way to rollback to 2-3 previous versions to get back to the 3.29.1 version which was working? I’m worried that there are either file or database corruptions that happened because of the insufficient disk space.
Note: maybe ghost update should also check for available space (as it checks for available memory) as an empty Ghost site (i.e. a standard installation with only the demo content) can occupy 2GB of disk space with 5 of its rollback versions…
Yes, I did all that, cleaned up the server, removed older Ghost versions and so on. Even with almost 6gb free space, ghost still fails to start, even though upgrades work and ghost doctor says there’s nothing wrong…
Can you run ghost ls & check if ghost services are running or not?
& if service is not running then start ghost ghost start & if running & on browser is still giving error bad gateway then its something related webserver
I’ve mentioned in the initial post that the sites are not running (restarting) and even if I reboot the server, while they show as running, they’re not actually running (give the gatewy 502 error). The only site running is the one I’ve left not upgraded, which is version 3.29.1 (so is not a web server problem, but a Ghost site problem).
I just updated my dev instance to 3.30.1 without any issue…
If you like i can take a look to your issue via teamviewer or similar. If so then you can DM me your teamviewer ID Password. I will be here for next 2 hours
Updates are not generating errors, but probably due to some data corruption (that lack of server space) the ghost instance is not starting anymore. I just updated to 3.30.2 now:
ghost update
+ sudo systemctl is-active ghost_antreprenor-digital
✔ Checking system Node.js version
✔ Ensuring user is not logged in as ghost user
✔ Checking if logged in user is directory owner
✔ Checking current folder permissions
✔ Checking folder permissions
✔ Checking file permissions
✔ Checking content folder ownership
✔ Checking memory availability
✔ Checking for available migrations
✔ Checking for latest Ghost version
✔ Downloading and updating Ghost to v3.30.2
✔ Linking latest Ghost and recording versions
ℹ Removing old Ghost versions [skipped]
But the instance is stopped and when I try to restart it:
ghost restart
+ sudo systemctl is-active ghost_antreprenor-digital
Ghost instance is not running! Starting...
+ sudo systemctl start ghost_antreprenor-digital
+ sudo systemctl stop ghost_antreprenor-digital
✖ Starting Ghost
A GhostError occurred.
Message: Could not communicate with Ghost
Suggestion: journalctl -u ghost_antreprenor-digital -n 50
Debug Information:
OS: Ubuntu, v18.04.5 LTS
Node Version: v10.22.0
Ghost Version: 3.30.2
Ghost-CLI Version: 1.14.1
Environment: production
Command: 'ghost restart'
Additional log info available in: /home/ubuntu/.ghost/logs/ghost-cli-debug-2020-08-21T11_27_29_203Z.log
Try running ghost doctor to check your system for known issues.
Even though now there is free space on the server (5gb) and ghost doctors shows everything to be fine.
On another site update:
? Unable to upgrade Ghost from v3.30.1 to v3.30.2. Would you like to revert back to v3.30.1? No
I tried that as well, upgraded nodejs from 10.x to 12.x and still the issue persists. I am thinking of migrating the sites to a new server (upgrade ubuntu 18 to 20 in the process) but unsure if the data corruption at db level will allow me to do this or not. Some of the sites are not important as I can just recreate those (demo sites), but I have a couple that have content that I don’t want to lose.
@dsecareanu this is definitely something we’re looking to improve in the CLI. Can you share the ghost logs (ghost log / ghost log -e), and /home/ubuntu/.ghost/logs/ghost-cli-debug-2020-08-21T11_27_29_203Z.log for more context? Thanks!
Yes, but I would like to send more logs (from the initial failed attempts as well), can I send them at support@ghost.org?
Debug Information:
OS: Ubuntu, v18.04.5 LTS
Node Version: v10.22.0
Ghost Version: 3.30.2
Ghost-CLI Version: 1.14.1
Environment: production
Command: 'ghost restart'
Message: Could not communicate with Ghost
Suggestion: journalctl -u ghost_antreprenor-digital -n 50
Stack: Error: Could not communicate with Ghost
at Server.server.close (/usr/lib/node_modules/ghost-cli/lib/utils/port-polling.js:56:20)
at Object.onceWrapper (events.js:286:20)
at Server.emit (events.js:198:13)
at emitCloseNT (net.js:1619:8)
at process._tickCallback (internal/process/next_tick.js:63:19)
ghost log -e:
An error occurred.
Message: 'Cannot read property 'join' of undefined'
Debug Information:
OS: Ubuntu, v18.04.5 LTS
Node Version: v12.18.3
Ghost-CLI Version: 1.14.1
Environment: production
Command: 'ghost log -e'
Additional log info available in: /home/ubuntu/.ghost/logs/ghost-cli-debug-2020-08-21T15_14_08_343Z.log
Try running ghost doctor to check your system for known issues.
You can always refer to https://ghost.org/docs/api/ghost-cli/ for troubleshooting.
I’m not part of the Ghost Foundation, so that’s out of my purview I can definitely work with you to improve the messaging / debug info provided by the CLI, but broken rollback migrations are not something I can do much about
Okies, I’ve sent the logs there as well, just in case.
Sure, if you want we can connect and go through the cli together a bit (me from a user’s perspective) as I think that there are some potential improvements:
available space on disk check (to avoid this kind of issue I ran into and data corruption in general)
rollback option to choose the version (not sure why 5 versions are kept and if you can actually rollback to an older version, i.e. -2-5 rather than just 1 version backwards
maybe even go towards a ghost backup option (as it should be easy to archive local folder and dump the sql database to provide a backup functionality)
unsure if this qualifies, but I’ve been searching on google to find an ansible driven way to update ghost instances (similar with updating entire servers) and I could only find a stretched command like ansible hostname -m shell -a “cd /var/www/domain.com/ && ghost update”, which works for individual sites, but doesn’t work for multiple sites like a bash script of this sort: for d in /var/www/*/ ; do (cd “$d” && ghost update); done
probably there are other improvements that can be done outside of the ones listed above
Hey @dsecareanu, those are definitely things we can look into:
available space on disk check (to avoid this kind of issue I ran into and data corruption in general)
this should be fairly straightforward, I’ll create an issue for this in the Ghost-CLI repository
rollback option to choose the version (not sure why 5 versions are kept and if you can actually rollback to an older version, i.e. -2-5 rather than just 1 version backwards
This change is already in the main repo branch (see commit), just hasn’t been released in a version yet.
maybe even go towards a ghost backup option (as it should be easy to archive local folder and dump the sql database to provide a backup functionality)
We’ve got an early issue tracking this in the repository - need to figure out the best way to design the command before we get it implemented.
unsure if this qualifies, but I’ve been searching on google to find an ansible driven way to update ghost instances (similar with updating entire servers) and I could only find a stretched command like ansible hostname -m shell -a “cd /var/www/domain.com/ && ghost update”, which works for individual sites, but doesn’t work for multiple sites like a bash script of this sort: for d in /var/www/*/ ; do (cd “$d” && ghost update); done
maybe we could look into adding some sort of ghost update --all? which would go through all ghost instances on the system and update them. Honestly not super familiar with Ansible so I’m not 100% sure what would make that process easier.
Question: what would be a good way to recover sites that do not start anymore? This means I cannot export the content json file anymore, but I have access to content folder and the database. Thanks!
An update: I’ve tried to update one blog that wasn’t updated/broken and even though there was enough space on the server, the update (from 3.29.1 to 3.33.0) failed with the same issue of being unable to restart ghost.
Anyone has any ideas how to recover these ghost sites? Update or rollbacks do not work, ghost doesn’t restart… Error is:
Debug Information:
OS: Ubuntu, v20.04.1 LTS
Node Version: v12.18.4
Ghost Version: 3.33.0
Ghost-CLI Version: 1.14.1
Environment: production
Command: 'ghost restart'
Message: Could not communicate with Ghost
Suggestion: journalctl -u ghost_codruvrabie-eu -n 50
Stack: Error: Could not communicate with Ghost
at Server.<anonymous> (/usr/lib/node_modules/ghost-cli/lib/utils/port-polling.js:56:20)
at Object.onceWrapper (events.js:421:28)
at Server.emit (events.js:315:20)
at emitCloseNT (net.js:1654:8)
at processTicksAndRejections (internal/process/task_queues.js:83:21)
As it appears to be an issue with the database, how can I export from sql to json to be able to reimport the content? Found a bunch of .json files in content/data, can I use these ones for a content restore?
Ok, I think I found my answer trying different things:
I’ve reinstalled a new ghost site on a new server
I’ve moved the content folder (most of it, the important parts, i.e. images, settings, themes)
I’ve backed up the newly installed ghost site database and then dropped all tables
I’ve done a database restore from the old site (which I wasn’t sure would work, but it seems it does)
I’ve restarted ghost
It seems there was no data corruption at database level, but the issues with ghost not restarting were somewhere else (maybe in some files rather, though a ghost update --force should have solved that somehow).
I can definitely work with you to improve the messaging / debug info provided by the CLI, but broken rollback migrations are not something I can do much about