SystemError occurred when running ghost start: “Systemd process manager has not been set up or is corrupted.”

I have multiple instances of Ghost running on a DigitalOcean Droplet. When going through the backup process, I was unable to backup the sites because they weren’t running. When I tried to start the sites, I received this error. This same error prevents start, backup, and update. I even attempted to restore the Droplet backup, but encountered the same errors.

Because I wasn’t able to complete a backup, I would rather not reinstall Ghost. But it seems that may be the only way I can restore my sites. I have older local versions, but I will lose data. I was able to successfully backup the MySQL databases.

stephen@builders:~$ cd /var/www/designadmin.com/
stephen@builders:/var/www/designadmin.com$ ghost start

Love open source? We’re hiring Node.js Engineers to work on Ghost full-time.
https://careers.ghost.org/product-engineer-node-js


+ sudo systemctl is-active ghost_designadmin-com
? Sudo Password [hidden]
✔ Checking system Node.js version - found v16.15.1
✔ Ensuring user is not logged in as ghost user
✔ Checking if logged in user is directory owner
✔ Checking current folder permissions
+ sudo systemctl is-active ghost_designadmin-com
✔ Validating config
✔ Checking folder permissions
✔ Checking file permissions
✔ Checking memory availability
✔ Checking binary dependencies
✔ Checking systemd unit file
✔ Checking systemd node version - found v16.15.1
✖ Starting Ghost: designadmin-com
A SystemError occurred.

Message: Systemd process manager has not been set up or is corrupted.
Help: Run ghost setup linux-user systemd and try again.

Debug Information:
    OS: Ubuntu, v20.04.4 LTS
    Node Version: v16.15.1
    Ghost Version: 5.2.2
    Ghost-CLI Version: 1.21.0
    Environment: production
    Command: 'ghost start'

Try running ghost doctor to check your system for known issues.

You can always refer to https://ghost.org/docs/ghost-cli/ for troubleshooting.

Can you run ghost doctor or ghost setup systemd?

Here is the output from ghost doctor:

stephen@builders:~$ cd /var/www/stephenbau.com
stephen@builders:/var/www/stephenbau.com$ ghost doctor

Love open source? We’re hiring Node.js Engineers to work on Ghost full-time.
https://careers.ghost.org/product-engineer-node-js


✔ Checking system Node.js version - found v16.15.1
✔ Checking logged in user
✔ Ensuring user is not logged in as ghost user
✔ Checking if logged in user is directory owner
✔ Checking current folder permissions
✔ Checking system compatibility
✔ Checking for a MySQL installation
+ sudo systemctl is-active ghost_stephenbau-com
? Sudo Password [hidden]
✔ Validating config
✔ Checking folder permissions
✔ Checking file permissions
✔ Checking memory availability
✔ Checking binary dependencies
✔ Checking free space
✔ Checking systemd unit file
✖ Checking systemd node version - found v16.15.1
One or more errors occurred.

1) SystemError

Message: Ghost v4.19.1 is not compatible with Node v16.15.1
Help: Check the version of Node configured in /lib/systemd/system/ghost_stephenbau-com.service and update it to a compatible version

Debug Information:
    OS: Ubuntu, v20.04.4 LTS
    Node Version: v16.15.1
    Ghost Version: 4.19.1
    Ghost-CLI Version: 1.21.0
    Environment: production
    Command: 'ghost doctor'

Try running ghost doctor to check your system for known issues.

You can always refer to https://ghost.org/docs/ghost-cli/ for troubleshooting.

Here is the output from ghost setup systemd:

stephen@builders:/var/www/stephenbau.com$ ghost setup systemd

Love open source? We’re hiring Node.js Engineers to work on Ghost full-time.
https://careers.ghost.org/product-engineer-node-js



Systemd service has already been set up. Skipping Systemd setup
ℹ Setting up Systemd [skipped]

I have several sites running on this server. I was trying to update Ubuntu for Ghost 5.0. I gave up on reviving my old sites, because I was intending to create a new instance running Ghost 5.0. I should have created a new droplet, but my intention was to update all the sites to 5.0 and things failed horribly. However, I was able to successfully install Ghost 5.2.2 for Luna Solterra.

stephen@builders:/var/www/stephenbau.com$ ghost ls

Love open source? We’re hiring Node.js Engineers to work on Ghost full-time.
https://careers.ghost.org/product-engineer-node-js


+ sudo systemctl is-active ghost_builderscollective-com
+ sudo systemctl is-active ghost_designadmin-com
+ sudo systemctl is-active ghost_faithhopelove-ca
+ sudo systemctl is-active ghost_imaginaxiom-com
+ sudo systemctl is-active ghost_socialarc-com
+ sudo systemctl is-active ghost_designinfluences-com
+ sudo systemctl is-active ghost_stephenbau-com
+ sudo systemctl is-active ghost_therecipebox-club
+ sudo systemctl is-active ghost_stopresetgo-builderscollective-org
+ sudo systemctl is-active ghost_world-builderscollective-org
+ sudo systemctl is-active ghost_lunasolterra-art
┌────────────────────────────────────┬─────────────────────────────────────────────┬─────────┬──────────────────────┬────────────────────────────────────────────┬──────┬─────────────────┐
│ Name                               │ Location                                    │ Version │ Status               │ URL                                        │ Port │ Process Manager │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ builderscollective-com             │ /var/www/builderscollective.com             │ 4.19.1  │ running (production) │ https://builderscollective.com             │ 2369 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ designadmin-com                    │ /var/www/designadmin.com                    │ 5.2.2   │ stopped              │ n/a                                        │ n/a  │ n/a             │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ faithhopelove-ca                   │ /var/www/faithhopelove.ca                   │ 4.19.1  │ stopped              │ n/a                                        │ n/a  │ n/a             │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ imaginaxiom-com                    │ /var/www/imaginaxiom.com                    │ 4.19.1  │ running (production) │ https://imaginaxiom.com                    │ 2372 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ socialarc-com                      │ /var/www/socialarc.com                      │ 4.19.1  │ stopped              │ n/a                                        │ n/a  │ n/a             │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ designinfluences-com               │ /var/www/designinfluences.com               │ 4.19.1  │ stopped              │ n/a                                        │ n/a  │ n/a             │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ stephenbau-com                     │ /var/www/stephenbau.com                     │ 4.19.1  │ stopped              │ n/a                                        │ n/a  │ n/a             │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ therecipebox-club                  │ /var/www/therecipebox.club                  │ 4.19.1  │ stopped              │ n/a                                        │ n/a  │ n/a             │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ stopresetgo-builderscollective-org │ /var/www/stopresetgo.builderscollective.org │ 4.19.1  │ running (production) │ https://stopresetgo.builderscollective.org │ 2377 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ world-builderscollective-org       │ /var/www/world.builderscollective.org       │ 4.20.1  │ running (production) │ https://world.builderscollective.org       │ 2378 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ lunasolterra-art                   │ /var/www/lunasolterra.art                   │ 5.2.2   │ running (production) │ https://lunasolterra.art                   │ 2368 │ systemd         │
└────────────────────────────────────┴─────────────────────────────────────────────┴─────────┴──────────────────────┴────────────────────────────────────────────┴──────┴─────────────────┘

You need to update to the latest V4 first, and make sure Node is compatible with Ghost 4.19.1, i.e., 14.16.1.

Downgrade Node as follows:


sudo apt remove -y nodejs
curl -sL https://deb.nodesource.com/setup_14.x | sudo -E bash -
sudo apt install -y nodejs

I tried to follow the downgrade instructions, but that did not seem to make a difference.

I have gone back to the oldest available backup to see if that helps. The first ghost ls returned a status of running for all domains.

stephen@builders:~$ ghost ls
+ sudo systemctl is-active ghost_builderscollective-com
+ sudo systemctl is-active ghost_designadmin-com
+ sudo systemctl is-active ghost_faithhopelove-ca
+ sudo systemctl is-active ghost_imaginaxiom-com
+ sudo systemctl is-active ghost_socialarc-com
+ sudo systemctl is-active ghost_designinfluences-com
+ sudo systemctl is-active ghost_stephenbau-com
+ sudo systemctl is-active ghost_therecipebox-club
+ sudo systemctl is-active ghost_stopresetgo-builderscollective-org
+ sudo systemctl is-active ghost_world-builderscollective-org
? Sudo Password [hidden]
┌────────────────────────────────────┬─────────────────────────────────────────────┬─────────┬──────────────────────┬────────────────────────────────────────────┬──────┬─────────────────┐
│ Name                               │ Location                                    │ Version │ Status               │ URL                                        │ Port │ Process Manager │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ builderscollective-com             │ /var/www/builderscollective.com             │ 4.19.1  │ running (production) │ https://builderscollective.com             │ 2369 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ designadmin-com                    │ /var/www/designadmin.com                    │ 4.19.1  │ running (production) │ https://designadmin.com                    │ 2370 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ faithhopelove-ca                   │ /var/www/faithhopelove.ca                   │ 4.19.1  │ running (production) │ https://faithhopelove.ca                   │ 2371 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ imaginaxiom-com                    │ /var/www/imaginaxiom.com                    │ 4.19.1  │ running (production) │ https://imaginaxiom.com                    │ 2372 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ socialarc-com                      │ /var/www/socialarc.com                      │ 4.19.1  │ running (production) │ https://socialarc.com                      │ 2373 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ designinfluences-com               │ /var/www/designinfluences.com               │ 4.19.1  │ running (production) │ https://designinfluences.com               │ 2374 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ stephenbau-com                     │ /var/www/stephenbau.com                     │ 4.19.1  │ running (production) │ https://stephenbau.com                     │ 2375 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ therecipebox-club                  │ /var/www/therecipebox.club                  │ 4.19.1  │ running (production) │ https://therecipebox.club                  │ 2376 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ stopresetgo-builderscollective-org │ /var/www/stopresetgo.builderscollective.org │ 4.19.1  │ running (production) │ https://stopresetgo.builderscollective.org │ 2377 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ world-builderscollective-org       │ /var/www/world.builderscollective.org       │ 4.20.1  │ running (production) │ https://world.builderscollective.org       │ 2378 │ systemd         │
└────────────────────────────────────┴─────────────────────────────────────────────┴─────────┴──────────────────────┴────────────────────────────────────────────┴──────┴─────────────────┘

I waited for the servers to start, and a few minutes later, ran ghost ls again, and the same servers that I had run ghost stop after this backup were still not able to start.

stephen@builders:~$ ghost ls
+ sudo systemctl is-active ghost_builderscollective-com
+ sudo systemctl is-active ghost_designadmin-com
+ sudo systemctl is-active ghost_faithhopelove-ca
+ sudo systemctl is-active ghost_imaginaxiom-com
+ sudo systemctl is-active ghost_socialarc-com
+ sudo systemctl is-active ghost_designinfluences-com
+ sudo systemctl is-active ghost_stephenbau-com
+ sudo systemctl is-active ghost_therecipebox-club
+ sudo systemctl is-active ghost_stopresetgo-builderscollective-org
+ sudo systemctl is-active ghost_world-builderscollective-org
+ sudo systemctl reset-failed ghost_designadmin-com
+ sudo systemctl reset-failed ghost_faithhopelove-ca
+ sudo systemctl reset-failed ghost_socialarc-com
+ sudo systemctl reset-failed ghost_designinfluences-com
+ sudo systemctl reset-failed ghost_stephenbau-com
+ sudo systemctl reset-failed ghost_therecipebox-club
┌────────────────────────────────────┬─────────────────────────────────────────────┬─────────┬──────────────────────┬────────────────────────────────────────────┬──────┬─────────────────┐
│ Name                               │ Location                                    │ Version │ Status               │ URL                                        │ Port │ Process Manager │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ builderscollective-com             │ /var/www/builderscollective.com             │ 4.19.1  │ running (production) │ https://builderscollective.com             │ 2369 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ designadmin-com                    │ /var/www/designadmin.com                    │ 4.19.1  │ stopped              │ n/a                                        │ n/a  │ n/a             │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ faithhopelove-ca                   │ /var/www/faithhopelove.ca                   │ 4.19.1  │ stopped              │ n/a                                        │ n/a  │ n/a             │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ imaginaxiom-com                    │ /var/www/imaginaxiom.com                    │ 4.19.1  │ running (production) │ https://imaginaxiom.com                    │ 2372 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ socialarc-com                      │ /var/www/socialarc.com                      │ 4.19.1  │ stopped              │ n/a                                        │ n/a  │ n/a             │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ designinfluences-com               │ /var/www/designinfluences.com               │ 4.19.1  │ stopped              │ n/a                                        │ n/a  │ n/a             │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ stephenbau-com                     │ /var/www/stephenbau.com                     │ 4.19.1  │ stopped              │ n/a                                        │ n/a  │ n/a             │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ therecipebox-club                  │ /var/www/therecipebox.club                  │ 4.19.1  │ stopped              │ n/a                                        │ n/a  │ n/a             │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ stopresetgo-builderscollective-org │ /var/www/stopresetgo.builderscollective.org │ 4.19.1  │ running (production) │ https://stopresetgo.builderscollective.org │ 2377 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ world-builderscollective-org       │ /var/www/world.builderscollective.org       │ 4.20.1  │ running (production) │ https://world.builderscollective.org       │ 2378 │ systemd         │
└────────────────────────────────────┴─────────────────────────────────────────────┴─────────┴──────────────────────┴────────────────────────────────────────────┴──────┴─────────────────┘

If I run ghost doctor on the sites, everything returns fine. But, if I try to start the stopped sites, I get the error.

stephen@builders:/var/www/builderscollective.com$ ghost doctor
✔ Checking system Node.js version - found v14.16.1
✔ Checking logged in user
✔ Ensuring user is not logged in as ghost user
✔ Checking if logged in user is directory owner
✔ Checking current folder permissions
✔ Checking system compatibility
✔ Checking for a MySQL installation
+ sudo systemctl is-active ghost_builderscollective-com
Instance is currently running
ℹ Validating config [skipped]
✔ Checking folder permissions
✔ Checking file permissions
✔ Checking content folder ownership
✔ Checking memory availability
✔ Checking binary dependencies
✔ Checking free space
✔ Checking systemd unit file
✔ Checking systemd node version - found v14.16.1
stephen@builders:/var/www/builderscollective.com$ cd ../designadmin.com
stephen@builders:/var/www/designadmin.com$ ghost doctor
✔ Checking system Node.js version - found v14.16.1
✔ Checking logged in user
✔ Ensuring user is not logged in as ghost user
✔ Checking if logged in user is directory owner
✔ Checking current folder permissions
✔ Checking system compatibility
✔ Checking for a MySQL installation
+ sudo systemctl is-active ghost_designadmin-com
✔ Validating config
✔ Checking folder permissions
✔ Checking file permissions
✔ Checking memory availability
✔ Checking binary dependencies
✔ Checking free space
✔ Checking systemd unit file
✔ Checking systemd node version - found v14.16.1
stephen@builders:/var/www/designadmin.com$ ghost start
+ sudo systemctl is-active ghost_designadmin-com
✔ Checking system Node.js version - found v14.16.1
✔ Ensuring user is not logged in as ghost user
✔ Checking if logged in user is directory owner
✔ Checking current folder permissions
+ sudo systemctl is-active ghost_designadmin-com
✔ Validating config
✔ Checking folder permissions
✔ Checking file permissions
✔ Checking memory availability
✔ Checking binary dependencies
✔ Checking systemd unit file
✔ Checking systemd node version - found v14.16.1
✖ Starting Ghost: designadmin-com
A SystemError occurred.

Message: Systemd process manager has not been set up or is corrupted.
Help: Run ghost setup linux-user systemd and try again.

Debug Information:
    OS: Ubuntu, v20.04.2 LTS
    Node Version: v14.16.1
    Ghost Version: 4.19.1
    Ghost-CLI Version: 1.17.4
    Environment: production
    Command: 'ghost start'

Try running ghost doctor to check your system for known issues.

You can always refer to https://ghost.org/docs/ghost-cli/ for troubleshooting.

Have you tried ghost setup systemd or ghost setup linux-user systemd for those sites?

Yes, I tried both.


stephen@builders:/var/www/designadmin.com$ ghost start
+ sudo systemctl is-active ghost_designadmin-com
✔ Checking system Node.js version - found v14.16.1
✔ Ensuring user is not logged in as ghost user
✔ Checking if logged in user is directory owner
✔ Checking current folder permissions
+ sudo systemctl is-active ghost_designadmin-com
✔ Validating config
✔ Checking folder permissions
✔ Checking file permissions
✔ Checking memory availability
✔ Checking binary dependencies
✔ Checking systemd unit file
✔ Checking systemd node version - found v14.16.1
✖ Starting Ghost: designadmin-com
A SystemError occurred.

Message: Systemd process manager has not been set up or is corrupted.
Help: Run ghost setup linux-user systemd and try again.

Debug Information:
    OS: Ubuntu, v20.04.2 LTS
    Node Version: v14.16.1
    Ghost Version: 4.19.1
    Ghost-CLI Version: 1.17.4
    Environment: production
    Command: 'ghost start'

Try running ghost doctor to check your system for known issues.

You can always refer to https://ghost.org/docs/ghost-cli/ for troubleshooting.
stephen@builders:/var/www/designadmin.com$ ghost setup linux-user systemd
You are running an outdated version of Ghost-CLI.
It is recommended that you upgrade before continuing.
Run `npm install -g ghost-cli@latest` to upgrade.

+ sudo chown -R ghost:ghost /var/www/designadmin.com/content
? Sudo Password [hidden]
✔ Setting up "ghost" system user
Systemd service has already been set up. Skipping Systemd setup
ℹ Setting up Systemd [skipped]
stephen@builders:/var/www/designadmin.com$ cd content/
stephen@builders:/var/www/designadmin.com/content$ ll
total 32
drwxrwxr-x 8 ghost   ghost   4096 Oct 19  2021 ./
drwxrwxr-x 5 stephen stephen 4096 Oct 20  2021 ../
drwxrwxr-x 2 ghost   ghost   4096 Oct 19  2021 apps/
drwxrwxr-x 2 ghost   ghost   4096 Oct 20  2021 data/
drwxrwxr-x 4 ghost   ghost   4096 Oct 20  2021 images/
drwxrwxr-x 2 ghost   ghost   4096 Oct 19  2021 logs/
drwxrwxr-x 2 ghost   ghost   4096 Oct 19  2021 settings/
drwxrwxr-x 2 ghost   ghost   4096 Oct 19  2021 themes/
stephen@builders:/var/www/designadmin.com/content$ cd ..
stephen@builders:/var/www/designadmin.com$ sudo chown -R stephen:stephen /var/www/designadmin.com/content
stephen@builders:/var/www/designadmin.com$ ll
total 28
drwxrwxr-x  5 stephen stephen 4096 Oct 20  2021 ./
drwxr-xr-x 14 stephen stephen 4096 Oct 23  2021 ../
-rw-rw-r--  1 stephen stephen  112 Jun  7 18:48 .ghost-cli
-rw-rw-r--  1 stephen stephen  547 Oct 19  2021 config.production.json
drwxrwxr-x  8 stephen stephen 4096 Oct 19  2021 content/
lrwxrwxrwx  1 stephen stephen   40 Oct 19  2021 current -> /var/www/designadmin.com/versions/4.19.1/
drwxr-xr-x  3 stephen stephen 4096 Oct 19  2021 system/
drwxrwxr-x  3 stephen stephen 4096 Oct 19  2021 versions/

I suspect this is a MySQL database issue with this solution.

1 Like

After running that, did you try to start Ghost?

You could also check the startup status with systemctl list-unit-files --type=service --state=enabled. Are all sites listed? Is vendor enabled for each? And finally, the output from ls -Flai /lib/systemd/system/ghost* will tell you which user runs each Ghost site.

I don’t see how since you’re not getting those errors.

Yes. I tried running ghost start and it failed with the same error.

Here is the output when checking the startup status as you suggested.

stephen@builders:/var/www$ ls -Flai /lib/systemd/system/ghost*
1054363 -rw-rw-r-- 1 stephen stephen 323 Oct 19  2021 /lib/systemd/system/ghost_builderscollective-com.service
1054517 -rw-rw-r-- 1 stephen stephen 309 Oct 19  2021 /lib/systemd/system/ghost_designadmin-com.service
1055247 -rw-rw-r-- 1 stephen stephen 319 Oct 19  2021 /lib/systemd/system/ghost_designinfluences-com.service
1056765 -rw-rw-r-- 1 stephen stephen 311 Oct 19  2021 /lib/systemd/system/ghost_faithhopelove-ca.service
1057726 -rw-rw-r-- 1 stephen stephen 309 Oct 19  2021 /lib/systemd/system/ghost_imaginaxiom-com.service
 782171 -rw-rw-r-- 1 stephen stephen 305 Oct 19  2021 /lib/systemd/system/ghost_socialarc-com.service
1568185 -rw-rw-r-- 1 root    root    314 Oct 17  2021 /lib/systemd/system/ghost_staging-builderscollective-com.service
 782180 -rw-rw-r-- 1 stephen stephen 307 Oct 19  2021 /lib/systemd/system/ghost_stephenbau-com.service
 784745 -rw-rw-r-- 1 stephen stephen 347 Oct 20  2021 /lib/systemd/system/ghost_stopresetgo-builderscollective-org.service
 782188 -rw-rw-r-- 1 stephen stephen 313 Oct 19  2021 /lib/systemd/system/ghost_therecipebox-club.service
 784862 -rw-rw-r-- 1 stephen stephen 335 Oct 23  2021 /lib/systemd/system/ghost_world-builderscollective-org.service
stephen@builders:/var/www$ systemctl list-unit-files --type=service --state=enabled
UNIT FILE                                        STATE   VENDOR PRESET
accounts-daemon.service                          enabled enabled      
apparmor.service                                 enabled enabled      
atd.service                                      enabled enabled      
autovt@.service                                  enabled enabled      
blk-availability.service                         enabled enabled      
cloud-config.service                             enabled enabled      
cloud-final.service                              enabled enabled      
cloud-init-local.service                         enabled enabled      
cloud-init.service                               enabled enabled      
console-setup.service                            enabled enabled      
cron.service                                     enabled enabled      
dbus-org.freedesktop.resolve1.service            enabled enabled      
dbus-org.freedesktop.timesync1.service           enabled enabled      
dmesg.service                                    enabled enabled      
droplet-agent.service                            enabled enabled      
e2scrub_reap.service                             enabled enabled      
finalrd.service                                  enabled enabled      
getty@.service                                   enabled enabled      
ghost_builderscollective-com.service             enabled enabled      
ghost_designadmin-com.service                    enabled enabled      
ghost_designinfluences-com.service               enabled enabled      
ghost_faithhopelove-ca.service                   enabled enabled      
ghost_imaginaxiom-com.service                    enabled enabled      
ghost_socialarc-com.service                      enabled enabled      
ghost_staging-builderscollective-com.service     enabled enabled      
ghost_stephenbau-com.service                     enabled enabled      
ghost_stopresetgo-builderscollective-org.service enabled enabled      
ghost_therecipebox-club.service                  enabled enabled      
ghost_world-builderscollective-org.service       enabled enabled      
grub-initrd-fallback.service                     enabled enabled      
irqbalance.service                               enabled enabled      
iscsi.service                                    enabled enabled      
keyboard-setup.service                           enabled enabled      
lvm2-monitor.service                             enabled enabled      
lxd-agent-9p.service                             enabled enabled      
lxd-agent.service                                enabled enabled      
multipath-tools.service                          enabled enabled      
multipathd.service                               enabled enabled      
mysql.service                                    enabled enabled      
networkd-dispatcher.service                      enabled enabled      
nginx.service                                    enabled enabled      
ondemand.service                                 enabled enabled      
open-iscsi.service                               enabled enabled      
open-vm-tools.service                            enabled enabled      
pollinate.service                                enabled enabled      
postfix.service                                  enabled enabled      
qemu-kvm.service                                 enabled enabled      
rsync.service                                    enabled enabled      
rsyslog.service                                  enabled enabled      
secureboot-db.service                            enabled enabled      
setvtrgb.service                                 enabled enabled      
snap.lxd.activate.service                        enabled enabled      
snapd.apparmor.service                           enabled enabled      
snapd.autoimport.service                         enabled enabled      
snapd.core-fixup.service                         enabled enabled      
snapd.recovery-chooser-trigger.service           enabled enabled      
snapd.seeded.service                             enabled enabled      
snapd.service                                    enabled enabled      
snapd.system-shutdown.service                    enabled enabled      
ssh.service                                      enabled enabled      
sshd.service                                     enabled enabled      
syslog.service                                   enabled enabled      
systemd-networkd-wait-online.service             enabled enabled      
systemd-networkd.service                         enabled enabled      
systemd-pstore.service                           enabled enabled      
systemd-resolved.service                         enabled enabled      
systemd-timesyncd.service                        enabled enabled      
ufw.service                                      enabled enabled      
unattended-upgrades.service                      enabled enabled      
vgauth.service                                   enabled enabled      
vmtoolsd.service                                 enabled enabled      

71 unit files listed.

Yes, nothing untoward there. I’m out of ideas. Hopefully, someone else can chip in?

I was just checking the collation values, and there does seem to be a mismatch.

mysql> SELECT SCHEMA_NAME DEFAULT_CHARACTER_SET_NAME, DEFAULT_COLLATION_NAME FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME='therecipebox_club_prod';
+----------------------------+------------------------+
| DEFAULT_CHARACTER_SET_NAME | DEFAULT_COLLATION_NAME |
+----------------------------+------------------------+
| therecipebox_club_prod     | utf8mb4_general_ci     |
+----------------------------+------------------------+
1 row in set (0.00 sec)

mysql> use therecipebox_club_prod;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show table status where name like 'members';
+---------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+--------------------+----------+----------------+---------+
| Name    | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time         | Update_time | Check_time | Collation          | Checksum | Create_options | Comment |
+---------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+--------------------+----------+----------------+---------+
| members | InnoDB |      10 | Dynamic    |    0 |              0 |       16384 |               0 |            0 |         0 |           NULL | 2021-10-19 17:08:29 | NULL        | NULL       | utf8mb4_0900_ai_ci |     NULL |                |         |
+---------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+--------------------+----------+----------------+---------+
1 row in set (0.00 sec)

Thank you for the suggestions you offered. I appreciate the effort.

1 Like

Okay, that makes sense. I’m seeing utf8mb4_0900_ai_ci throughout. This is the new default for MySQL 8.0, so that’s what you need to change.

The problem was finally resolved by creating a snapshot of the DigitalOcean Droplet of the backup when I had all the sites working. I used that snapshot to create a new Droplet. I waited for the DNS to resolve, then ran ghost ls and it showed that all the sites were running.

stephen@bau:/var/www/stephenbau.com$ ghost ls
+ sudo systemctl is-active ghost_builderscollective-com
+ sudo systemctl is-active ghost_designadmin-com
+ sudo systemctl is-active ghost_faithhopelove-ca
+ sudo systemctl is-active ghost_imaginaxiom-com
+ sudo systemctl is-active ghost_socialarc-com
+ sudo systemctl is-active ghost_designinfluences-com
+ sudo systemctl is-active ghost_stephenbau-com
+ sudo systemctl is-active ghost_therecipebox-club
+ sudo systemctl is-active ghost_stopresetgo-builderscollective-org
+ sudo systemctl is-active ghost_world-builderscollective-org
┌────────────────────────────────────┬─────────────────────────────────────────────┬─────────┬──────────────────────┬────────────────────────────────────────────┬──────┬─────────────────┐
│ Name                               │ Location                                    │ Version │ Status               │ URL                                        │ Port │ Process Manager │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ builderscollective-com             │ /var/www/builderscollective.com             │ 4.19.1  │ running (production) │ https://builderscollective.com             │ 2369 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ designadmin-com                    │ /var/www/designadmin.com                    │ 4.19.1  │ running (production) │ https://designadmin.com                    │ 2370 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ faithhopelove-ca                   │ /var/www/faithhopelove.ca                   │ 4.19.1  │ running (production) │ https://faithhopelove.ca                   │ 2371 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ imaginaxiom-com                    │ /var/www/imaginaxiom.com                    │ 4.19.1  │ running (production) │ https://imaginaxiom.com                    │ 2372 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ socialarc-com                      │ /var/www/socialarc.com                      │ 4.19.1  │ running (production) │ https://socialarc.com                      │ 2373 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ designinfluences-com               │ /var/www/designinfluences.com               │ 4.19.1  │ running (production) │ https://designinfluences.com               │ 2374 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ stephenbau-com                     │ /var/www/stephenbau.com                     │ 4.19.1  │ running (production) │ https://stephenbau.com                     │ 2375 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ therecipebox-club                  │ /var/www/therecipebox.club                  │ 4.19.1  │ running (production) │ https://therecipebox.club                  │ 2376 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ stopresetgo-builderscollective-org │ /var/www/stopresetgo.builderscollective.org │ 4.19.1  │ running (production) │ https://stopresetgo.builderscollective.org │ 2377 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ world-builderscollective-org       │ /var/www/world.builderscollective.org       │ 4.20.1  │ running (production) │ https://world.builderscollective.org       │ 2378 │ systemd         │
└────────────────────────────────────┴─────────────────────────────────────────────┴─────────┴──────────────────────┴────────────────────────────────────────────┴──────┴─────────────────┘

But on the frontend, I was still getting a 502 Bad Gateway error. I had thought that the DNS still hadn’t resolved, but when I was checking the IP address with ping, the DNS had definitely resolved.

I discovered that the same sites that had failed on the other Droplet were not actually running. I found that ghost stop failed for these sites.

stephen@bau:/var/www/stephenbau.com$ ghost stop
+ sudo systemctl is-active ghost_stephenbau-com
✖ Stopping Ghost: stephenbau-com
A SystemError occurred.

Message: Systemd process manager has not been set up or is corrupted.
Help: Run ghost setup linux-user systemd and try again.

Debug Information:
    OS: Ubuntu, v20.04.2 LTS
    Node Version: v14.16.1
    Ghost Version: 4.19.1
    Ghost-CLI Version: 1.17.4
    Environment: production
    Command: 'ghost stop'

Try running ghost doctor to check your system for known issues.

You can always refer to https://ghost.org/docs/ghost-cli/ for troubleshooting.
stephen@bau:/var/www/stephenbau.com$ ghost setup linux-user systemd
You are running an outdated version of Ghost-CLI.
It is recommended that you upgrade before continuing.
Run `npm install -g ghost-cli@latest` to upgrade.

+ sudo chown -R ghost:ghost /var/www/stephenbau.com/content
✔ Setting up "ghost" system user
Systemd service has already been set up. Skipping Systemd setup
ℹ Setting up Systemd [skipped]
stephen@bau:/var/www/stephenbau.com$ ghost start
+ sudo systemctl is-active ghost_stephenbau-com

Ghost is already running! For more information, run: 

    ghost ls

stephen@bau:/var/www/stephenbau.com$ ghost stop
+ sudo systemctl is-active ghost_stephenbau-com
+ sudo systemctl stop ghost_stephenbau-com
✔ Stopping Ghost: stephenbau-com
stephen@bau:/var/www/stephenbau.com$ ghost ls
+ sudo systemctl is-active ghost_builderscollective-com
+ sudo systemctl is-active ghost_designadmin-com
+ sudo systemctl is-active ghost_faithhopelove-ca
+ sudo systemctl is-active ghost_imaginaxiom-com
+ sudo systemctl is-active ghost_socialarc-com
+ sudo systemctl is-active ghost_designinfluences-com
+ sudo systemctl is-active ghost_stephenbau-com
+ sudo systemctl is-active ghost_therecipebox-club
+ sudo systemctl is-active ghost_stopresetgo-builderscollective-org
+ sudo systemctl is-active ghost_world-builderscollective-org
┌────────────────────────────────────┬─────────────────────────────────────────────┬─────────┬──────────────────────┬────────────────────────────────────────────┬──────┬─────────────────┐
│ Name                               │ Location                                    │ Version │ Status               │ URL                                        │ Port │ Process Manager │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ builderscollective-com             │ /var/www/builderscollective.com             │ 4.19.1  │ running (production) │ https://builderscollective.com             │ 2369 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ designadmin-com                    │ /var/www/designadmin.com                    │ 4.19.1  │ running (production) │ https://designadmin.com                    │ 2370 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ faithhopelove-ca                   │ /var/www/faithhopelove.ca                   │ 4.19.1  │ running (production) │ https://faithhopelove.ca                   │ 2371 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ imaginaxiom-com                    │ /var/www/imaginaxiom.com                    │ 4.19.1  │ running (production) │ https://imaginaxiom.com                    │ 2372 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ socialarc-com                      │ /var/www/socialarc.com                      │ 4.19.1  │ running (production) │ https://socialarc.com                      │ 2373 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ designinfluences-com               │ /var/www/designinfluences.com               │ 4.19.1  │ running (production) │ https://designinfluences.com               │ 2374 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ stephenbau-com                     │ /var/www/stephenbau.com                     │ 4.19.1  │ stopped              │ n/a                                        │ n/a  │ n/a             │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ therecipebox-club                  │ /var/www/therecipebox.club                  │ 4.19.1  │ running (production) │ https://therecipebox.club                  │ 2376 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ stopresetgo-builderscollective-org │ /var/www/stopresetgo.builderscollective.org │ 4.19.1  │ running (production) │ https://stopresetgo.builderscollective.org │ 2377 │ systemd         │
├────────────────────────────────────┼─────────────────────────────────────────────┼─────────┼──────────────────────┼────────────────────────────────────────────┼──────┼─────────────────┤
│ world-builderscollective-org       │ /var/www/world.builderscollective.org       │ 4.20.1  │ running (production) │ https://world.builderscollective.org       │ 2378 │ systemd         │
└────────────────────────────────────┴─────────────────────────────────────────────┴─────────┴──────────────────────┴────────────────────────────────────────────┴──────┴─────────────────┘
stephen@bau:/var/www/stephenbau.com$ ghost start
+ sudo systemctl is-active ghost_stephenbau-com
✔ Checking system Node.js version - found v14.16.1
✔ Ensuring user is not logged in as ghost user
✔ Checking if logged in user is directory owner
✔ Checking current folder permissions
+ sudo systemctl is-active ghost_stephenbau-com
✔ Validating config
✔ Checking folder permissions
✔ Checking file permissions
✔ Checking content folder ownership
✔ Checking memory availability
✔ Checking binary dependencies
✔ Checking systemd unit file
✔ Checking systemd node version - found v14.16.1
+ sudo systemctl start ghost_stephenbau-com
+ sudo systemctl is-enabled ghost_stephenbau-com
✔ Starting Ghost: stephenbau-com

------------------------------------------------------------------------------

Your admin interface is located at: 

    https://stephenbau.com/ghost/


I had to go through this process for each of the failed sites.

  • ghost setup linux-user systemd
  • ghost stop
  • ghost start

I still have no idea what happened, or how the systemd files became corrupted, or why they were not able to be fixed on the original server, but I am glad this has finally been resolved.

1 Like