Upgrade to v5.72.0 - Systemd process manager has not been set up or is corrupted

Upgrade v5.53.1 to v5.72.0, failing with “Systemd process manager has not been set up or is corrupted”

Debug Information:
    OS: Debian GNU/Linux, v11
    Node Version: v18.18.2
    Ghost Version: 5.72.0
    Ghost-CLI Version: 1.25.3
    Environment: production
    Command: 'ghost upgrade'

And ghost doctor

✔ Checking system Node.js version - found v18.18.2
✔ Checking logged in user
✔ Ensuring user is not logged in as ghost user
✔ Checking if logged in user is directory owner
✔ Checking current folder permissions
System checks failed with message: 'Linux version is not Ubuntu 16, 18, 20, or 22'
Some features of Ghost-CLI may not work without additional configuration.
For local installs we recommend using `ghost install local` instead.
? Continue anyway? Yes
System stack check skipped
ℹ Checking system compatibility [skipped]
✔ Checking for a MySQL installation
+ sudo systemctl is-active ghost_jaguart-tech
Instance is currently running
ℹ Validating config [skipped]
✔ Checking folder permissions
✔ Checking file permissions
✔ Checking memory availability
✔ Checking binary dependencies
✔ Checking free space
✔ Checking systemd unit file
✔ Checking systemd node version - found v18.18.2

I ran ghost setup linux-user systemd - but this didn’t help.

ghost ls says:

jaguart-tech │ /var/www/www.jaguart.tech │ 5.72.0  │ running (production) │ https://jaguart.tech │ 2368 │ systemd

but NGINX is reporting ‘bad gateway’ and netstat shows that there is nothing listening on port 2368

Any pointers?

I should also add the sudo systemctl restart ghost_jaguart-tech works without error, and status reports the service running…

and syslog is saying:

node: {
node:   message: {
node:     statusCode: 500,
node:     errorType: 'MigrationsAreLockedError',
node:     level: 'critical',
node:     message: 'Ghost was able to start, but errored during boot with: Migration lock was never released or currently a migration is running.',
node:     id: 500,
node:     help: 'If you are sure no migration is running, check your data and if your database is in a broken state, you could run `yarn knex-migrator rollback`.',
node:     name: 'MigrationsAreLockedError',
node:     code: null,
node:     property: null,
node:     redirect: null,
node:     hideStack: false
node:   }
node: }

Anyone know how to deal with MigrationsAreLockedError error?

I’ve run mysqlcheck -o ghost_jaguart and the database is not reporting any corruption or structural issues…

So I found the migrations_lock table and updated the lock to 0 as per Migration lock after updating Ghost - #4 by KeenenCharles - but when I sudo systemctl restart ghost_jaguart-tech the MigrationsLock error recurrs, and the database has been updated to 1 again…

I also tried ghost update --rollback - which fails with permissions because the Ghost logs are owned by Ghost rather than normal user. Opened the permissions on the logs - but --rollback fails at the systemctl stop stage.

ghost update --force says it has succeeded, but systemctl start and while it thinks its running, syslog says:

{
   message: {
     statusCode: 500,
     errorType: 'InternalServerError',
     level: 'critical',
     message: "Ghost was able to start, but errored during boot with: EACCES: permission denied, open '/var/www/www.jaguart.tech/content/data/the-art-of-jagu.ghost.2023-11-06-07-08-11.json'",
     id: '3ee655f0-7c73-11ee-bf5e-7d632264e5ee',
     name: 'InternalServerError',
     code: 'EACCES',
     property: null,
     redirect: null,
     hideStack: false,
     errno: -13,
     syscall: 'open',
     path: '/var/www/www.jaguart.tech/content/data/the-art-of-jagu.ghost.2023-11-06-07-08-11.json'
   }
}

The file that is is unable to open does not exist - it has the MigrationsLock timestamp in it’s name. So at a guess, the reinstall even with --force is missing a step…

Not sure what to try next - any help appreciated.

I’ve restored the site to v5.53.1 and the database structure and content using backups. The site is now running again - on v5.53.1, and all features seem to be working correctly.

I have retried several variations on ghost update but they all fail when attempting to stop ghost. Here’s a verbose log:

jeff@jort:/var/www/www.jaguart.tech$ ghost update --no-restart --verbose

MySQL: running query > SELECT @@version AS version

[20:25:36] Checking for Ghost-CLI updates [started]
[20:25:36] Checking for Ghost-CLI updates [completed]
[20:25:36] Ensuring correct ~/.config folder ownership [started]
[20:25:36] Ensuring correct ~/.config folder ownership [completed]

+ sudo systemctl is-active ghost_jaguart-tech
[20:25:37] Checking system Node.js version [started]
[20:25:37] Checking system Node.js version - found v18.18.2 [title changed]
[20:25:37] Checking system Node.js version - found v18.18.2 [completed]
[20:25:37] Ensuring user is not logged in as ghost user [started]
[20:25:37] Ensuring user is not logged in as ghost user [completed]
[20:25:37] Checking if logged in user is directory owner [started]
[20:25:37] Checking if logged in user is directory owner [completed]
[20:25:37] Checking current folder permissions [started]
[20:25:37] Checking current folder permissions [completed]
[20:25:37] Checking folder permissions [started]
[20:25:37] Checking folder permissions [completed]
[20:25:37] Checking file permissions [started]
[20:25:37] Checking file permissions [completed]
[20:25:37] Checking memory availability [started]
[20:25:37] Checking memory availability [completed]
[20:25:37] Checking free space [started]
[20:25:37] Checking free space [completed]
✔ Checking for available migrations
✔ Checking for latest Ghost version
[20:25:37] Fetching release notes [started]
[20:25:38] Fetched release notes [title changed]

# 5.72.0
View the changelog for full details: https://github.com/tryghost/ghost/compare/v5.71.2...v5.72.0

[20:25:38] Fetched release notes [completed]
[20:25:38] Downloading and updating Ghost [started]
[20:25:38] Downloading and updating Ghost [skipped]
[20:25:38] → Version already installed.
[20:25:38] Stopping Ghost [started]
[20:25:38] Stopping Ghost [failed]
[20:25:38] → Systemd process manager has not been set up or is corrupted.
A SystemError occurred.

Message: Systemd process manager has not been set up or is corrupted.
Help: Run ghost setup linux-user systemd and try again.

Debug Information:
    OS: Debian GNU/Linux, v11
    Node Version: v18.18.2
    Ghost Version: 5.53.1
    Ghost-CLI Version: 1.25.3
    Environment: production
    Command: 'ghost update --no-restart --verbose'

This is the output of ghost doctor:

jeff@jort:/var/www/www.jaguart.tech$ ghost doctor

✔ Checking system Node.js version - found v18.18.2
✔ Checking logged in user
✔ Ensuring user is not logged in as ghost user
✔ Checking if logged in user is directory owner
✔ Checking current folder permissions
System checks failed with message: 'Linux version is not Ubuntu 16, 18, 20, or 22'
Some features of Ghost-CLI may not work without additional configuration.
For local installs we recommend using `ghost install local` instead.
? Continue anyway? Yes
System stack check skipped
ℹ Checking system compatibility [skipped]
✔ Checking for a MySQL installation
+ sudo systemctl is-active ghost_jaguart-tech
Instance is currently running
ℹ Validating config [skipped]
✔ Checking folder permissions
✔ Checking file permissions
✔ Checking memory availability
✔ Checking binary dependencies
✔ Checking free space
✔ Checking systemd unit file
✔ Checking systemd node version - found v18.18.2

Systemctl seems to be configured correctly:

jeff@jort:/var/www/www.jaguart.tech$ sudo systemctl start ghost_jaguart-tech.service

jeff@jort:/var/www/www.jaguart.tech$ sudo systemctl status ghost_jaguart-tech.service
● ghost_jaguart-tech.service - Ghost systemd service for blog: jaguart-tech
     Loaded: loaded (/var/www/www.jaguart.tech/system/files/ghost_jaguart-tech.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2023-11-06 20:22:28 CET; 9min ago
       Docs: https://ghost.org/docs/
   Main PID: 618021 (ghost run)
      Tasks: 22 (limit: 4568)
     Memory: 166.2M
        CPU: 9.791s
     CGroup: /system.slice/ghost_jaguart-tech.service
             ├─618021 ghost run
             └─618052 /usr/local/bin/node current/index.js

Nov 06 20:22:29 jort node[618021]: Love open source? We’re hiring JavaScript Engineers to work on Ghost full-time.
Nov 06 20:22:29 jort node[618021]: https://careers.ghost.org
Nov 06 20:22:29 jort node[618021]: - Inspecting operating system

jeff@jort:/var/www/www.jaguart.tech$ sudo systemctl stop ghost_jaguart-tech.service
jeff@jort:/var/www/www.jaguart.tech$ sudo systemctl status ghost_jaguart-tech.service
● ghost_jaguart-tech.service - Ghost systemd service for blog: jaguart-tech
     Loaded: loaded (/var/www/www.jaguart.tech/system/files/ghost_jaguart-tech.service; enabled; vendor preset: enabled)
     Active: inactive (dead) since Mon 2023-11-06 20:34:01 CET; 2s ago
       Docs: https://ghost.org/docs/
    Process: 618021 ExecStart=/usr/local/bin/node /usr/local/bin/ghost run (code=exited, status=0/SUCCESS)
   Main PID: 618021 (code=exited, status=0/SUCCESS)
        CPU: 9.867s

Nov 06 20:22:29 jort node[618021]: Love open source? We’re hiring JavaScript Engineers to work on Ghost full-time.
Nov 06 20:22:29 jort node[618021]: https://careers.ghost.org
Nov 06 20:22:29 jort node[618021]: - Inspecting operating system
Nov 06 20:34:01 jort systemd[1]: ghost_jaguart-tech.service: Consumed 9.867s CPU time. 

My conclusion is that the ghost-cli interaction with sudo systemctl - which is not logged verbosely, has an issue.

FWIW the systemd setup looks ok:

jeff@jort:/var/www/www.jaguart.tech$ ls -al /etc/systemd/system/ghost_jaguart-tech.service
lrwxrwxrwx 1 root root 65 Sep 14  2020 /etc/systemd/system/ghost_jaguart-tech.service -> /var/www/www.jaguart.tech/system/files/ghost_jaguart-tech.service

jeff@jort:/var/www/www.jaguart.tech$ cat /var/www/www.jaguart.tech/system/files/ghost_jaguart-tech.service
[Unit]
Description=Ghost systemd service for blog: jaguart-tech
Documentation=https://ghost.org/docs/

[Service]
Type=simple
WorkingDirectory=/var/www/www.jaguart.tech
User=998
Environment="NODE_ENV=production"
ExecStart=/usr/local/bin/node /usr/local/bin/ghost run
Restart=always

[Install]
WantedBy=multi-user.target

And the user details are correct:

jeff@jort:/var/www/www.jaguart.tech$ grep 998 /etc/passwd
ghost:x:998:998::/home/ghost:/bin/sh

:man_shrugging:

So weirdly, a few minutes later, with the ghost instance stopped using sudo systemctl stop ghost_jaguart-tech.service

I ran

ghost update --no-restart --verbose --debug

which completed without error. When I start the service using sudo systemctl start ghost_jaguart-tech.service the following journalctl error is logged:

Love open source? We’re hiring JavaScript Engineers to work on Ghost full-time.
https://careers.ghost.org
- Inspecting operating system
/usr/local/lib/node_modules/ghost-cli/lib/process-manager.js:46
        throw error;
        ^
{
  message: {
    statusCode: 500,
    errorType: 'InternalServerError',
    level: 'critical',
    message: "Ghost was able to start, but errored during boot with: EACCES: permission denied, open '/var/www/www.jaguart.tech/content/data/the-art-of-jagu.ghost.2023-11-06-20-04-56.json'",
    id: 'c1b0e150-7cdf-11ee-86f5-59782ca6795b',
    name: 'InternalServerError',
    code: 'EACCES',
    property: null,
    redirect: null,
    hideStack: false,
    errno: -13,
    syscall: 'open',
    path: '/var/www/www.jaguart.tech/content/data/the-art-of-jagu.ghost.2023-11-06-20-04-56.json'
  }
}
Node.js v18.18.2

The file: content/data/the-art-of-jagu.ghost.2023-11-06-20-04-56.json does not exist. There are a bunch of files with older date-stamps, I assume from previous updates.

Once this error has occurred, Ghost get’s stuck in a MigrationsAreLockedError error state.

Inference: During update/boot - the generation of the-art-of-jagu.ghost.2023-11-06-20-04-56.json is failing silently.

This is a repeatable upgrade failure for v5.53.1 to v5.72.0

So in the mean time, v5.72.1 is out - Ghost moves fast :sweat_smile:

I ran

ghost buster 
ghost update --force
# and the upgrade to v5.72.1 worked without error

Whew. I have another set of upgrades to run. Fingers crossed.