Upgrade to Ghost v4, but a lot of 503 error

Seems I’m not the only one with this problem, and I can see there’s several threads reporting this issue:

Please let me know if it’s best I started a new thread, or contributed here, which I’ll do for now.

I self-host and upgraded using ghost update from version 3.42.2 to 4.0.1. I switched to the Caspar theme before doing so as a result of the warning I saw on attempting to upgrade. Other useful versions:

  • Node: v14.16.0 (also tried with v12.16.2)
  • Npm: 7.6.3
  • OS: Debian 9.13
  • MySQL: Ver 15.1 Distrib 10.2.30-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2

I don’t believe the issue relates to the versions above because running ghost run allows the blog to start correctly (that’s the state I’ve left it in for now). At the moment I’m completely stumped and don’t know what to try next. ghost log tells me:

[2021-03-17 08:07:24] INFO Ghost is running in production...
[2021-03-17 08:07:24] INFO Your site is now available on https://blog.jonsdocs.org.uk/
[2021-03-17 08:07:24] INFO Ctrl+C to shut down
[2021-03-17 08:07:24] INFO Ghost server started in 0.833s
[2021-03-17 08:07:25] INFO Database is in a ready state.
[2021-03-17 08:07:25] INFO Ghost database ready in 1.264s
[2021-03-17 08:07:27] INFO Ghost booted in 3.536s
[2021-03-17 08:07:28] WARN Ghost is shutting down
[2021-03-17 08:07:28] WARN Ghost has shut down
[2021-03-17 08:07:28] WARN Your site is now offline
[2021-03-17 08:07:28] WARN Ghost was running for a few seconds

ghost ls advises that the blog is running, on the correct port as confirmed against my Nginx config.

An excerpt from my migrations table shows:

+-----+------------------------------------------------------------------+---------+----------------+
| id  | name                                                             | version | currentVersion |
+-----+------------------------------------------------------------------+---------+----------------+
| 153 | 01-add-firstpromoter-settings.js                                 | 3.41    | 3.41           |
| 154 | 01-update-mobiledoc.js                                           | 4.0     | 4.0            |
| 155 | 02-add-status-column-to-members.js                               | 4.0     | 4.0            |
| 156 | 03-populate-status-column-for-members.js                         | 4.0     | 4.0            |
| 157 | 04-drop-apps-related-tables.js                                   | 4.0     | 4.0            |
| 158 | 05-add-members-subscribe-events-table.js                         | 4.0     | 4.0            |
| 159 | 06-populate-members-subscribe-events-table.js                    | 4.0     | 4.0            |
| 160 | 07-alter-unique-constraint-for-posts-slug.js                     | 4.0     | 4.0            |
| 161 | 08-add-members-login-events-table.js                             | 4.0     | 4.0            |
| 162 | 09-add-members-email-change-events-table.js                      | 4.0     | 4.0            |
| 163 | 10-add-members-status-events-table.js                            | 4.0     | 4.0            |
| 164 | 11-add-members-paid-subscription-events-table.js                 | 4.0     | 4.0            |
| 165 | 12-delete-apps-related-settings-keys.js                          | 4.0     | 4.0            |
| 166 | 13-add-members-payment-events-table.js                           | 4.0     | 4.0            |
| 167 | 14-remove-orphaned-stripe-records.js                             | 4.0     | 4.0            |
| 168 | 15-add-frontmatter-column-to-meta.js                             | 4.0     | 4.0            |
| 169 | 16-refactor-slack-setting.js                                     | 4.0     | 4.0            |
| 170 | 17-populate-members-status-events-table.js                       | 4.0     | 4.0            |
| 171 | 18-transform-urls-absolute-to-transform-ready.js                 | 4.0     | 4.0            |
| 172 | 19-remove-labs-members-setting.js                                | 4.0     | 4.0            |
| 173 | 20-refactor-unsplash-setting.js                                  | 4.0     | 4.0            |
| 174 | 21-sanitize-email-batches-provider-id.js                         | 4.0     | 4.0            |
| 175 | 22-solve-orphaned-webhooks.js                                    | 4.0     | 4.0            |
| 176 | 23-regenerate-posts-html.js                                      | 4.0     | 4.0            |
| 177 | 24-add-missing-email-permissions.js                              | 4.0     | 4.0            |
| 178 | 25-populate-members-paid-subscription-events-table.js            | 4.0     | 4.0            |
| 179 | 26-add-cascade-on-delete.js                                      | 4.0     | 4.0            |
| 180 | 27-add-primary-key-brute-migrations-lock.js                      | 4.0     | 4.0            |
| 181 | 28-add-webhook-intergrations-foreign-key.js                      | 4.0     | 4.0            |
| 182 | 29-fix-foreign-key-for-members-stripe-customers-subscriptions.js | 4.0     | 4.0            |
| 183 | 30-set-default-accent-color.js                                   | 4.0     | 4.0            |
+-----+------------------------------------------------------------------+---------+----------------+

Which at a quick glance appears consistent with the one pasted earlier.

Any advice would be greatly appreciated.

1 Like

I am getting the exact same behaviour.

I am running on a Digital Ocean droplet, Ubuntu 18.04. I’ve tried upgrading to the latest node.js version, hasn’t made any difference.

Running ghost run instead of ghost start works, implying the problem might be around permissions when running through the systemd service. I will happily help with debugging more.

I am experiencing the same problem. Ghost Run works, but ghost start doesn’t work…

I started other mentioned topic Ghost won't start after upgrading to v4.0.1 and I just resolved my issue!
If it helps anyone, I resolved my issue by completely uninstalling my failed installation and installing a fresh version 4.0.1 Fresh install failed to run automatically, but after manually running sudo systemctl start ghost_your-blog-com everything started up just fine.

If I had another chance, I’d try to just execute above command before uninstalling.

Hey,

I just tried your command, and it did not help :confused:

@cronco

I use ghost run, but system tells me Permission denied:

[myghostuser ghost]$ ls -la /root/.nvm/versions/node/v12.18.2/bin/node
-rwxrwxrwx 1 myghostuser myghostuser 48634352 6月  30 2020 /root/.nvm/versions/node/v12.18.2/bin/node
[myghostuser ghost]$ ghost run -D
The `ghost run` command is used by the configured Ghost process manager and for debugging. If you're not running this to debug something, you should run `ghost start` instead.
+ sudo node current/index.js
sudo: unable to execute /root/.nvm/versions/node/v12.18.2/bin/node: Permission denied

I install nvm to manage my node version in root user. Should I reinstall node in this user group?

@mitkomitrovski I use ghost run , but system tells me Permission denied:

[myghostuser ghost]$ ls -la /root/.nvm/versions/node/v12.18.2/bin/node
-rwxrwxrwx 1 myghostuser myghostuser 48634352 6月  30 2020 /root/.nvm/versions/node/v12.18.2/bin/node
[myghostuser ghost]$ ghost run -D
The `ghost run` command is used by the configured Ghost process manager and for debugging. If you're not running this to debug something, you should run `ghost start` instead.
+ sudo node current/index.js
sudo: unable to execute /root/.nvm/versions/node/v12.18.2/bin/node: Permission denied

I install nvm to manage my node version in root user. Should I reinstall node in this user group?

@halfrost can you confirm ghost start works for you? In case it does, it would clearly point to a problem with permissions in systemd.

Can you post your systemd configuration please? Do you have any special handling for managing node versions, like nvm?

@naz Before I upgrade to Ghost v4, ghost start is working well. Now, I am in Ghost v4, ghost start works failed. I try to ghost run to run Ghost in development, but failed again:

[myghostuser@iZ ghost]$ ghost run -D
The `ghost run` command is used by the configured Ghost process manager and for debugging. If you're not running this to debug something, you should run `ghost start` instead.
+ sudo node current/index.js
sudo: unable to execute /root/.nvm/versions/node/v12.18.2/bin/node: Permission denied

[myghostuser@iZ ghost]$ sudo node current/index.js
/var/www/ghost/versions/4.0.1/core/boot.js:46
async function initDatabase({config, logging}) {
      ^^^^^^^^

SyntaxError: Unexpected token function
    at createScript (vm.js:56:10)
    at Object.runInThisContext (vm.js:97:10)
    at Module._compile (module.js:549:28)
    at Object.Module._extensions..js (module.js:586:10)
    at Module.load (module.js:494:32)
    at tryModuleLoad (module.js:453:12)
    at Function.Module._load (module.js:445:3)
    at Module.require (module.js:504:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/var/www/ghost/versions/4.0.1/ghost.js:19:5)

Can you please try a fix suggested here - 502 errors after upgrading to Ghost 4 - #9 by DanieleSalatti. Let me know if this solves the problem!

Not Working for me… basically I have only this part of code there
> [Unit]
> Description=Ghost systemd service for blog: opcije-com
> Documentation=https://ghost.org/docs/
>
> [Service]
> Type=simple
> WorkingDirectory=/var/www/ghost
> User=999
> Environment=“NODE_ENV=production”
> ExecStart=/usr/bin/node /usr/bin/ghost run
> Restart=always
>
> [Install]
> WantedBy=multi-user.target

@naz I did not configure special item in systemd, it’s default setting by Ghost:

[Unit]
Description=Ghost systemd service for blog: halfrost-com
Documentation=https://docs.ghost.org

[Service]
Type=simple
WorkingDirectory=/var/www/ghost
User=993
Environment="NODE_ENV=production"
ExecStart=/usr/local/bin/node /usr/local/bin/ghost run
Restart=always

[Install]
WantedBy=multi-user.target

My node version have changed to lts(v.14.16.0), but it is still not working. Maybe the solution does not solves the problem for us.

Reading this thread and 502 errors after upgrading to Ghost 4 - #13 by Hannah

From what I can tell, everyone affected is on Node v14, and the problem is not with Node v14 itself but rather with how Node has been updated meaning Ghost is getting two different node versions.

We know the CLI is not good at catching and making these issues visible right now and it’s something we’re looking into to try to help.

If you are using nvm to manage Node, please make sure you’ve read this: Using nvm with local and production Ghost installs

If you are updating Node, it’s a really good idea to do this separately to updating Ghost to 4.0.

My recommendation for anyone updating is to do the following:

  • ghost version to get your current Ghost version
  • update node
  • run ghost update [ghost version] --force to do a reinstall of the current Ghost version
  • check everything is ok
  • now run ghost update to update Ghost itself

If you’ve already done the update and are running into problems, it will be easier to help if everyone starts their own threads.

Please share the normal debug info and also:

  • did you just update node? If so how?
  • output from which node
  • output from node --version
  • your systemd file
  • :warning: Important, take care: The output from running node --version with the full path from Exec Start in systemd, so if your ExecStart says ExecStart=/usr/bin/node /usr/bin/ghost run then share the output from running /usr/bin/node --version`

This should be a good start.


This thread was started by @halfrost and we will try to help them here. Please can anyone else who is still having problems start their own thread sharing the info I have asked for?

3 Likes

@Hannah Thank you very much! My website is online. https://halfrost.com/

I follow your recommendation:

  1. uninstall nvm
  2. reinstall node & npm
  3. reinstall Ghost-CLI
  4. run ghost update 4.0.1 --force

Then everything is OK~ I hope these steps can be useful.

3 Likes

That’s great news!

1 Like

After reinstalling nodejs and ghost, I found ghost in different folder than before:

whereis ghost: /usr/bin/ghost (used to be /usr/local/bin/ghost)

the systemd config has different path for ghost:
cat /etc/systemd/system/ghost_test.service

[Service]
Type=simple
WorkingDirectory=/var/www/ghost
User=999
Environment="NODE_ENV=production"
ExecStart=/usr/bin/node /usr/local/bin/ghost run
Restart=always

Then, i changed /usr/local/bin/ghost to /usr/bin/ghost and did:

sudo systemctl daemon-reload
ghost start

which finally worked. Hope it helps somebody!

1 Like

Can anyone solve this also: https://forum.ghost.org/t/eacces-permission-denied-open-var-www-ghost-ghost-cli/20988

In my case, I needed nvm for other applications so It would be a hassle to have to uninstall it. Instead I did the following:

which ghost

# /home/<your_user>/.nvm/versions/node/v12.17.0/bin/ghost

Then I modified the ghost.service with the ghost binary path in nvm:

sudo vim /etc/systemd/system/ghost_your-blog-com

[Unit]
Description=Ghost systemd service for blog: your-blog-com
Documentation=https://ghost.org/docs/

[Service]
Type=simple
WorkingDirectory=/var/www/html/your-blog
User=998
Environment="NODE_ENV=production"
#ExecStart=/usr/bin/node /usr/bin/ghost run
ExecStart=/usr/bin/node /home/<your_user>/.nvm/versions/node/v12.17.0/bin/ghost run
Restart=always

[Install]
WantedBy=multi-user.target

Afterward, run

sudo systemctl daemon-reload
ghost start

Thanks @elergy for the hints :raised_hands:

2 Likes

Thanks so much, this is the right solution for nvm users