502 Bad Gateway - Ghost Doctor says MySQL Check failed

If you’re looking for some help, it’s important to provide as much context as possible so that people are able to assist you. Try to always mention:

  • What’s your URL? https://stories.jkbaseer.com/

  • What version of Ghost are you using?
    OS: Ubuntu, v20.04.3 LTS
    Node Version: v14.18.1
    Ghost Version: 4.22.4
    Ghost-CLI Version: 1.18.0
    Environment: production

  • What configuration?
    Used Digitalocean one click installation with cloudflare

  • What browser? Chrome

  • What errors or information do you see in the console?

ghost-mgr@ghost2:/var/www/ghost$ ghost doctor
✔ Checking system Node.js version - found v14.18.1
✔ Checking logged in user
✔ Ensuring user is not logged in as ghost user
✔ Checking if logged in user is directory owner
✔ Checking current folder permissions
✔ Checking system compatibility
Local MySQL install was not found or is stopped. You can ignore this if you are using a remote MySQL host.
Alternatively you could:
a) install/start MySQL locally
b) run `ghost install --db=sqlite3` to use sqlite
c) run `ghost install local` to get a development install using sqlite3.
? Continue anyway? No
✖ Checking for a MySQL installation
+ sudo systemctl is-active ghost_stories-jkbaseer-com
✔ Validating config
✔ Checking folder permissions
✔ Checking file permissions
✔ Checking content folder ownership
✔ Checking memory availability
✔ Checking binary dependencies
✔ Checking free space
✔ Checking systemd unit file
✔ Checking systemd node version - found v14.18.1
One or more errors occurred.

1) Checking for a MySQL installation

Message: MySQL check failed.

Debug Information:
    OS: Ubuntu, v20.04.3 LTS
    Node Version: v14.18.1
    Ghost Version: 4.22.4
    Ghost-CLI Version: 1.18.0
    Environment: production
    Command: 'ghost doctor'
  • What steps could someone else take to reproduce the issue you’re having?
    Ghost Status:
    image
    I do not know but I can share what I have tried from read other post on 502 Topic here.

Background:
The site has been running for a year. And never touched anything on the configuration.

Attempt1: Ghost restart

ghost-mgr@ghost2:/var/www/ghost$ ghost restart
+ sudo systemctl is-active ghost_stories-jkbaseer-com
Ghost instance is not running! Starting...
+ sudo systemctl start ghost_stories-jkbaseer-com
+ sudo systemctl stop ghost_stories-jkbaseer-com
✖ Starting Ghost
A GhostError occurred.

Message: Ghost was able to start, but errored during boot with: connect ECONNREFUSED 127.0.0.1:3306
Help: Unknown database error
Suggestion: journalctl -u ghost_stories-jkbaseer-com -n 50

Debug Information:
    OS: Ubuntu, v20.04.3 LTS
    Node Version: v14.18.1
    Ghost Version: 4.22.4
    Ghost-CLI Version: 1.18.0
    Environment: production
    Command: 'ghost restart'

Additional log info available in: /home/ghost-mgr/.ghost/logs/ghost-cli-debug-2022-11-15T05_53_42_972Z.log

Try running ghost doctor to check your system for known issues.

You can always refer to https://ghost.org/docs/ghost-cli/ for troubleshooting.

Attemp2: Ghost Doctor and the result is shown on the first

As per it, it ask me to install mysql. But I already had one and it’s running. So not sure why it’s not running.

Attempt3: “systemctl status mysql.service”

● mysql.service - MySQL Community Server
     Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor pr>
     Active: failed (Result: exit-code) since Tue 2022-11-15 05:38:33 UTC;>
    Process: 143964 ExecStartPre=/usr/share/mysql/mysql-system-start pre >
    Process: 143972 ExecStart=/usr/sbin/mysqld (code=exited, status=1/FAIL>
   Main PID: 143972 (code=exited, status=1/FAILURE)
     Status: "Server shutdown complete"

Nov 15 05:38:33 ghost2 systemd[1]: mysql.service: Main process exited, cod>
Nov 15 05:38:33 ghost2 systemd[1]: mysql.service: Failed with result 'exit>
Nov 15 05:38:33 ghost2 systemd[1]: Failed to start MySQL Community Server.
Nov 15 05:38:33 ghost2 systemd[1]: mysql.service: Scheduled restart job, r>
Nov 15 05:38:33 ghost2 systemd[1]: Stopped MySQL Community Server.
Nov 15 05:38:33 ghost2 systemd[1]: mysql.service: Start request repeated t>
Nov 15 05:38:33 ghost2 systemd[1]: mysql.service: Failed with result 'exit>
Nov 15 05:38:33 ghost2 systemd[1]: Failed to start MySQL Community Server.
set mark: ...skipping...
● mysql.service - MySQL Community Server
     Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor pr>
     Active: failed (Result: exit-code) since Tue 2022-11-15 05:38:33 UTC;>
    Process: 143964 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre >
    Process: 143972 ExecStart=/usr/sbin/mysqld (code=exited, status=1/FAIL>
   Main PID: 143972 (code=exited, status=1/FAILURE)
     Status: "Server shutdown complete"

Nov 15 05:38:33 ghost2 systemd[1]: mysql.service: Main process exited, cod>
Nov 15 05:38:33 ghost2 systemd[1]: mysql.service: Failed with result 'exit>
Nov 15 05:38:33 ghost2 systemd[1]: Failed to start MySQL Community Server.
Nov 15 05:38:33 ghost2 systemd[1]: mysql.service: Scheduled restart job, r>
Nov 15 05:38:33 ghost2 systemd[1]: Stopped MySQL Community Server.
Nov 15 05:38:33 ghost2 systemd[1]: mysql.service: Start request repeated t>
Nov 15 05:38:33 ghost2 systemd[1]: mysql.service: Failed with result 'exit>
Nov 15 05:38:33 ghost2 systemd[1]: Failed to start MySQL Community Server.

and also restarting mysql leads

Attempt4: sudo service mysql start

Job for mysql.service failed because the control process exited with error code.
See "systemctl status mysql.service" and "journalctl -xe" for details.

Attempt5: Change in cloudflare setup

Switching off proxy in cloudflare and also making the TTL to 1Minute as per another topic in the forum. It doesn’t change as well. I see it’s not due to cloudflare but just wanted to try.

##Attemp5: Using official troubleshooting for Mysql

I used this to debug the MYSQL. I have few question adding to the above setup

  1. If this error occurred after using ghost install, once resolved, re-run the setup phase using ghost setup

If i re run ghost install, will it affect my existing content (critical) and it’s stored configuration?

Adding some additional information based on this: Can't connect to the bootstrap socket (localhost 8000) ECONNREFUSED · Issue #1017 · TryGhost/Ghost-CLI · GitHub

I checked the port configuration on config.production.json and nginx configuration is using the same port. (2368)

The very first console entry points to database. Your third attempt confirms this.

The database isn’t running, and fails to start.

What’s the relevant output from systemctl status mysql.service and journalctl -xe?

Hi @mjw
Thanks for checking this. Here it is from

mysql.service

● mysql.service - MySQL Community Server
     Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Tue 2022-11-15 06:17:29 UTC; 5h 24min ago
    Process: 144979 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
    Process: 144987 ExecStart=/usr/sbin/mysqld (code=exited, status=1/FAILURE)
   Main PID: 144987 (code=exited, status=1/FAILURE)
     Status: "Server shutdown complete"

Nov 15 06:17:29 ghost2 systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
Nov 15 06:17:29 ghost2 systemd[1]: mysql.service: Failed with result 'exit-code'.
Nov 15 06:17:29 ghost2 systemd[1]: Failed to start MySQL Community Server.
Nov 15 06:17:29 ghost2 systemd[1]: mysql.service: Scheduled restart job, restart counter is at 5.
Nov 15 06:17:29 ghost2 systemd[1]: Stopped MySQL Community Server.
Nov 15 06:17:29 ghost2 systemd[1]: mysql.service: Start request repeated too quickly.
Nov 15 06:17:29 ghost2 systemd[1]: mysql.service: Failed with result 'exit-code'.
Nov 15 06:17:29 ghost2 systemd[1]: Failed to start MySQL Community Server.

Journalctl -xe

it’s 1K lines. so am pasting few alarming if that doesn’t make us overwhelm with the thread message.
LAST PART: First few parts


-- A start job for unit UNIT has finished successfully.
-- 
-- The job identifier is 8.
Nov 15 11:41:29 ghost2 systemd[148325]: Reached target Sockets.
-- Subject: A start job for unit UNIT has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit UNIT has finished successfully.
-- 
-- The job identifier is 5.
Nov 15 11:41:29 ghost2 systemd[148325]: Reached target Basic System.
-- Subject: A start job for unit UNIT has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit UNIT has finished successfully.
-- 
-- The job identifier is 2.
Nov 15 11:41:29 ghost2 systemd[1]: Started User Manager for UID 0.
-- Subject: A start job for unit user@0.service has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit user@0.service has finished successfully.
-- 
-- The job identifier is 963774.
Nov 15 11:41:29 ghost2 systemd[1]: Started Session 9315 of user root.
-- Subject: A start job for unit session-9315.scope has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit session-9315.scope has finished successfully.
-- 
-- The job identifier is 963856.
Nov 15 11:41:29 ghost2 systemd[148325]: Reached target Main User Target.
-- Subject: A start job for unit UNIT has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit UNIT has finished successfully.
-- 
-- The job identifier is 1.
Nov 15 11:41:29 ghost2 systemd[148325]: Startup finished in 188ms.
-- Subject: User manager start-up is now complete
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The user manager instance for user 0 has been started. All services queued
-- for starting have been started. Note that other services might still be starting
-- up or be started at any later time.
-- 
-- Startup of the manager took 188341 microseconds.
Nov 15 11:41:36 ghost2 kernel: [UFW BLOCK] IN=eth0 OUT= MAC=1e:b6:0a:a8:e7:65:fe:00:00:00:01:01:08:00 SRC=178.6>
Nov 15 11:41:42 ghost2 kernel: [UFW BLOCK] IN=eth0 OUT= MAC=1e:b6:0a:a8:e7:65:fe:00:00:00:01:01:08:00 SRC=152.8>
Nov 15 11:41:43 ghost2 kernel: [UFW BLOCK] IN=eth0 OUT= MAC=1e:b6:0a:a8:e7:65:fe:00:00:00:01:01:08:00 SRC=206.1>
Nov 15 11:42:04 ghost2 kernel: [UFW BLOCK] IN=eth0 OUT= MAC=1e:b6:0a:a8:e7:65:fe:00:00:00:01:01:08:00 SRC=162.1>
Nov 15 11:42:28 ghost2 sshd[148450]: Invalid user marilyn from 112.217.169.138 port 36939
Nov 15 11:42:28 ghost2 sshd[148450]: pam_unix(sshd:auth): check pass; user unknown
Nov 15 11:42:28 ghost2 sshd[148450]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh>
Nov 15 11:42:30 ghost2 sshd[148450]: Failed password for invalid user marilyn from 112.217.169.138 port 36939 s>
Nov 15 11:42:30 ghost2 kernel: [UFW BLOCK] IN=eth0 OUT= MAC=1e:b6:0a:a8:e7:65:fe:00:00:00:01:01:08:00 SRC=159.6>
Nov 15 11:42:30 ghost2 sshd[148450]: Received disconnect from 112.217.169.138 port 36939:11: Bye Bye [preauth]
Nov 15 11:42:30 ghost2 sshd[148450]: Disconnected from invalid user marilyn 112.217.169.138 port 36939 [preauth]
Nov 15 11:42:43 ghost2 sshd[148452]: Invalid user mak from 168.138.130.56 port 51210
Nov 15 11:42:43 ghost2 sshd[148452]: pam_unix(sshd:auth): check pass; user unknown
Nov 15 11:42:43 ghost2 sshd[148452]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh>
Nov 15 11:42:45 ghost2 sshd[148452]: Failed password for invalid user mak from 168.138.130.56 port 51210 ssh2
Nov 15 11:42:47 ghost2 sshd[148452]: Received disconnect from 168.138.130.56 port 51210:11: Bye Bye [preauth]
Nov 15 11:42:47 ghost2 sshd[148452]: Disconnected from invalid user mak 168.138.130.56 port 51210 [preauth]
Nov 15 11:42:53 ghost2 kernel: [UFW BLOCK] IN=eth0 OUT= MAC=1e:b6:0a:a8:e7:65:fe:00:00:00:01:01:08:00 SRC=113.2>
lines 1086-1158/1158 (END)

Does this help?

Please provide results for the database server:

sudo journalctl -u mysql.service

What version of MySQL are you running? And, do you have a backup of the database and server? Since you are on DO, have you considered rolling back to an earlier snapshot (assuming you take one before performing an upgrade.)

Aug 11 06:51:02 ghost2 systemd[1]: mysql.service: Scheduled restart job, restart counter is at 1.
Aug 11 06:51:02 ghost2 systemd[1]: Stopped MySQL Community Server.
Aug 11 06:51:02 ghost2 systemd[1]: Starting MySQL Community Server...
Aug 11 06:51:04 ghost2 systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
Aug 11 06:51:04 ghost2 systemd[1]: mysql.service: Failed with result 'exit-code'.
Aug 11 06:51:04 ghost2 systemd[1]: Failed to start MySQL Community Server.
Aug 11 06:51:04 ghost2 systemd[1]: mysql.service: Scheduled restart job, restart counter is at 2.
Aug 11 06:51:04 ghost2 systemd[1]: Stopped MySQL Community Server.
Aug 11 06:51:04 ghost2 systemd[1]: Starting MySQL Community Server...
Aug 11 06:51:05 ghost2 systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
Aug 11 06:51:05 ghost2 systemd[1]: mysql.service: Failed with result 'exit-code'.
Aug 11 06:51:05 ghost2 systemd[1]: Failed to start MySQL Community Server.
Aug 11 06:51:05 ghost2 systemd[1]: mysql.service: Scheduled restart job, restart counter is at 3.
Aug 11 06:51:05 ghost2 systemd[1]: Stopped MySQL Community Server.
Aug 11 06:51:05 ghost2 systemd[1]: Starting MySQL Community Server...
Aug 11 06:51:07 ghost2 systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
Aug 11 06:51:07 ghost2 systemd[1]: mysql.service: Failed with result 'exit-code'.
Aug 11 06:51:07 ghost2 systemd[1]: Failed to start MySQL Community Server.
Aug 11 06:51:07 ghost2 systemd[1]: mysql.service: Scheduled restart job, restart counter is at 4.
Aug 11 06:51:07 ghost2 systemd[1]: Stopped MySQL Community Server.
Aug 11 06:51:07 ghost2 systemd[1]: Starting MySQL Community Server...
Aug 11 06:51:08 ghost2 systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
Aug 11 06:51:08 ghost2 systemd[1]: mysql.service: Failed with result 'exit-code'.
Aug 11 06:51:08 ghost2 systemd[1]: Failed to start MySQL Community Server.
Aug 11 06:51:08 ghost2 systemd[1]: mysql.service: Scheduled restart job, restart counter is at 5.
Aug 11 06:51:08 ghost2 systemd[1]: Stopped MySQL Community Server.
Aug 11 06:51:08 ghost2 systemd[1]: mysql.service: Start request repeated too quickly.
Aug 11 06:51:08 ghost2 systemd[1]: mysql.service: Failed with result 'exit-code'.
Aug 11 06:51:08 ghost2 systemd[1]: Failed to start MySQL Community Server.
Aug 12 06:43:23 ghost2 systemd[1]: Starting MySQL Community Server...
Aug 12 06:43:24 ghost2 systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
Aug 12 06:43:24 ghost2 systemd[1]: mysql.service: Failed with result 'exit-code'.
Aug 12 06:43:24 ghost2 systemd[1]: Failed to start MySQL Community Server.
Aug 12 06:43:24 ghost2 systemd[1]: mysql.service: Scheduled restart job, restart counter is at 1.
Aug 12 06:43:24 ghost2 systemd[1]: Stopped MySQL Community Server.
Aug 12 06:43:24 ghost2 systemd[1]: Starting MySQL Community Server...
Aug 12 06:43:25 ghost2 systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
Aug 12 06:43:25 ghost2 systemd[1]: mysql.service: Failed with result 'exit-code'.
Aug 12 06:43:25 ghost2 systemd[1]: Failed to start MySQL Community Server.
Aug 12 06:43:25 ghost2 systemd[1]: mysql.service: Scheduled restart job, restart counter is at 2.
Aug 12 06:43:25 ghost2 systemd[1]: Stopped MySQL Community Server.
Aug 12 06:43:25 ghost2 systemd[1]: Starting MySQL Community Server...
Aug 12 06:43:27 ghost2 systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
Aug 12 06:43:27 ghost2 systemd[1]: mysql.service: Failed with result 'exit-code'.
Aug 12 06:43:27 ghost2 systemd[1]: Failed to start MySQL Community Server.
Aug 12 06:43:27 ghost2 systemd[1]: mysql.service: Scheduled restart job, restart counter is at 3.
Aug 12 06:43:27 ghost2 systemd[1]: Stopped MySQL Community Server.
Aug 12 06:43:27 ghost2 systemd[1]: Starting MySQL Community Server...
Aug 12 06:43:28 ghost2 systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
Aug 12 06:43:28 ghost2 systemd[1]: mysql.service: Failed with result 'exit-code'.
Aug 12 06:43:28 ghost2 systemd[1]: Failed to start MySQL Community Server.
Aug 12 06:43:28 ghost2 systemd[1]: mysql.service: Scheduled restart job, restart counter is at 4.
Aug 12 06:43:28 ghost2 systemd[1]: Stopped MySQL Community Server.
Aug 12 06:43:28 ghost2 systemd[1]: Starting MySQL Community Server...
Aug 12 06:43:30 ghost2 systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
Aug 12 06:43:30 ghost2 systemd[1]: mysql.service: Failed with result 'exit-code'.
Aug 12 06:43:30 ghost2 systemd[1]: Failed to start MySQL Community Server.
Aug 12 06:43:30 ghost2 systemd[1]: mysql.service: Scheduled restart job, restart counter is at 5.
Aug 12 06:43:30 ghost2 systemd[1]: Stopped MySQL Community Server.
Aug 12 06:43:30 ghost2 systemd[1]: mysql.service: Start request repeated too quickly.
Aug 12 06:43:30 ghost2 systemd[1]: mysql.service: Failed with result 'exit-code'.
Aug 12 06:43:30 ghost2 systemd[1]: Failed to start MySQL Community Server.
Aug 13 06:52:47 ghost2 systemd[1]: Starting MySQL Community Server...
Aug 13 06:52:48 ghost2 systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
Aug 13 06:52:48 ghost2 systemd[1]: mysql.service: Failed with result 'exit-code'.
Aug 13 06:52:48 ghost2 systemd[1]: Failed to start MySQL Community Server.
Aug 13 06:52:48 ghost2 systemd[1]: mysql.service: Scheduled restart job, restart counter is at 1.
Aug 13 06:52:48 ghost2 systemd[1]: Stopped MySQL Community Server.

This is the server.

Backup

Unfortunately no, I didn’t take any snapshots. and I don’t have the backup of the database. I don’t have many users though but lot of post and other configuration.

Mysqal version

mysql Ver 8.0.31-0ubuntu0.20.04.1 for Linux on x86_64 ((Ubuntu))

is there any other way to solve this? Also is there a possibility that mysql configuration have messed up anytime.

There’s nothing useful there to aid diagnosis. You could check to see if you have run out of disk space. Alternatively, remove MySQL and reinstall. This shouldn’t remove /var/lib/mysql which is where your databases are located.

I recommend you have a backup strategy. Digital Ocean, Linode and Hetzner all offer snapshots and backups, so make use of these. Likewise, routinely use ghost backup, automysqlbackup, and mysqldump` to offload a copy.

Hi Mjw,
I checked it, but I think the usage is not that high. But let me try to resize the droplet and see. Thanks for your time so far!!!
image

Trying this
Root user permissions fix. Just got to see this.

image

Does this information helps about the root cause? When i run Ghost run, I get this. This is definitely some issue with database

2-11-15 16:44:46] WARN Can't connect to the bootstrap socket (localhost 8000) ECONNREFUSED.
[2022-11-15 16:44:46] WARN Tries: 1
[2022-11-15 16:44:46] WARN Retrying...
[2022-11-15 16:44:47] WARN Can't connect to the bootstrap socket (localhost 8000) ECONNREFUSED.
[2022-11-15 16:44:47] WARN Tries: 2
[2022-11-15 16:44:47] WARN Retrying...
[2022-11-15 16:44:47] WARN Can't connect to the bootstrap socket (localhost 8000) ECONNREFUSED.
[2022-11-15 16:44:47] INFO Ghost server started in 0.905s
[2022-11-15 16:44:47] ERROR connect ECONNREFUSED 127.0.0.1:3306

connect ECONNREFUSED 127.0.0.1:3306

"Unknown database error"

Error ID:
    500

Error Code: 
    ECONNREFUSED

----------------------------------------

InternalServerError: connect ECONNREFUSED 127.0.0.1:3306
    at DatabaseStateManager.getState (/var/www/ghost/versions/4.22.4/core/server/data/db/state-manager.js:63:32)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at async DatabaseStateManager.makeReady (/var/www/ghost/versions/4.22.4/core/server/data/db/state-manager.js:72:25)
    at async initDatabase (/var/www/ghost/versions/4.22.4/core/boot.js:62:5)
    at async bootGhost (/var/www/ghost/versions/4.22.4/core/boot.js:357:9)

DatabaseError: connect ECONNREFUSED 127.0.0.1:3306
    at DatabaseError.KnexMigrateError (/var/www/ghost/versions/4.22.4/node_modules/knex-migrator/lib/errors.js:7:26)
    at new DatabaseError (/var/www/ghost/versions/4.22.4/node_modules/knex-migrator/lib/errors.js:55:26)
    at /var/www/ghost/versions/4.22.4/node_modules/knex-migrator/lib/database.js:48:19
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at async DatabaseStateManager.getState (/var/www/ghost/versions/4.22.4/core/server/data/db/state-manager.js:39:13)
    at async DatabaseStateManager.makeReady (/var/www/ghost/versions/4.22.4/core/server/data/db/state-manager.js:72:25)
    at async initDatabase (/var/www/ghost/versions/4.22.4/core/boot.js:62:5)
    at async bootGhost (/var/www/ghost/versions/4.22.4/core/boot.js:357:9)

Error: connect ECONNREFUSED 127.0.0.1:3306
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1159:16)
    --------------------
    at Protocol._enqueue (/var/www/ghost/versions/4.22.4/node_modules/mysql/lib/protocol/Protocol.js:144:48)
    at Protocol.handshake (/var/www/ghost/versions/4.22.4/node_modules/mysql/lib/protocol/Protocol.js:51:23)
    at Connection.connect (/var/www/ghost/versions/4.22.4/node_modules/mysql/lib/Connection.js:116:18)
    at /var/www/ghost/versions/4.22.4/node_modules/knex-migrator/node_modules/knex/lib/dialects/mysql/index.js:68:18
    at new Promise (<anonymous>)
    at Client_MySQL.acquireRawConnection (/var/www/ghost/versions/4.22.4/node_modules/knex-migrator/node_modules/knex/lib/dialects/mysql/index.js:63:12)
    at create (/var/www/ghost/versions/4.22.4/node_modules/knex-migrator/node_modules/knex/lib/client.js:290:39)

[2022-11-15 16:44:47] WARN Ghost is shutting down
[2022-11-15 16:44:47] WARN Can't connect to the bootstrap socket (localhost 8000) ECONNREFUSED.
[2022-11-15 16:44:47] WARN Tries: 0
[2022-11-15 16:44:47] WARN Retrying...
[2022-11-15 16:44:47] WARN Ghost has shut down
[2022-11-15 16:44:47] WARN Your site is now offline
[2022-11-15 16:44:47] WARN Ghost was running for a few seconds

You’re getting these messages because MySQL isn’t running. The issue has nothing to do with Ghost; it’s an OS / MySQL problem.

Your root (/) partition has enough space, so this is not the issue.

I suggest you take a snapshot now before making further adjustments to the system. Then, I’d back up what you can before completely removing and reinstalling MySQL, and finally restoring /var/lib/mysql. I haven’t been in this situation, so I cannot guarantee this will work.

Hi @mjw ,
Yes i’m trying to backup this before something goes wrong. Just want to share with you this below error log. this output is from Ghost setup though.

✔ Setting up "ghost" system user
? Do you wish to set up "ghost" mysql user? Yes
✖ Setting up "ghost" mysql user
Nginx configuration already found for this url. Skipping Nginx setup.
ℹ Setting up Nginx [skipped]
Nginx setup task was skipped, skipping SSL setup
ℹ Setting up SSL [skipped]
Systemd service has already been set up. Skipping Systemd setup
ℹ Setting up Systemd [skipped]
+ sudo systemctl is-active ghost_stories-jkbaseer-com
+ sudo systemctl reset-failed ghost_stories-jkbaseer-com
? Do you want to start Ghost? Yes
+ sudo systemctl start ghost_stories-jkbaseer-com
+ sudo systemctl stop ghost_stories-jkbaseer-com
✖ Starting Ghost
One or more errors occurred.

1) ConfigError

Error detected in the production configuration.

Message: connect ECONNREFUSED 127.0.0.1:3306
Configuration Key(s): database.connection.host / database.connection.port
Current Value(s): localhost / 3306

Help: Ensure that MySQL is installed and reachable. You can always re-run `ghost setup` to try again.


2) GhostError

Message: Ghost was able to start, but errored during boot with: connect ECONNREFUSED 127.0.0.1:3306
Help: Unknown database error
Suggestion: journalctl -u ghost_stories-jkbaseer-com -n 50


As I’ve previously said, this means Ghost can’t connect to MySQL. This is because MySQL isn’t running.

Something is stopping your MySQL server from starting properly. You’ve already checked space and that doesn’t seem to be an issue.

Logs for MySQL are usually at /var/log/mysql/error.log - so you could try something like:

sudo tail /var/log/mysql/error.log

and see if MySQL is logging a more helpful error message.

If you cannot find where your logs are, you can run the MySQL server interactively as root - if it fails to start, it should tell you why:

sudo mysqld

thanks Jeff,

Sudo Tail Result

2022-11-16T06:12:19.800608Z 0 [ERROR] [MY-010119] [Server] Aborting
2022-11-16T06:12:19.801257Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.31-0ubuntu0.20.04.1)  (Ubuntu).
2022-11-16T06:12:20.636889Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.31-0ubuntu0.20.04.1) starting as process 36703
2022-11-16T06:12:20.653463Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-11-16T06:12:21.158819Z 1 [ERROR] [MY-012526] [InnoDB] Upgrade is not supported after a crash or shutdown with innodb_fast_shutdown = 2. This redo log was created with MySQL 8.0.27, and it appears logically non empty. Please follow the instructions at http://dev.mysql.com/doc/refman/8.0/en/upgrading.html
2022-11-16T06:12:21.158859Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Generic error.
2022-11-16T06:12:21.292385Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2022-11-16T06:12:21.292599Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2022-11-16T06:12:21.292616Z 0 [ERROR] [MY-010119] [Server] Aborting
2022-11-16T06:12:21.293198Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.31-0ubuntu0.20.04.1)  (Ubuntu).

I also did something else to see if it works

Adding sock line in connection section of config

There is one stackoverflow and they ask to add sock line in the below code

You need to add the value of socket path to the config object:

socketPath: '/var/run/mysqld/mysqld.sock'

That just change the error message from econnrefused from 3306 port to socket. Doesn’t work

"url": "https://example.com",
"server": {
  "port": 2368,
  "host": "127.0.0.1"
},
"database": {
  "client": "mysql",
  "connection": {
    "host": "localhost",
    "user": "root",
    "password": "abc",
    "database": "ghost_production"
  }
},
"mail": {
  "transport": "Direct",
  "options": {
    "service": "Mailgun",
    "host": "smtp.mailgun.org",
    "port": 587,
    "secure": true,
    "auth": {
      "user": "postmaster@abc",
      "pass": "abc"
    }
  }
},
"logging": {
  "transports": [
    "file",
    "stdout"
  ]
},
"process": "systemd",
"paths": {
  "contentPath": "/var/www/ghost/content"
},
"bootstrap-socket": {
  "port": 8000,
  "host": "localhost"
}
}

Attempt 9 : Launched a new DO Ghost Droplet to see what’s missing

So I launched a new Ghost one click installation to compare if there is any issue in mysql conf or other setup.

Both installation match the same thing, except one.
ghost-mgr@ghost2:/var/run$

  1. There supposed to be folder (msqld) under var/run
  2. Under mysqld, there need to be 3 files as well.

Could it be this issue?

As a last chance, i want to connect this mysql table with Tableplus to download the tables but as mysql not running, I can connect as well.

It looks like MySQL crashed during an upgrade.

Upgrade is not supported after a crash or shutdown with innodb_fast_shutdown = 2. This redo log was created with MySQL 8.0.27, and it appears logically non empty...

Therefore, take a backup of your data directory as described earlier. Now check /var/lib/mysqlfor files named ib_logfile.* and move these elsewhere. Now, try starting MySQL.

That sounds a bit risky to me? It’s rarely a good idea to remove non-empty transaction logs unless there is no other option, because it may lead to data loss / inconsistency.

It might be better to roll back the MySQL version to 8.0.27 and do a clean shutdown. The debs are available for download from the community repo archive.

I would also say that whoever did the MySQL shutdown with innodb_fast_shutdown=2 or configured this as an option in my.cnf, needs to take over. This is almost never a good option - so they must have had a reason, and should now do the manual recovery.

I try to take the mysql through this code

mysqldump -uroot -p --all-databases > /tmp/all-databases.sql

and the error is

mysqldump: Got error: 2002: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) when trying to connect

it’s a bit of similar error I had before about /mysqld/mysqld.sock file missing. I never went to this directory to delete this file. but what could have deleted this file? (1)

@jeff nope, never did shutdown with innodb_fast_shutdown=2. Can I remove that for sure? (2).

@mjw Last time you shared about only taking droplets. which am doing now for a secondary backup after this issue. So when we remove mysql and add new one, the folder won’t be deleted? /var/lib/mysql

It is, but the OP has zero backups, and manually downgrading is not straightforward either. Earlier, I recommended taking a snapshot of the current state before tinkering, so these steps should be reversible.

Agree here.

@Jkbaseer, this is a config setting and needs changing to 0 or 1.

I said you should take a snapshot, and also copy the relevant folders. I also said mysqldump work work because the database server isn’t working.

However, it’s clear now that the database upgrade failed because transactions were outstanding when the upgrade was performed (caused by innodb_fast_shutdown=2), so you have two options:

  1. Remove (move) the transaction logs, so they are recreated. But as @jeff points out this has risks.
  2. Rollback MySQL by manually uninstalling and reinstalling the earlier binaries.
2 Likes