MySQL won't start: "Failed to upgrade server"...but no upgrade was ever requested!

Developer help Self-hosting
1

Self hosted on a DigitalOcean droplet, been running fantastic for several years now.

Made a blog post on Monday of this week as a matter fact.

Went to the URL today: site is dead.

ghost doctor

✔ Checking system Node.js version - found v14.19.3
✔ Checking logged in user
✔ Ensuring user is not logged in as ghost user
✔ Checking if logged in user is directory owner
✔ Checking current folder permissions
✔ Checking system compatibility
✔ Checking for a MySQL installation
+ sudo systemctl is-active ghost_thefamouslastpull-com
+ sudo systemctl reset-failed ghost_thefamouslastpull-com
✔ Validating config
✔ Checking folder permissions
✔ Checking file permissions
✔ Checking content folder ownership
✔ Checking memory availability
✔ Checking binary dependencies
✔ Checking free space
✔ Checking systemd unit file
✔ Checking systemd node version - found v14.19.3

ghost ls - not running

ghost start:

A GhostError occurred.

Message: Ghost was able to start, but errored during boot with: connect ECONNREFUSED 127.0.0.1:3306
Help: Unknown database error
Suggestion: journalctl -u ghost_thefamouslastpull-com -n 50

Debug Information:
    OS: Ubuntu, v20.04.4 LTS
    Node Version: v14.19.3
    Ghost Version: 5.26.3
    Ghost-CLI Version: 1.24.0
    Environment: production
    Command: 'ghost start'

Additional log info available in: /home/ghost-mgr/.ghost/logs/ghost-cli-debug-2023-05-11T18_08_04_492Z.log

Tailing that log:

Debug Information:
    OS: Ubuntu, v20.04.4 LTS
    Node Version: v14.19.3
    Ghost Version: 5.26.3
    Ghost-CLI Version: 1.24.0
    Environment: production
    Command: 'ghost start'
Message: Ghost was able to start, but errored during boot with: connect ECONNREFUSED 127.0.0.1:3306
Help: Unknown database error
Suggestion: journalctl -u ghost_thefamouslastpull-com -n 50
Stack: Error: Ghost was able to start, but errored during boot with: connect ECONNREFUSED 127.0.0.1:3306
    at Server.<anonymous> (/usr/local/lib/node_modules/ghost-cli/lib/utils/port-polling.js:73:28)
    at Object.onceWrapper (events.js:519:28)
    at Server.emit (events.js:400:28)
    at emitCloseNT (net.js:1677:8)
    at processTicksAndRejections (internal/process/task_queues.js:81:21)

Tailing the actual mysql log:

2023-05-11T18:11:00.200487Z 0 [ERROR] [MY-013380] [Server] Failed to upgrade server.
2023-05-11T18:11:00.201111Z 0 [ERROR] [MY-010119] [Server] Aborting
2023-05-11T18:11:00.882906Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.33-0ubuntu0.20.04.1)  (Ubuntu).
2023-05-11T18:11:01.424729Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.33-0ubuntu0.20.04.1) starting as process 23491
2023-05-11T18:11:01.436497Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2023-05-11T18:11:01.848439Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2023-05-11T18:11:02.735054Z 4 [System] [MY-013381] [Server] Server upgrade from '80032' to '80033' started.
2023-05-11T18:11:53.374591Z 4 [ERROR] [MY-013178] [Server] Execution of server-side SQL statement 'EXECUTE stmt; ' failed with error code = 1205, error message = 'Lock wait timeout exceeded; try restarting transaction'.
2023-05-11T18:11:53.377440Z 0 [ERROR] [MY-013380] [Server] Failed to upgrade server.
2023-05-11T18:11:53.378241Z 0 [ERROR] [MY-010119] [Server] Aborting
2023-05-11T18:11:54.680326Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.33-0ubuntu0.20.04.1)  (Ubuntu).
2023-05-11T18:11:55.516639Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.33-0ubuntu0.20.04.1) starting as process 23546
2023-05-11T18:11:55.530678Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2023-05-11T18:11:56.001288Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2023-05-11T18:11:56.793740Z 4 [System] [MY-013381] [Server] Server upgrade from '80032' to '80033' started.

I never requested an upgrade, and I never kicked one off manually. What’s going on?

2 Likes
2

My site is just down today and it got exactly the same problem as yours .

A GhostError occurred.

Message: Ghost was able to start, but errored during boot with: connect ECONNREFUSED 127.0.0.1:3306
Help: Unknown database error

Debug Information:
    OS: Ubuntu, v22.04.1 LTS
    Node Version: v16.17.0
    Ghost Version: 5.43.0
    Ghost-CLI Version: 1.24.0
    Environment: production
    Command: 'ghost start'

So I ran $ systemctl status mysql
I got Status: "Server upgrade in progress"
And from mySQL log, I got the same

Failed to upgrade server.
Execution of server-side SQL statement 'EXECUTE stmt; failed with error code = 1205, error message = 'Lock wait timeout exceeded; try restarting transaction'.

I haven’t done any upgrade or anything related to that too and mine is also self-hosted on a DigitalOcean Droplet.

If you find the solution, please share here too.

Thank you for your support

3

I encountered the same exact issue this week - I can’t start mysql server and see the ‘80032’ → ‘80033’ messages in my log.

Initially, my website was hanging and returning a 504. I updated my ghost version, and now its returning a 502. Regardless, still hitting the same issue with being unable to connect to mysql and being unable to start it.

4

I couldn’t sort out the mysql problem, so I created a new digital ocean droplet. Not ideal, but fortunately it’s pretty easy to spin them up with the marketplace ghost app.

5

Hi there, just adding a voice to this exact error.
Did not touch my ghost installation based off an digitalocean droplet, same error.
As I am not technically savvy enough to get to the root of the problem, I nuked the installation, rebuilt the droplet from the Ghost image that digitalocean offers, installed fresh, re-uploaded my content…
only to hit the same error a few days later.
So if someone has found out WHY this seems to be happening to ghost hosted on digitalocean droplets, I’d be really grateful if you could share that here!

6

I am not a Digital Ocean customer, but their docs say that “Most Ubuntu servers come with [unattended-upgrades] already installed and configured”

That tool causes security updates to automatically be applied nightly.

This could explain why MySQL is upgrading itself when you did’t request it.

It would also explain why when you “start over”, it happens again a few days later-- If you keep starting with the same outdated MySQL version each time, it’s going to keep trying to apply the same security upgrade to it.

There are some ways to stop the upgrades at the expense of security, like uninstalling unattended-upgrades or pinning the MySQL version.

But I recommend instead trying to get this security update applied. To make it more likely to succeed, shutdown Ghost and MySQL and /then/ run sudo apt update && sudo apt upgrade to upgrade to the latest version of MySQL and other packages.

Finally, if you are starting from scratch, just run sudo apt update first, before you install MySQL so you are starting with the latest version before you do anything else.

4 Likes
7

that makes a whole lot of sense - thanks a bunch, especially for the practical advice that I can put into action immediately. Much appreciated!

1 Like
8

I had the same issue. I tried to apply updates but nothing helped. Lost half a day because of this. I still don’t get why Ghost is relying on MySQL. Literally everyone else uses PostgreSQL nowadays. I bet most users run Ghost and the DB on the same machine so we could even use a simple sqlite by default.

Luckily I faced the issue on a brand new ghost instance so I just migrated it to sqlite. Works like a charm now.

9

The question today for Ghost is not “Is PostgreSQL better than MySQL”, it would be “Even if PostgreSQL is better than MySQL, is it worth the cost of switching?”

And that cost is pretty high now, with thousands of installs of Ghost out there.

10

Even if PostgreSQL is better than MySQL, is it worth the cost of switching?

Yeah, switching is never easy.

It’s getting a bit off-topic but I assume the main reason why MySQL is still a thing are the managed ghost blogs, which btw is totally fine because they can hire MySQL experts to get the support they need to run the blogs. For smaller self-hosted blogs I’d rather use sqlite. Simple to use, small memory requirements, and therefore cheaper hosting.

11

It’s just the managed blogs, it’s also the thousands of of self-hosted blogs. If you are going to ask thousands of people to update their database hosting, there ought to be a compelling reason.

Sure, SQLite is fine for small blogs. It’s designed for to be updated by a single user. If you have multiple blog admins, there could be problems.

Also, as a small team, it’s simplest to support a SQL database and that scale up as well as down.

The post you found about “Migrating to Sqlite” is outdated. Since Ghost 5, Ghost no longer supports Sqlite in production:

I don’t know there some features that break with SQLite or if they simply aren’t officially supported. If you run into problems with running SQLite in production, post something in the forum to let us know.

12

As long as we are getting off topic about database choices, here’s an anecdote about blog software and database support:

There was once a popular blog platform called Movable Type. One of the features of Movable Type was that it supported multiple databases. If I recall, it may have included at least Oracle, PostgreSQL and MySQL.

To support all these databases, they used a database abstraction layer. So, even if you were familiar with writing SQL for PostgreSQL or MySQL, it was still hard write extensions for the blog, because you had to learn the abstraction layer’s syntax that would in turn generate the SQL you already knew.

Later, a competing blog engine appeared with fewer features to start with, called WordPress. It only supported a single database, MySQL. If you knew basic PHP and MySQL, you could easily create extensions for WordPress, and many people did.

I was one of the people who preferred PostgreSQL and tried to write a plugin for Movable Type. It was painful, due to the database abstraction layer.

We know what happened next: WordPress extensions spread like wildfire and people hardly remember Movable Type.

Right now, the Ghost design is more like Movable Type in the sense that is uses “knex” as a database abstraction layer that allows it support both SQLite and MySQL. This is supposed to make local development easier because local devs don’t have to set up MySQL, but was it the best call?

The situations with Plugins is also different now. WordPress plugins are a constant security headache for WordPress users. Ghost essentially doesn’t have them by design, using APIs for integrations instead. Because of that, People extending don’t usually need to learn MySQL or any other database-- they use Ghost’s APIs.

I also once published my own open source software. I wrote it for PostgreSQL, but accepted patches from someone who wanted it to support MySQL as well. Great! More users! More database support!

Then I wanted to add a new feature, and this contributor wasn’t around anymore. I didn’t like MySQL and didn’t want to work on it. So I had an unpleasant choice: Make a release that only fully worked on PostgreSQL, or install MySQL and do twice as much testing and database work as I wanted to. Worse, I was faced with possibly dumbing down my SQL to use only the features that both databases support. Ultimately, I believe I ripped out the second database support because I had neither the time or interest to support two.

If someone wanted to fork my software to support another database, they were welcome to. And it’s the same situation with Ghost. If you’d like better SQLite support, you are welcome to fork it or try submitting patches.

1 Like
13

This is exactly why Ghost dropped PostgreSQL support early on:

1 Like
15

It’s happening again (this time via Server upgrade from '80033' to '80034' started), and while it still is not clear what the true fix is, I’ve managed to get the system back up and running temporarily (in order to export data if I have to go down the nuclear option path and kill the entire droplet).

Assume all these steps as root or include sudo as needed:

  1. Ensure MySQL is shutdown first:
systemctl stop mysql
  1. Modify the systemd service file (mine was at /lib/systemd/system/mysql.service; you’ll see yours when you run systemctl status mysql and read the Loaded: value), to add a command-line parameter:

Change this:

ExecStart=/usr/sbin/mysqld

To this:

ExecStart=/usr/sbin/mysqld --upgrade=MINIMAL
  1. Reload the systemd settings:
systemctl daemon-reload
  1. Start MySQL back up
systemctl start mysql

This should get Ghost back up and running at least to a place where you can export your data.

Still really would like to know what the correct settings are so that when MySQL determines it’s time to upgrade (did I mention this is a default MySQL behavior now? It had nothing to do with DigitalOcean/Droplets/Unattended updates) that it doesn’t fall into an infinite loop of "I’m trying to upgrade but can’t because the settings aren’t correct…so I’ll just keep trying…forever???

16

Thanks for your long and detail answer about different blog engines answer, I also have the same question: why MySQL and not PostgreSQL initially? or not both since Ghost uses knex?

MySQL 8 provides features that are optimal for Ghost installations

Do you know what these features are or where to look into in the docs? It’s from FAQ page.

17

These are addressed in the linked post on why PostgreSQL support was dropped:

The Ghost team use Sqlite3 for development […] Postgres is a particular challenge because it is quite different to MySQL and SQLite. Code that automatically works for MySQL & SQLite often needs special modifications for Postgres, meaning it is time consuming and hard to maintain.

It didn’t matter that Knex supported both, as the “abstraction is leaking” some details of maintaining PostgreSQL were still required to be supported.

Indeed, I wonder if it’s worth continuing to support both SQLite and MySQL at all. Already now MySQL is the only engine supported in production. If MySQL were required for development-- even easier these days with progress on containerized local services-- then the code could be further simplified by ripping out Knex completely and working directly with a single database engine.

A concrete benefit is that some parts of Knex used by Ghost are out of date and not well-maintained. Removing the need for those bits would permanently solve that problem.

18

Thank you, but it is not clear to me. I don’t see MySQL advantages as they said. It’s likely the reason was they just wanted both MySQL and SQLite. Why not just PostgreSQL for both development and production since it’s easy to start a docker container. MySQL is only option now, it looks like I want a banana but get a gorrilla and a banana jungle :smiley:

Right now when I need a ghost instance for some tests locally, I spent more time to make a MySQL container running properly instead of ghost, issues such as default-authentication-plugin=mysql_native_password makes me sick.

19

It looks like they have to do a significant upgrade willy-nilly in the future to rip out Knex too :D

20

Postgres was added and later removed because it was too complicated to support (which I linked earlier in the thread, lol):