Ghost Database disappeared/inaccessible

Hi! I’ve been trying to fix this error for quite a while, and I can’t seem to figure it out :sweat_smile:

I had Ghost 5 (with the latest version, 5.9.4) deployed via Docker Swarm, using MariaDB 10.6.8 (also containerized, on the same host).

The Ghost installation was working fine until I rebooted the stack, and then Ghost started shooting errors like:


ERROR Access denied for user 'placeholder'@'10.0.5.4' (using password: NO)
"Unknown database error"
ERROR Access denied for user 'placeholder'@'10.0.5.4' (using password: NO)
Error ID: 500
Access denied for user 'placeholder' (using password: NO)
ERROR Access denied for user 'placeholder'@'10.0.5.4' (using password: NO)
"Unknown database error"
Access denied for user 'placeholder'@'10.0.5.4' (using password: NO)
"Unknown database error"
Error Code:
ER_ACCESS_DENIED_ERROR
Error: Access denied for user 'placeholder'@'10.0.5.4' (using password: NO)
at /var/lib/ghost/versions/5.9.4/node_modules/knex-migrator/lib/database.js:57:19
at Packet.asError (/var/lib/ghost/versions/5.9.4/node_modules/mysql2/lib/packets/packet.js:728:17)
at ClientHandshake.execute (/var/lib/ghost/versions/5.9.4/node_modules/mysql2/lib/commands/command.js:29:26)
at Connection.handlePacket (/var/lib/ghost/versions/5.9.4/node_modules/mysql2/lib/connection.js:456:32)
at PacketParser.onPacket (/var/lib/ghost/versions/5.9.4/node_modules/mysql2/lib/connection.js:85:12)
at PacketParser.executeStart (/var/lib/ghost/versions/5.9.4/node_modules/mysql2/lib/packet_parser.js:75:16)
at Socket.<anonymous> (/var/lib/ghost/versions/5.9.4/node_modules/mysql2/lib/connection.js:92:25)
at Socket.emit (node:events:513:28)
at addChunk (node:internal/streams/readable:315:12)

I’ve double-checked and triple-checked, and my configuration options haven’t changed (I’m using an env file to set database and SMTP options). I’ve included a database password etc., I’ve gone into MariaDB container itself to check that the user and database exists.

But when I go into the MariaDB container, and I use mysql to check the database, it tells me that the database is empty. Not sure why this is so, since clearly we’ve been using that database, and the site’s been loading data from it. Checking the MariaDB Docker volume (i.e. the /var/lib/mysql folder), it’s 144MB in size, but the folder with the database itself is almost empty, with only db.opt.

The server does daily backups, and I’ve tried restoring the backups of the past week, but they all share the same problem: my Ghost database has no tables, shows an empty set.

Below is my MariaDB folder structure

drwxr-xr-x 6 systemd-coredump systemd-coredump 4.0K Aug 19 15:03 ./
drwx-----x 3 root             root             4.0K Aug  3 21:42 ../
drwx------ 2 systemd-coredump systemd-coredump 4.0K Aug  3 21:43 ghost-database/
-rw-rw---- 1 systemd-coredump systemd-coredump  18M Aug 16 00:13 aria_log.00000001
-rw-rw---- 1 systemd-coredump systemd-coredump   52 Aug 16 00:13 aria_log_control
-rw-rw---- 1 systemd-coredump systemd-coredump    9 Aug 19 15:03 ddl_recovery-backup.log
-rw-rw---- 1 systemd-coredump systemd-coredump    9 Aug 19 15:03 ddl_recovery.log
-rw-rw---- 1 systemd-coredump systemd-coredump  872 Aug 16 00:13 ib_buffer_pool
-rw-rw---- 1 systemd-coredump systemd-coredump  96M Aug 19 15:03 ib_logfile0
-rw-rw---- 1 systemd-coredump systemd-coredump  12M Aug 16 00:13 ibdata1
-rw-rw---- 1 systemd-coredump systemd-coredump  12M Aug 19 15:03 ibtmp1
-rw-rw---- 1 systemd-coredump systemd-coredump    0 Aug  3 21:42 multi-master.info
drwx------ 2 systemd-coredump systemd-coredump 4.0K Aug  3 21:43 mysql/
-rw-r--r-- 1 systemd-coredump systemd-coredump   14 Aug  3 21:42 mysql_upgrade_info
drwx------ 2 systemd-coredump systemd-coredump 4.0K Aug  3 21:42 performance_schema/
drwx------ 2 systemd-coredump systemd-coredump  12K Aug  3 21:42 sys/

root@server: /var/lib/docker/volumes/ghost_mariadb/_data# du -h .
600K	./sys
12M	./mysql
8.0K	./ghost-database
8.0K	./performance_schema
149M	.

Since I experienced the trouble, I’ve tried to migrate over to MySQL 8, but I can’t even use mysqldump, since mysqldump reproduces an empty database.

I’ve tried adding the line database__client=mysql (it wasn’t there before) to my env file, but although that change allows Ghost to connect to MariaDB, Ghost then creates a fresh installation, with a whole new set of tables.

Any help would be greatly appreciated!