My ghost is completely broken

If you’re looking for some help, it’s important to provide as much context as possible so that people are able to assist you. Try to always mention:

  • What’s your URL? This is the easiest way for others to debug your issue:
    a-view.org
  • What version of Ghost are you using?
    cli 1.4.1, server 2.1.0
  • What configuration?
    nginx with mysql on Digital Ocean
  • What browser? Firefox 77.01
  • What errors or information do you see in the console?

none: ghost doctor returns no errors:

✔ Checking system Node.js version
✔ Checking logged in user
✔ Ensuring user is not logged in as ghost user
✔ Checking if logged in user is directory owner
✔ Checking current folder permissions
✔ Checking operating system compatibility
✔ Checking for a MySQL installation
+ sudo systemctl is-active ghost_a-view-org
+ sudo systemctl reset-failed ghost_a-view-org
✔ Validating config
✔ Checking folder permissions
✔ Checking file permissions
✔ Checking content folder ownership
✔ Checking memory availability
✔ Checking binary dependencies

here is the output of ghost run:

The `ghost run` command is used by the configured Ghost process manager and for debugging. If you're not running this to debug something, you should run `ghost start` instead.
+ sudo node current/index.js
(node:17039) [DEP0126] DeprecationWarning: timers.active() is deprecated. Please use timeout.refresh() instead.
(node:17039) [DEP0096] DeprecationWarning: timers.unenroll() is deprecated. Please use clearTimeout instead.
(node:17039) [DEP0095] DeprecationWarning: timers.enroll() is deprecated. Please use setTimeout instead.
[2020-06-12 23:34:27] WARN Theme's file locales/en.json not found.
[2020-06-12 23:34:27] INFO Ghost is running in production...
[2020-06-12 23:34:27] INFO Your blog is now available on https://a-view.org/
[2020-06-12 23:34:27] INFO Ctrl+C to shut down
[2020-06-12 23:34:27] WARN Can't connect to the bootstrap socket (localhost 8000) ECONNREFUSED
[2020-06-12 23:34:27] WARN Tries: 0
[2020-06-12 23:34:27] WARN Retrying...
[2020-06-12 23:34:27] WARN Can't connect to the bootstrap socket (localhost 8000) ECONNREFUSED
[2020-06-12 23:34:27] WARN Tries: 1
[2020-06-12 23:34:27] WARN Retrying...
[2020-06-12 23:34:28] WARN Can't connect to the bootstrap socket (localhost 8000) ECONNREFUSED
[2020-06-12 23:34:28] WARN Tries: 2
[2020-06-12 23:34:28] WARN Retrying...
[2020-06-12 23:34:28] WARN Can't connect to the bootstrap socket (localhost 8000) ECONNREFUSED
[2020-06-12 23:34:28] INFO Ghost boot 3.38s

I haven’t touched localhost 8000 since my original setup. I don’t know why it is broken now.

The theme was fine for 2 years and hasn’t been touched so I don’t know what is hte probelm with the theme files. Perhaps no localization? Don’t need it.

I don’t understand the errors with timers.active, timers.enroll, timers.unenroll. This is some Ghost dependency that I have never setup or changed.

  • What steps could someone else take to reproduce the issue you’re having?

mysql was not running for some reason. I never touched it but some maintenance Digital Ocean did killed it. Nothing I could do would restarted. I ended up installing 5.7 and replacing 5.5. I can verify with the mysql cmdline clieint that I can connect locally to the databases.

lnotes.a-view.org is just a stub of a site with no posts. I could connect to it until moments ago.
I originally got an ECONN error on a-view.org but that went away with the reinstall of mysql.
Now I get a 502 bad gateway, which suggests some problem with the nginx config.

Now service xxx status shows that both mysql and nginx are running.

The symptom is that ghost start will not work:
It does through several steps and then the line that says ghost starting just flickers continously and effectively ghost hangs because it can’t start.

What’s infuriating is that this all worked for 2 years and only broke on 5/31/2020 after Digital Ocean maintenance.

Here is the output of nginx -t

sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

So, there is something wrong with the config of ghost. What else can I obtain for diagnosis?

Something is wrong between the web server (nginx) and ghost but I don’t understand what:

ghost-mgr@ghost-trial:/var/www/a-view/ghost$ sudo tail -f /var/log/nginx/error.log
2020/06/12 23:39:29 [error] 16870#16870: *13 connect() failed (111: Connection refused) while connecting to upstream, client: 70.67.177.65, server: a-view.org, request: "GET / HTTP/2.0", upstream: "http://127.0.0.1:2369/", host: "a-view.org"
2020/06/12 23:39:29 [error] 16870#16870: *13 connect() failed (111: Connection refused) while connecting to upstream, client: 70.67.177.65, server: a-view.org, request: "GET /favicon.ico HTTP/2.0", upstream: "http://127.0.0.1:2369/favicon.ico", host: "a-view.org"
2020/06/12 23:40:22 [error] 16870#16870: *13 connect() failed (111: Connection refused) while connecting to upstream, client: 70.67.177.65, server: a-view.org, request: "GET / HTTP/2.0", upstream: "http://127.0.0.1:2369/", host: "a-view.org"
2020/06/12 23:40:30 [error] 16870#16870: *13 connect() failed (111: Connection refused) while connecting to upstream, client: 70.67.177.65, server: a-view.org, request: "GET / HTTP/2.0", upstream: "http://127.0.0.1:2369/", host: "a-view.org"
2020/06/12 23:44:14 [error] 16870#16870: *21 connect() failed (111: Connection refused) while connecting to upstream, client: 77.65.86.180, server: a-view.org, request: "GET / HTTP/2.0", upstream: "http://127.0.0.1:2369/", host: "a-view.org"
2020/06/12 23:44:15 [error] 16870#16870: *23 connect() failed (111: Connection refused) while connecting to upstream, client: 77.65.86.180, server: a-view.org, request: "GET /favicon.ico HTTP/2.0", upstream: "http://127.0.0.1:2369/favicon.ico", host: "a-view.org", referrer: "https://a-view.org/"
2020/06/12 23:47:46 [error] 16870#16870: *25 connect() failed (111: Connection refused) while connecting to upstream, client: 70.67.177.65, server: lnotes.a-view.org, request: "GET / HTTP/2.0", upstream: "http://127.0.0.1:2369/", host: "lnotes.a-view.org"
2020/06/12 23:47:46 [error] 16870#16870: *25 connect() failed (111: Connection refused) while connecting to upstream, client: 70.67.177.65, server: lnotes.a-view.org, request: "GET /favicon.ico HTTP/2.0", upstream: "http://127.0.0.1:2369/favicon.ico", host: "lnotes.a-view.org"
2020/06/12 23:47:56 [error] 16870#16870: *28 connect() failed (111: Connection refused) while connecting to upstream, client: 70.67.177.65, server: lnotes.a-view.org, request: "GET / HTTP/2.0", upstream: "http://127.0.0.1:2369/", host: "lnotes.a-view.org"
2020/06/12 23:47:56 [error] 16870#16870: *28 connect() failed (111: Connection refused) while connecting to upstream, client: 70.67.177.65, server: lnotes.a-view.org, request: "GET /favicon.ico HTTP/2.0", upstream: "http://127.0.0.1:2369/favicon.ico", host: "lnotes.a-view.org"

Just to simplify things a bit I got rid of my http: connections so that there are only https: connections.

Note also that I have 2 instances of ghost running on the same machine serving two different websites. But, there is only 1 instance of nginx.

Believe me, this all used to work just fine.

I suspect the problem is a mismatch in ports between nginx and ghost.
How to diagnose?

Not an expert but those logs just show warnings. Warnings shouldn’t result in an error like 502 bad gateway when reaching your site. Could you post other logs such as Ghost’s internal logs, or your Nginx logs?

What does running

ghost doctor

give you?

Thanks. I posted above. Ghost doctor is helpful.

As someone suggested, I need to look at the nginx and ghost logs.

This is all very opaque to diagnose.

I posted ghost doctor above: it has always checked ouit ok.

Here is a tiny bit from the production log, when ghost would still load (now it won’t even start!):
Many people report this port 3306 connection problem. I don’t understand where the request to port 3306 is coming from: from my browser? from nginx to ghost? from ghost to nginx? is mysql involved in this? My config for nginx/ghost has explicit port settings.

a tiny fragment: "msg":“connect ECONNREFUSED 127.0.0.1:3306”,“time”:"2020-06-12

One humongous single logged event:

{"name":"Log","hostname":"ghost-trial","pid":692,"level":50,"err":{"domain":"https://a-view.org","code":"ECONNREFUSED","message":"connect ECONNREFUSED 127.0.0.1:3306","stack":"Error: connect ECONNREFUSED 127.0.0.1:3306\n    at Object.exports._errnoException (util.js:1020:11)\n    at exports._exceptionWithHostPort (util.js:1043:20)\n    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1099:14)\n    --------------------\n    at Protocol._enqueue (/var/www/a-view/ghost/versions/2.1.0/node_modules/mysql/lib/protocol/Protocol.js:145:48)\n    at Protocol.handshake (/var/www/a-view/ghost/versions/2.1.0/node_modules/mysql/lib/protocol/Protocol.js:52:23)\n    at Connection.connect (/var/www/a-view/ghost/versions/2.1.0/node_modules/mysql/lib/Connection.js:130:18)\n    at /var/www/a-view/ghost/versions/2.1.0/node_modules/knex/lib/dialects/mysql/index.js:109:18\n    at Promise._execute (/var/www/a-view/ghost/versions/2.1.0/node_modules/bluebird/js/release/debuggability.js:303:9)\n    at Promise._resolveFromExecutor (/var/www/a-view/ghost/versions/2.1.0/node_modules/bluebird/js/release/promise.js:483:18)\n    at new Promise (/var/www/a-view/ghost/versions/2.1.0/node_modules/bluebird/js/release/promise.js:79:10)\n    at Client_MySQL.acquireRawConnection (/var/www/a-view/ghost/versions/2.1.0/node_modules/knex/lib/dialects/mysql/index.js:104:12)\n    at create (/var/www/a-view/ghost/versions/2.1.0/node_modules/knex/lib/client.js:283:23)\n    at tryPromise (/var/www/a-view/ghost/versions/2.1.0/node_modules/tarn/lib/Pool.js:366:22)\n    at tryPromise (/var/www/a-view/ghost/versions/2.1.0/node_modules/tarn/lib/utils.js:57:20)\n    at Promise (/var/www/a-view/ghost/versions/2.1.0/node_modules/tarn/lib/Pool.js:366:5)\n    at Promise._execute (/var/www/a-view/ghost/versions/2.1.0/node_modules/bluebird/js/release/debuggability.js:303:9)\n    at Promise._resolveFromExecutor (/var/www/a-view/ghost/versions/2.1.0/node_modules/bluebird/js/release/promise.js:483:18)\n    at new Promise (/var/www/a-view/ghost/versions/2.1.0/node_modules/bluebird/js/release/promise.js:79:10)\n    at callbackOrPromise (/var/www/a-view/ghost/versions/2.1.0/node_modules/tarn/lib/Pool.js:357:10)"},"msg":"connect ECONNREFUSED 127.0.0.1:3306","time":"2020-06-12T00:39:44.981Z","v":0}

Since this time, I’ve reinstalled MySQL, updated Ghost-cli and server and modified the .conf files.

Without fixing this a bit more not sure that Nginx logs will help now that ghost won’t even start. They’ll just show a failure to connect to the application server.

I have backed up my content and I’ll cross my fingers that the database will be OK. I will uninstall ghost and reinstall. It’s a shame that uninstalling ghost deletes data. That shouldn’t be the case.

Seems like you got it up and running again (: If you don’t want Ghost to uninstall data, you should install it via Docker with an external volume. Additionally, you can still access the logs even if Ghost is unable to start. Grab an FTP client and remote into your server, and go to your Ghost content folder. There you can find the logs as well as any error logs there.

Yup. The documentation is pretty thorough and the logs are all there.

It is all up and working. I was disappointed in the responses here. To be sure there was a big red herring in the troubles I reported. My site and VPS hadn’t been updated enough so it was necessary to do lots of updating: MySQL, NPM, nginx not so much (there were updates of course, but enhancements with no breaking changes–nginx is a rock), and of course ghost, which was most troubling to update.

The big problem is that an upgrade of 2+ versions of ghost seems broken. What fixed it was never suggested here but in independent blogs that suggest things that are “not recommended” or not approved here. Basically, I crossed my fingers that the databases would be all right. This turned out to be true. I made one offline back up of the ghost directories of the two blogs and one on the VPS (so it would be faster to copy selectively back into place). Then, it was necessary to completely uninstall and reinstall ghost. For 1 blog, the uninstall didn’t work and I had to rm -R. Since I wanted to leave nginx, MySQL, Let’s Encrypt in place with all of the existing users and passwords it was ok to not thoroughly purge–in fact, essential. This is the strategy recommended on an independent blog. For the 2nd blog I didn’t even bother with ghost uninstall and just deleted the ghost directory.

I tested out the approach on the smaller of the 2 blogs, which is really just a stub for experimenting–it has no content. This worked out the kinks, which were settings and finding out really how little of the old ghost directories were actually needed–not so much stuff, but all critical.

Then I did the same for the larger blog: this is a front door to 3 complete blogs (shared instance, just split apart in top level nav) in Ghost, plus links on the front door to static web sites generated with Hugo.

The weirdness of the database I’ll never understand. I don’t know what caused it because it happened without my touching anything. One night it crashed and wouldn’t restart. I think it ran out of disk space because dropbox was syncing stuff that should have been excluded (all it needs to sync are tiny markdown files for the statically generated web pages–but, I had accreted some new directories that I hadn’t “excluded”). I think this is the best explanation.

The real breakage was the failure of Ghost to be able to update itself across 2 major versions (I was already > 1.0 so at least the cli could be used).

I had only 1 config problem as I frantically tried things that had nothing to do with the real problem. For the smaller blog, the production.json and nginx server blocks were correct. The larger blog had one leettle error that was huge: I get the port crossed with the smaller blog, which–of course broke both. But, the breakage of Ghost itself had masked this because I couldn’t even start Ghost after upgrading. After the uninstall and reinstall it started right up (both times) and then a 502 and looking at ports for running processes showed there was a port mismatch.

Also, “bootstrap-socket” is no longer used in the most recent versions of ghost–the new config for nginx simply doesn’t use it. Not clear whether this caused ghost to not start. I doubt it. With new versions of Ghost, it is not only Ghost that changes. Some of the correct or preferred settings also change. Not everyone realizes what so there is some out-of-date information about what the setting should be (note elimination of bootstrap-socket).

I got both working–the second one worked immediately with only one change for the correct port.

I then had to copy assets and some settings from the backed up old ghost directories. I did not do a bulk copy over the new (that would have been a disaster!): I compared the two and found that only a few things needed to be changed:

  • settings/routes.yaml
  • copy over the images (entire)
  • copy over the old data.json files just to have the complete history–not functionally necessary
  • not the logs: completely out of date, so let’s start fresh
  • not the versions
  • not config.production.json: the new versions were created correctly during the prompting sequence because I had the database name, db user, and db password in a secure password store–so I just answered the prompts. Of course, the url is easy and localhost is std for a simple default installation of MySQL. The databases flowed into the new installation w/o a hitch.
  • not systems/files containing the source config files to symlink into /etc/nginx/sites-enabled. Essential to look at the old and new versions and prefer the new except for content related items from the old (locations for static files in my case).

It was unfortunate that no one here seemed to even have a hint of what needed to be done. When I pieced it together from external sources and looked carefully at the thorough install/config documenation and realized I could salvage everything–all I needed was 2 clean instances of Ghost and complete reinstallation was the only way to do that.

Once I had a strategy that would work, it took a little over an hour to do twice. Most of that time was making the offline backup of the two ghost directories across the internet. But, when I actually did the work, it was WAY quicker to just cp ghost directories to another place on the vps and was quick to copy “locally” on the vps the fragments I needed in the new installation. The fresh installs of ghost were very quick. I needed to set permissions and owner:group for the ghost directories and create one symlink because I broke the config when I was scurrying around earlier.

The cli is a bit of a mixed bag. There are great things it does for a 1st time “correct” install. Running it and skipping things works pretty well to build “config” for the reinstall if you have all of the right information for the prompts. It’s not very good as a diagnostic tool because it can give a false positive: it says all is ok when all is not ok. That is really frustrating: it is perfectly obvious the blogs don’t work or ghost itself won’t start but the doctor has no idea why. On the other hand, it doesn’t give false negatives: when the cli said something was broken it was always right and gave correction suggestions that worked 1st time. And that was really helpful.

Advice to the team: ghost needs diagnostics for upgrading itself. Or–if it detects it is upgrading for too much of a jump, it should offer to do a safe uninstall and safe fresh install. This would be made safer by ghost-cli preserving the old stuff by renaming and then giving one of its helpful hints of what to bring back. There is not that much user stuff in the ghost directory (by type: images can obviously be big but it is a safe copy or rename in place). It wouldn’t be that hard to do a safe uninstall/reinstall process. I managed to do it manually with very little work simply by inspecting the contents of the directories.

All in all: very frustrating to figure out what to do and relatively easy to actually do it. I’d say that self-hosting Ghost is much harder than self-hosting Wordpress or Hugo (not fair–it’s just files, a theme, and one executable). Ghost is more polished so it should have shown itself better. But, I think there is over-reliance on the -cli as a black-box. The black-box approach needs a deeper mode that exposes more of the underpinnings: essential when things go wrong (too often judging from the forum) and when a config is acceptable but non-standard. This seems possible. Despite all of the dependencies, under the hood the config is not that complex.

Most of the time spent blogging is writing not doing sys admin. Then Ghost is really good. One can go months and months without doing any sys admin. But, when it comes up, it is a truly a pain when we do not do it for a living.

Yes, while the cli is recommended, it’s really only made for people who really don’t want to have to do anything related to server side setup. Ghost doctor merely checks if the install is done correctly, and doesn’t show many other errors.

I think the docs did say somewhere not to upgrade major versions of Ghost immediately, unless you are at the latest minor version. You should upgrade minor versions first to ensure non breaking changes, since the major versions usually break compatibility with previous major versions.

The sys admin thing is a very polarized thing - on one hand, Ghost doesn’t have as large a community as wordpress. As a result, many errors are new or unheard of and not much help can be found there other than official responses from the staff - but again, there are not that many staff so responses can be slow or non existent. If you are self hosting Ghost then I guess you have to learn by yourself the hard way.

Thanks.

Is there an easy way to choose the next version in a sequence rather than the latest version? Would it work better to upgrade one item at a time?

Yes, you can read the CLI docs to update to a specific version. Minor versions usually don’t break. So you should update to the latest minor version first, and then perform a major upgrade after.

Make sure to backup before any updates!

Saw that.

Have to say Wordpress works a lot better and is nearly self-maintaining. The Ghost developers clearly had some big dispute with Mullenweg but that should not be our concern.

Ghost seems to evolve very slowly and not really in particularly meaningful directions. But, it started well.

I think they each have their own issues - Wordpress being coded on PHP and that their general codebase is a mess, as well as having constant security issues and plugins breaking with updates. Ghost being that it’s a very small team and so moves quite slowly and if what they value is not what you want out of Ghost (there are a lot of posts like that), then you’re out of luck.

The language something is written in is a “cool kids” kind of thing. Unless there is a tangible performance difference or a language specific UI difference it doesn’t matter unless you are a maintainer of that code.

The language ghost is written in doesn’t matter. As users we may only interact with ghost via its GUI (client or web) or through its config files (horrific, but rare). Ghost is a much more closed system: good or bad depending on your preference. Generally you can do far more with Wordpress through its dashboard than is remotely possible with Ghost by any means. You may feel better because Ghost is written in Nodejs (why not Haskell if they were really cool) but there is no tangible benefit to users. You could claim performance but that is really because caching is built in.

Like all things built of code by human hands, some good things and some bad things.

  • Lewis