I am unable to start Ghost and its preventing NGINX from also starting due to a .well-known directory issue. How do I fix it?

Hi there,

I have been running my website powered by Ghost successfully for several months now. Last night I updated my server, and then updated Ghost. Since the update, I am unable to start Ghost, and it is preventing me from even starting NGINX.

Here are my Ghost and System Details

Domain = mythbound.online

GHOST:

Ghost-CLI version: 1.24.2
Ghost version: 5.65.0 (at /var/www/mythbound/website/ghost)

SYSTEM:

OS: Ubuntu 22.04.3 LTS
Kernel: 5.15.0-84-generic
Node: v18.17.1
NPM: 9.6.7
NGINX: nginx/1.18.0 (Ubuntu)

Running ghost doctor gives the following:


✔ Checking system Node.js version - found v18.17.1
✔ Checking logged in user
✔ Ensuring user is not logged in as ghost user
✔ Checking if logged in user is directory owner
✔ Checking current folder permissions
System checks failed with message: 'Missing package(s): nginx'
Some features of Ghost-CLI may not work without additional configuration.
For local installs we recommend using `ghost install local` instead.
? Continue anyway? Yes
System stack check skipped
ℹ Checking system compatibility [skipped]
✔ Checking for a MySQL installation
+ sudo systemctl is-active ghost_mythbound-online
Instance is currently running
ℹ Validating config [skipped]
✔ Checking folder permissions
✔ Checking file permissions
✔ Checking content folder ownership
✔ Checking memory availability
✔ Checking binary dependencies
✔ Checking free space
✔ Checking systemd unit file
✔ Checking systemd node version - found v18.17.1

That is to be expected as I cannot start NGINX, but at least you can see that all the perms are set correctly, and that Ghost is finding the right Node version.

When I test NGINX config the tests pass perfectly fine, as showing here:


sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

When I look at the NGINC logs, I see that these errors occurred while trying to start NGIX.

2023/09/22 00:23:21 [error] 582065#582065: *142949 open() "/var/www/mythbound/website/ghost/system/nginx-root/.well-known/acme-challenge/6s-F8N9doZMrknm_58YhxG5GPIORbgjyTU8soZAJEWI" failed (2: No such file or directory), client: 18.222.24.83, server: mythbound.online, request: "GET /.well-known/acme-challenge/6s-F8N9doZMrknm_58YhxG5GPIORbgjyTU8soZAJEWI HTTP/1.1", host: "mythbound.online"
2023/09/22 00:23:21 [error] 582065#582065: *142950 open() "/var/www/mythbound/website/ghost/system/nginx-root/.well-known/acme-challenge/6s-F8N9doZMrknm_58YhxG5GPIORbgjyTU8soZAJEWI" failed (2: No such file or directory), client: 23.178.112.200, server: mythbound.online, request: "GET /.well-known/acme-challenge/6s-F8N9doZMrknm_58YhxG5GPIORbgjyTU8soZAJEWI HTTP/1.1", host: "mythbound.online"
2023/09/22 00:23:21 [error] 582065#582065: *142951 open() "/var/www/mythbound/website/ghost/system/nginx-root/.well-known/acme-challenge/6s-F8N9doZMrknm_58YhxG5GPIORbgjyTU8soZAJEWI" failed (2: No such file or directory), client: 35.88.153.50, server: mythbound.online, request: "GET /.well-known/acme-challenge/6s-F8N9doZMrknm_58YhxG5GPIORbgjyTU8soZAJEWI HTTP/1.1", host: "mythbound.online"


The issue seems to be that the .well-known directory does not exist, and NGINX is trying to look there for SSL certificates.

Here is my NGINX server block for my ghost powered website:


server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    server_name mythbound.online;
    root /var/www/mythbound/website/ghost/system/nginx-root; # Used for acme.sh SSL verification (https://acme.sh)

    ssl_certificate /etc/letsencrypt/mythbound.online/fullchain.cer;
    ssl_certificate_key /etc/letsencrypt/mythbound.online/mythbound.online.key;
    include /etc/nginx/snippets/ssl-params.conf;

    location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $http_host;
        proxy_pass http://127.0.0.1:2368;

    }

    location ~ /.well-known {
        allow all;
    }

    client_max_body_size 1g;
}

I do not know how to fix this, can any of you help me? If I just remove this part of the server block:


    }

    location ~ /.well-known {
        allow all;
    }

Will that fix the problem?

Is there a better way to fix this issue?

Everything was working fine until last night when I updated the server and then updated Ghost.

Thanks

Jay

Welcome to the Ghost community, @j-telford.

Please share the output from which nginx.

Hi, here is the response you asked for. Running which nginx gives me:


/usr/sbin/nginx

Okay, check that Nginx is running with sudo systemctl status nginx (note that nginx -t works when the server isn’t running.)

Doing so, shows me that NGINX is not running:


× nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Sat 2023-09-23 10:56:35 UTC; 1h 36min ago
       Docs: man:nginx(8)
    Process: 11084 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
    Process: 11085 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE)
        CPU: 143ms

Sep 23 10:56:33 zod.jaytelford.me systemd[1]: Starting A high performance web server and a reverse proxy server...
Sep 23 10:56:33 zod.jaytelford.me nginx[11085]: nginx: [emerg] bind() to [::]:80 failed (98: Unknown error)
Sep 23 10:56:33 zod.jaytelford.me nginx[11085]: nginx: [emerg] bind() to [::]:80 failed (98: Unknown error)
Sep 23 10:56:34 zod.jaytelford.me nginx[11085]: nginx: [emerg] bind() to [::]:80 failed (98: Unknown error)
Sep 23 10:56:34 zod.jaytelford.me nginx[11085]: nginx: [emerg] bind() to [::]:80 failed (98: Unknown error)
Sep 23 10:56:35 zod.jaytelford.me nginx[11085]: nginx: [emerg] bind() to [::]:80 failed (98: Unknown error)
Sep 23 10:56:35 zod.jaytelford.me nginx[11085]: nginx: [emerg] still could not bind()
Sep 23 10:56:35 zod.jaytelford.me systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Sep 23 10:56:35 zod.jaytelford.me systemd[1]: nginx.service: Failed with result 'exit-code'.
Sep 23 10:56:35 zod.jaytelford.me systemd[1]: Failed to start A high performance web server and a reverse proxy server.

Attempting to start it gives me the error:


Job for nginx.service failed because the control process exited with error code.
See "systemctl status nginx.service" and "journalctl -xeu nginx.service" for details.

The journal shows:


░░ Subject: A start job for unit nginx.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit nginx.service has finished with a failure.
░░
░░ The job identifier is 5084 and the job result is failed.

It is really confusing me because everything was working till I ran the server update and updated ghost last night. Other than that, no files have been changed and the server block is exactly the same as it was before it stopped working.

That was my suspicion. It looks like you have another web server or server listening on port 80.

I am really confused by that, because I should not have anything else listening on port 80. I looked at all the listening ports that are currently in use on my server, and this is what I got.


sudo sudo netstat -tunlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      979/sshd: /usr/sbin
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      884/systemd-resolve
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      1032/mysqld
tcp        0      0 127.0.0.1:2368          0.0.0.0:*               LISTEN      1139/node
tcp        0      0 127.0.0.1:33060         0.0.0.0:*               LISTEN      1032/mysqld
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      923/redis-server 12
tcp6       0      0 :::22                   :::*                    LISTEN      979/sshd: /usr/sbin
tcp6       0      0 ::1:6379                :::*                    LISTEN      923/redis-server 12
udp        0      0 redacted:53             0.0.0.0:*                           884/systemd-resolve
udp        0      0 redacted:68             0.0.0.0:*                           882/systemd-network
udp6       0      0 redacted:546 :::*                                           882/systemd-network

As you can see, there is nothing currently using port 80. So I am not sure what is going on then.

This is strange. I have read that this can occur if Nginx is attempting to bind to port 80 more than once, or you’re trying to listen on [::]:80 when IPV6 isn’t configured. Do you still have the default server in your Nginx config? If so, remove this from sites-enabled.

Still no joy I’m afraid. I moved the default server block out of both sites-enabled (still would not start) so i moved it out of sites-available, and the server still would not start. At this point, I am wondering if I should just uninstall Ghost, move all my other server blocks out of the sites-available dir, purge nginx and start from scratch.

I don’t think the issue is related to Ghost; Ghost can’t start because of the Nginx failed state. Instead, try removing all symbolic links from sites-enabled and then add your Ghost site first, followed by the others one at a time.

Use …

nginx -t
systemctl reload nginx

I believe the problem is related to renewing the SSL certificate for the domain, because it ties together everything here:

  1. Certs are renewed every few months, explaining why something can appear to “suddenly stop working”.
  2. The “.well-known” directory is used during the renewal process.
  3. One method of renewal involves stopping Nginx just briefly, starting an alternate web server on port 80, renewing the certificate, then stopping that and starting Nginx again. Sounds like this process got stuck.

Honestly, I recommend rebooting the server. I presume that Nginx is set to start on boot and the certbot server is not, so if my theory holds, it will reset that state.

Then run:

sudo certbot renew

To manually kick-off the renewal process and see if it succeeds or fails.

Good luck!

2 Likes

I removed everything and started to add them back one at a time. Testing and then restarting with each one.

I’ve found the culprit.

I had set up a custom redirect, from a subdomain to a discord invite link using NGINX.

It was this redirect that was the problem.

I removed the redirect and everything works again.

Thanks everyone.

3 Likes

2 posts were split to a new topic: Welcome to Nginx message