External Caddy Reverse Proxy and "cannot stop container" error for ghost-db-1

Developer help Self-hosting
1

Hello all,

I’m attempting to self-host Ghost on my ubuntu 24.04 server running rootless docker and a caddy reverse proxy. The caddy reverse proxy is running on the server itself (not in a container) and is the latest version from the caddy repositories. This reverse proxy sits in front of all my self-hosted services, whether they are containers or running directly on a server. I’d like to keep in this way and not have two different reverse proxies for a number of technical reasons on my self-hosted setup.

So I’d like help with two things:

  1. verifying my compose file and caddy snippet edits are correct to make ghost work with an external caddy reverse proxy.

  2. help with understanding why (after I successfully deploy ghost and login in for the first time) anytime I try to do a docker compose down for the ghost compose file, I get this error (and whether the two things are related) which results in ghost not working anymore.

✘ Container ghost-db-1 Error while Stopping 14.0s
Error response from daemon: cannot stop container

Ok, first the edits I made to make ghost-docker work with an external caddy reverse proxy:

  1. I copied the ghost-docker repository to my computer. Note: I copied it to a directory in my home directory, not to /opt/ghost… but i don’t think this matters?
  2. I filled out the .env per install instructions.
    1. I did change the UPLOAD_LOCATION and MYSQL_DATA_LOCATION to a different directory that is regularly backed up. I used absolute paths, not relative paths.
  3. I commented out the caddy service definition in the compose file.
  4. I added the following to the ghost service definition in the compose file so my caddy reverse proxy (running on the same server) can access ghost.

    1. ports:

      • 127.0.0.1:2368:2368
  5. I copied the caddy config from the Caddyfile in the ghost-docker git into my Caddyfile and made the necessary edits.
  6. I copied the caddy snippets folder over and replaced the activitypub variable in the ActivityPub snippet to point directly to https://ap.ghost.org, since there would be no way that it’d know the value of a variable that is only defined inside of docker…..

I run “docker compose up” in the ghost directory and things start perfectly! I’m able to create the admin user, create a site, and email (using smtp.gmail.com) and things seem to work great.

So i think i got #1 right? Is there anything I missed as far as getting ghost-docker running with an external reverse proxy?

break break

Ok, here is where things start to break down. If i run “docker compose down” to bring my containers down i get the following:

docker compose down
[+] Running 2/2
:check_mark: Container ghost-ghost-1 Removed 0.3s
✘ Container ghost-db-1 Error while Stopping 14.0s
Error response from daemon: cannot stop container: e91d03f0dd72f61fe01ebaddd7eeb68bf18d44277a55cae447d4cebfce9eb20e: permission denied

No matter what I try (e.g. docker kill) nothing works. The only thing that does work is restarting the docker process:

sudo systemctl --user restart docker.service

and then the ghost-db-1 container is finally stopped and I can remove the container. If however, I try to bring ghost-docker back up using “docker compose up -d” i get the following error:

docker compose up -d
[+] Running 2/2
✘ Container ghost-db-1 Error 100.8s
:check_mark: Container ghost-ghost-1 Created 0.0s
dependency failed to start: container ghost-db-1 is unhealthy

The docker logs show this for ghost-deb-1

db-1 | 2025-11-12T14:31:04.123178Z 0 [ERROR] [MY-010338] [Server] Can’t find error-message file ‘/usr/share/mysql-8.0/errmsg.sys’. Check error-message file location and ‘lc-messages-dir’ configuration directive.
db-1 | 2025-11-12T14:31:04.126802Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
db-1 | 2025-11-12T14:31:04.149420Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
db-1 | 2025-11-12T14:31:05.149529Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
db-1 | 2025-11-12T14:31:06.149681Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11

and I am right back where i started. i could try and disable the lock? but even if that works, that doesn’t solve the problem that i can’t use “docker compose down” without encountering this error.

If I nuke everything, and restart the install from the beginning, the same thing happens. I can get the site up and running, but as soon as I try and bring ghost-docker down using “docker compose down,” the whole thing goes to

Any help would be greatly appreciated!

Brian

2

The database is calling out a permission error. Do you have the compose file on hand so we can analyze it’s contents? Along with the .env file with sensitive contents redacted?
Additionally, I prefer using NGINX for reverse proxy and have not used Caddy, so take my assistance with a grain of salt.

  1. I did change the UPLOAD_LOCATION and MYSQL_DATA_LOCATION to a different directory that is regularly backed up. I used absolute paths, not relative paths.

Reading your message again, does the docker user have permission to write in this directory? Also, does your user have permission for docker? If not: sudo usermod -aG docker $USER
If that fixes it, I would adivse running: docker compose down -v and start fresh. Then bring the container back up.

  • 127.0.0.1:2368:2368

Additionally, if you’re connecting internally from a docker image to an outside service, shouldn’t you use a different IP Pool? Docker has it’s own set of IP’s and using 127.x.x is going to be localhost to INSIDE of that docker container. If your compose file is calling for a service external to the container itself, you need to change the ip. You can find this by running ‘ipconfig a’. Correct me if I’m wrong to assume your compose file is doing such though.

3

KBExit - thx for the reply!

“Reading your message again, does the docker user have permission to write in this directory?” - Yes. The easiest way to confirm this is when I run “docker compose up” initially, the directory is empty, and the two folders …/data/ghost and ../data/mysql are created. In addition, there are lots of files and folders created in the /data/mysql folder….

“Additionally, if you’re connecting internally from a docker image to an outside service, shouldn’t you use a different IP Pool? Docker has it’s own set of IP’s and using 127.x.x is going to be localhost to INSIDE of that docker container.” - i’m confused….. i believe all this does is bind the docker to the localhost IP address of the host to 2368 so that my caddy reverse proxy can access it via reverse-proxy 127.0.0.1:2368. I do this for all my containers that use compose so that caddy can access them (though the ports chart)… maybe i’m missing something?

About to jump on a call, but will post the compose and env file here soon. thank you again for your response!

4

Here is the compose file:

# yaml-language-server: $schema=https://raw.githubusercontent.com/compose-spec/compose-spec/main/schema/compose-spec.json
services:
  ghost:
    # Do not alter this without updating the Tinybird Sync container as well
    image: ghost:${GHOST_VERSION:-6-alpine}
    restart: unless-stopped
    ports:
      - 127.0.0.1:2368:2368
    expose:
      - "127.0.0.1:${GHOST_PORT:-2368}:2368"
    # This is required to import current config when migrating
    env_file:
      - .env
    environment:
      NODE_ENV: production
      url: https://${DOMAIN:?DOMAIN environment variable is required}
      admin__url: ${ADMIN_DOMAIN:+https://${ADMIN_DOMAIN}}
      database__client: mysql
      database__connection__host: db
      database__connection__user: ${DATABASE_USER:-ghost}
      database__connection__password: ${DATABASE_PASSWORD:?DATABASE_PASSWORD environment variable is required}
      database__connection__database: ghost
      tinybird__tracker__endpoint: https://${DOMAIN:?DOMAIN environment variable is required}/.ghost/analytics/api/v1/page_hit
      tinybird__adminToken: ${TINYBIRD_ADMIN_TOKEN:-}
      tinybird__workspaceId: ${TINYBIRD_WORKSPACE_ID:-}
      tinybird__tracker__datasource: analytics_events
      tinybird__stats__endpoint: ${TINYBIRD_API_URL:-https://api.tinybird.co}
    volumes:
      - ${UPLOAD_LOCATION:-./data/ghost}:/var/lib/ghost/content
    depends_on:
      db:
        condition: service_healthy
      tinybird-sync:
        condition: service_completed_successfully
        required: false
      tinybird-deploy:
        condition: service_completed_successfully
        required: false
      activitypub:
        condition: service_started
        required: false
    networks:
      - ghost_network

  db:
    image: mysql:8.0.44@sha256:f37951fc3753a6a22d6c7bf6978c5e5fefcf6f31814d98c582524f98eae52b21
    restart: unless-stopped
    expose:
      - "3306"
    environment:
      MYSQL_ROOT_PASSWORD: ${DATABASE_ROOT_PASSWORD:?DATABASE_ROOT_PASSWORD environment variable is required}
      MYSQL_USER: ${DATABASE_USER:-ghost}
      MYSQL_PASSWORD: ${DATABASE_PASSWORD:?DATABASE_PASSWORD environment variable is required}
      MYSQL_DATABASE: ghost
      MYSQL_MULTIPLE_DATABASES: activitypub
    volumes:
      - ${MYSQL_DATA_LOCATION:-./data/mysql}:/var/lib/mysql
      - ./mysql-init:/docker-entrypoint-initdb.d
    healthcheck:
      test: mysqladmin ping -p$$MYSQL_ROOT_PASSWORD -h 127.0.0.1
      interval: 1s
      start_period: 30s
      start_interval: 10s
      retries: 120
    networks:
      - ghost_network

  traffic-analytics:
    image: ghost/traffic-analytics:1.0.20@sha256:a72573d89457e778b00e9061422516d2d266d79a72a0fc02005ba6466e391859
    restart: unless-stopped
    expose:
      - "3000"
    volumes:
      - traffic_analytics_data:/data
    environment:
      NODE_ENV: production
      PROXY_TARGET: ${TINYBIRD_API_URL:-https://api.tinybird.co}/v0/events
      SALT_STORE_TYPE: ${SALT_STORE_TYPE:-file}
      SALT_STORE_FILE_PATH: /data/salts.json
      TINYBIRD_TRACKER_TOKEN: ${TINYBIRD_TRACKER_TOKEN:-}
      LOG_LEVEL: debug
    profiles: [analytics]
    networks:
      - ghost_network

  activitypub:
    image: ghcr.io/tryghost/activitypub:1.1.0@sha256:39c212fe23603b182d68e67d555c6b9b04b1e57459dfc0bef26d6e4980eb04d1
    restart: unless-stopped
    expose:
      - "8080"
    volumes:
      - ${UPLOAD_LOCATION:-./data/ghost}:/opt/activitypub/content
    environment:
      # See https://github.com/TryGhost/ActivityPub/blob/main/docs/env-vars.md
      NODE_ENV: production
      MYSQL_HOST: db
      MYSQL_USER: ${DATABASE_USER:-ghost}
      MYSQL_PASSWORD: ${DATABASE_PASSWORD:?DATABASE_PASSWORD environment variable is required}
      MYSQL_DATABASE: activitypub
      LOCAL_STORAGE_PATH: /opt/activitypub/content/images/activitypub
      LOCAL_STORAGE_HOSTING_URL: https://${DOMAIN}/content/images/activitypub
    depends_on:
      db:
        condition: service_healthy
      activitypub-migrate:
        condition: service_completed_successfully
    profiles: [activitypub]
    networks:
      - ghost_network

  # Suporting Services

 tinybird-login:
    build:
      context: ./tinybird
      dockerfile: Dockerfile
    working_dir: /home/tinybird
    command: /usr/local/bin/tinybird-login
    volumes:
      - tinybird_home:/home/tinybird
      - tinybird_files:/data/tinybird
    profiles: [analytics]
    networks:
      - ghost_network
    tty: false
    restart: no

  tinybird-sync:
    # Do not alter this without updating the Ghost container as well
    image: ghost:${GHOST_VERSION:-6-alpine}
    command: >
      sh -c "
        if [ -d /var/lib/ghost/current/core/server/data/tinybird ]; then
          rm -rf /data/tinybird/*;
          cp -rf /var/lib/ghost/current/core/server/data/tinybird/* /data/tinybird/;
          echo 'Tinybird files synced into shared volume.';
        else
          echo 'Tinybird source directory not found.';
        fi
      "
    volumes:
      - tinybird_files:/data/tinybird
    depends_on:
      tinybird-login:
        condition: service_completed_successfully
    networks:
      - ghost_network
    profiles: [analytics]
    restart: no

  tinybird-deploy:
    build:
      context: ./tinybird
      dockerfile: Dockerfile
    working_dir: /data/tinybird
    command: >
      sh -c "
        tb-wrapper --cloud deploy --allow-destructive-operations
      "
    volumes:
      - tinybird_home:/home/tinybird
      - tinybird_files:/data/tinybird
    depends_on:
      tinybird-sync:
        condition: service_completed_successfully
    profiles: [analytics]
    networks:
      - ghost_network
    tty: true

  activitypub-migrate:
    image: ghcr.io/tryghost/activitypub-migrations:1.1.0@sha256:b3ab20f55d66eb79090130ff91b57fe93f8a4254b446c2c7fa4507535f503662
    environment:
      MYSQL_DB: mysql://${DATABASE_USER:-ghost}:${DATABASE_PASSWORD:?DATABASE_PASSWORD environment variable is required}@tcp(db:3306)/activitypub
    networks:
      - ghost_network
    depends_on:
      db:
        condition: service_healthy
    profiles: [activitypub]
    restart: no


volumes:
  tinybird_files:
  tinybird_home:
  traffic_analytics_data:

networks:
  ghost_network:
    external: true

Here is the .env file

# Use the below flags to enable the Analytics or ActivityPub containers as well
# COMPOSE_PROFILES=analytics,activitypub
# COMPOSE_PROFILES=analytics

# Ghost domain
# Custom public domain Ghost will run on
DOMAIN=**************

# Ghost Admin domain
# If you have Ghost Admin setup on a separate domain uncomment the line below and add the domain
# You also need to uncomment the corresponding block in your Caddyfile
ADMIN_DOMAIN=************

# Database settings
# All database settings must not be changed once the database is initialised
DATABASE_ROOT_PASSWORD=***********
# DATABASE_USER=optionalusername
DATABASE_PASSWORD=************

# ActivityPub
# If you'd prefer to self-host ActivityPub yourself uncomment the line below
# ACTIVITYPUB_TARGET=activitypub:8080

# Tinybird configuration
# If you want to run Analytics, paste the output from `docker compose run --rm tinybird-login get-tokens` below
TINYBIRD_API_URL=************
TINYBIRD_WORKSPACE_ID=********
TINYBIRD_ADMIN_TOKEN=*********
TINYBIRD_TRACKER_TOKEN=********

# Ghost configuration (https://ghost.org/docs/config/)

# SMTP Email (https://ghost.org/docs/config/#mail)
# Transactional email is required for logins, account creation (staff invites), password resets and other features
# This is not related to bulk mail / newsletter sending
mail__transport=SMTP
mail__options__host=smtp.gmail.com
mail__options__port=587
mail__options__secure=false
mail__options__auth__user=**********
mail__options__auth__pass=**********
mail__from=***********

# Advanced customizations

# Force Ghost version
# You should only do this if you need to pin a specific version
# The update commands won't work
# GHOST_VERSION=6-alpine

# Port Ghost should listen on
# You should only need to edit this if you want to host
# multiple sites on the same server
# GHOST_PORT=2368

# Data locations
# Location to store uploaded data
UPLOAD_LOCATION=/path/to/data/ghost

# Location for database data
MYSQL_DATA_LOCATION=/path/to/data/mysql
5

So, rootless Docker is interesting, especially if you pair it with bind volumes rather than named volumes (which ghost-docker does). There could very well be a mismatch between the user running Docker and whatever happens inside the container.

After you’ve started Ghost (which creates MySQL structures), can you check file ownership and your user id?

ls -ln /path/to/data/mysql | head -5
id -u
6

so i had a suspicion that it might be rootless docker, but hoping it wasn’t because 1) i run everything else rootless, and don’t want to have two different stacks, and 2) figured if it was a rootless issue someone else would have noticed, as I thought most people had moved to rootless for the security.

Also… everything works fine. I can login in and use ghost and everything (see OP). It all just stops working when I do “docker compose down.” Before that, I can build a site, host it, etc… all fine.

i’d actually done what you are suggesting, and should have put it in the original post

ls -ln
total 8
drwxr-xr-x 11 100999 1000 4096 Nov 12 06:56 ghost
drwxr-xr-x  8 100998 1000 4096 Nov 12 09:31 mysql

perhaps it was motivated reasoning :smiley: (like i didn’t want that to be the answer), but I will note, that i checked some of my other docker, and then have odd users too, but seem to work fine.

For example, my immich folder has the following:

drwx------ 20 100998 1000 4096 Nov 12 08:57 immich

and my authelia folder’s redis_alpine folder also has that user:

drwxr-xr-x 2 100998 1000 4096 Nov 12 09:57 redis_alpine

As you can see, immich and authelia both use the 100998 user-id as well, but with no issues. In addition, the ghost folder is 100999, and has no problems.

I could probably spin up another vm and install rootful docker, but was hoping that that wasn’t the case….

thx as always for the help!

7

that’s it. I spun it up on rootful docker, and it went down and came back up fine. It just doesn’t work on rootless docker. The ghost-db-1 container specifically. Any interest in figuring this out? It’s the only container in my stack that doesn’t work on rootless….

Also, what’s odd, is when I look at the logs, it mostly works. It spins up fine, creates the database, is able to write to the mysql folder, the website comes up and i’m able to create a post. the only thing that doesn’t seem to work is bringing down the containers… And once you try, it seems to nuke the whole setup…..

not sure why this is happening.

8

Sorry I have completely missed “rootless docker” and thought you meant you were using sudo instead. Lol

9

does this mean the ghost-docker doesn’t yet completely support rootless? Or is this a problem with the mysql image specifically? I have other mysql databases running rootless and it works fine….

10

Since the ghost-docker repository only uses a standard MySQL image, I would argue that the “issue” is with that.

Shutting down a MySQL database actually involves quite a few things in the background. The MySQL container needs to flush buffers, release locks, close connections gracefully, etc.

While never having encountered that myself, it does sound a bit like this graceful shutdown cannot be performed properly – or cannot complete within the standard 10 seconds that Docker allocates for that.

You could try increasing that:

  db:
    ...  
    stop_grace_period: 60s
    ...
1 Like
11

janis - i went down the same rabbit hole! Oddly, the stop_grace_period: 60s did not work, BUT adding this:

init: true

to the db service in the docker compose did work….. I have no clue why…. I’d have guessed your suggestion should work (i even increased it to 180s) and it didn’t change anything….. The init: true suggestion came from chat-gpt after I struck out with increasing the grace period…. I don’t know why it works (other than it helps with the shutting down process) and I didn’t see a down-side….

So i guess this problem is solved!

I’m not sure I set up external reverse proxy (caddy) completely right…. but things seem to be fully functional right now…. I’m sure i’ll have more questions for you all as I start my journey with ghost (it’s pretty neat!), but in the meantime, do you see anything obviously wrong with the initial setup of the external reverse proxy?

Ok, first the edits I made to make ghost-docker work with an external caddy reverse proxy:

  1. I copied the ghost-docker repository to my computer. Note: I copied it to a directory in my home directory, not to /opt/ghost… but i don’t think this matters?

  2. I filled out the .env per install instructions.

    1. I did change the UPLOAD_LOCATION and MYSQL_DATA_LOCATION to a different directory that is regularly backed up. I used absolute paths, not relative paths.

  3. I commented out the caddy service definition in the compose file.

  4. I added the following to the ghost service definition in the compose file so my caddy reverse proxy (running on the same server) can access ghost.

    1. ports:

      • 127.0.0.1:2368:2368

  5. I copied the caddy config from the Caddyfile in the ghost-docker git into my Caddyfile and made the necessary edits.

  6. I copied the caddy snippets folder over and replaced the activitypub variable in the ActivityPub snippet to point directly to https://ap.ghost.org, since there would be no way that it’d know the value of a variable that is only defined inside of docker…..

again, thank you for all your time on this. I wouldn’t have found the answer without the pointers you were giving me.

Brian

1 Like
12

That’s really smart! Cool solution for this!

What it essentially does is create a little init process inside the container that helps with signal forwarding.

The Caddy setup generally looks fine to me – at least, I cannot see any obvious issues. So, if it works, I guess it works? :smiley:

13

ok, so i was missing a couple things to make it all work: i still needed to edit the TrafficAnalytics caddy snippet to not point to a container-name because caddy was not in the stack but on the server itself, and then i needed to bind the trafficanalytics service to a host port so the reverse proxy could access it.

If you don’t mind, i’d like to do a clean version of this post to let people know how to get ghost-docker working with an external proxy, and the potential gotcha of the mysql image and working with rootless. If you don’t mind, where should I post this? New post here, or other category on the forum? Just reply to this thread and clean it all up (or edit the OP)?

Brian

1 Like
14

I am no moderator here, so don’t take my word for granted :smiley:

Personally, I would not edit the OP, since that’s the major part of the conversation underneath.

My own preference would just be a conclusion at the end of this thread, so it’s all coherent and users in the future (hi) can follow how this all came together :smiley:

1 Like
15

I agree with Jannis, you can summarize the key takeaways/configuration here. Be sure to mark it as the solution so it won’t get lost in the noise :slight_smile:

2 Likes
16

Ok, so they were unrelated, but first part of this post is how to get ghost-docker up and running with an external reverse proxy (caddy in my case). The second part of the post, which is much shorter, is a solution for a potential issue you may encounter with rootless docker, apparmor, and the mysql service not responding to “docker compose down” and getting a “permission denied” error.

Ok, first the edits needed to make ghost-docker work with an external caddy reverse proxy:

  1. Copy the ghost-docker repository to your computer and fill out the .env per install instructions.

  2. Comment out or delete the caddy service definition in the compose file.

  3. In the ghost service definition in the compose file, bind to a host port so your external reverse proxy can access ghost:
    ports:
    -2368:2368

  4. In the traffic-analytics service definition in the compose file, bind to a host port so your external reverse proxy can access the traffic-analytics service:
    ports:
    -3000:3000

  5. copy the config from the Caddyfile in the ghost-docker git into you Caddyfile, replace the variables with the actual domains, and make sure to edit the reverse proxy directives to point to the IP address and port of the ghost service (use the host’s IP address running the ghost service).

  6. Copy the caddy snippets folder over to /etc/caddy/ and replace the $ACTIVITYPUB_TARGET variables in the ActivityPub snippet to point directly to https://ap.ghost.org (this assumes you are using the default activitypub config).

  7. In addition, in the TrafficAnalytics snippet, replace the target of the reverse proxy directive with the IP address and port of the traffic-analytics service (use the host’s IP address running the ghost service).

This assumes firewalls settings and iptables/nftables are set up properly. That should be it! follow the instructions on the ghost website and you should have a running system in a few minutes (assuming no other errors!)

If your reverse-proxy is running directly on the same host as your docker containers (but not a container itself), you can bind to the loopback address in steps 3 and 4 for more security, and add the loopback address to the reverse-proxy directives in steps 5 and 7.

break break

Ok, If after you get things running and everything seems fine, but when you use “docker compose down” to bring your containers down, you get the following:

✘ Container ghost-db-1 Error while Stopping 14.0s
Error response from daemon: cannot stop container: e91d03f0dd72f61fe01ebaddd7eeb68bf18d44277a55cae447d4cebfce9eb20e: permission denied

And if you do get the container stopped using,

sudo systemctl --user restart docker.service

but get the following error using “docker compose up -d”:

✘ Container ghost-db-1 Error 100.8s
dependency failed to start: container ghost-db-1 is unhealthy

likely the issue can be solved by adding the following to the db service definition in the compose file:

init: true

You also have to delete the ghost data directory, related volumes, and do a docker compose down and docker compose up -d (basically, start from scratch).

after some investigation, I’m pretty sure this is occurring because I’m running rootless docker, there is some clean-up that isn’t happening, a kill signal gets sent and apparmor throws a flag on the play. So if you’re getting this error, running rootless docker, and apparmor (e.g. ubunutu), then try the init: true solution out.

Also - thx for everyone on this thread that helped me solve the problem

Brian

1 Like