- I am running the latest Docker image of Ghost
- My url is blairtech.org
- I’ll be honest in that I don’t know what Node or database I have, or if I have either at all. I can’t remember anymore, I’m pretty sure that I only have one container and no external databases.
- I am running this on Debian 10, accessing the site with Firefox. Chrome results in the same issue.
- There is absolutely zero errors in the console. It simply begins outputting startup information after resuming from the crash. Nothing is outputted before that.
- To reproduce, try to upload an image. Anywhere.
I think that’s everything the template asks for.
I’ve already tried following every bit of advice that comes up on Google when you search up this problem, with no success. It is highly likely that this is a “just you” problem. If so, I may end up trying to reinstall everything, but I want to make sure that I backup and migrate everything perfectly. If I did reinstall, I would follow this page here.
Is there a solution that hasn’t already been suggested? Do I have no choice but to reinstall?
EDIT: I just went ahead and recreated the blog using the docker-compose file I linked prior, setting up a proper separate mysql database. It still silently crashes when uploading images. I am officially out of ideas.
I find it very disheartening that nobody has offered a solution or attempted to find one. Is there a better place to go for advice on this matter?
You might want to use the 1 click install from your host (if it’s renowned and offers one) rather than docker. Since there is nodejs, I am guessing it is eating up all your memory.
I have 16 GB of RAM and my server never uses more than a fraction of that. The server itself is physical hardware in my own home, not a VPS from a hosting provider. I built the machine myself. I can promise you that I’m not hurting for any resources.
Can you post your docker config or docker-compose file that you used when you deployed Ghost?
Can you post the log info from your container? (you can find your container with docker ps, and then docker logs <container id> will output the logs from your ghost container).
Well, this is what I followed in the reinstall docker-compose-ghost-quickstart/docker-compose.yml at master · robincher/docker-compose-ghost-quickstart · GitHub. I removed bits related to SSL due to me using a reverse proxy setup with Nginx Proxy Manager that handles SSL externally. That said, I’m not sure if you will encounter the same issues that I have. Oftentimes I find that I’ll have issues that are exclusive to me. Just my kind of luck.
Ok, thanks.
Can you post the logs from your ghost container and from your mysql container? The logs might have some error messages server-side that will be useful.
There will be nothing useful, but here you go. The logs on the original docker container without an external DB when I upload an image:
* today at 9:47:12 AM [2021-10-02 13:47:12] INFO Ghost is running in production...
* today at 9:47:12 AM [2021-10-02 13:47:12] INFO Your site is now available on https://blog.blairtech.org/
* today at 9:47:12 AM [2021-10-02 13:47:12] INFO Ctrl+C to shut down
* today at 9:47:12 AM [2021-10-02 13:47:12] INFO Ghost server started in 0.702s
* today at 9:47:12 AM [2021-10-02 13:47:12] INFO Database is in a ready state.
* today at 9:47:12 AM [2021-10-02 13:47:12] INFO Ghost database ready in 1.24s
* today at 9:47:17 AM [2021-10-02 13:47:17] INFO Ghost booted in 5.442s
* today at 9:47:17 AM [2021-10-02 13:47:17] INFO Adding offloaded job to the queue
* today at 9:47:17 AM [2021-10-02 13:47:17] INFO Scheduling job update-check at 17 44 8 * * *. Next run on: Sun Oct 03 2021 08:44:17 GMT+0000 (Coordinated Universal Time)
As you can see, there are no errors. It just restarts, instantly. The logs for the Compose version are completely identical except that it restarts multiple times instead of once.
Hmm… from my reading of this it doesn’t demonstrate the issue you are experiencing. These logs don’t show a crash and restart, it just shows a single startup (that is, it doesn’t show that Ghost was operating properly and then crashed silently before restarting, it just shows a single start).
Without more data it isn’t possible to troubleshoot.
Can you provide:
- your ghost config (redact any sensitive portions),
- the docker command you are using to start ghost,
- output of
docker ps,
- and any logging changes you made to your docker daemon.
Your docker logs should be showing that Ghost was running before it crashed. If you haven’t already, take a look at all of the log output from each of your containers and see if there are any additional clues there.
I’m telling you that there is no more information in the logs. If I were to repeatedly upload a file, you would see that series of messages repeated for every time I did it. There are no outputted errors.
- Forgive me, but I don’t know what you’re asking here. Do you mean my environmental variables? Production config?
- I am simply automatically starting it with Portainer.
- Here you go: https://note.blairtech.org/?bfdd2097609948cc#9kajMrDErx3LBCDqH9CiYhfSjqXPAtpHQacrxcRqyumq
- I have not made any logging changes to the daemon as far as I know, but I may not understand what you want here.
Forgive me if I don’t fully understand what you’re asking for, I’m not an expert.
Hmm, by “there is no more information in the logs” are you saying that A). that is the entire log output, or are you saying B). that is the only portion of the log output you think is relevant?
If A, then the issue is not so much that docker is failing silently as it is that you don’t have any logs to review (this may sound like a hair-splitting distinction, but it reveals a different problem: in the first case you don’t get any log output, in the second case there is log output but it isn’t retained after a crash. The solution for the second case is figuring out how to keep your logs after a crash. The solution for the first case is doing lots of painful troubleshooting without an log/error data.)
If B, then I don’t think I can help you any more. If you have an issue you don’t understand but are only willing to share the data you do already understand and this is relevant, I will not be able to shed any more light on this issue than you already understand yourself.
- Are you using a ‘stack’ within portainer? If so, then it will be the docker-compose data you put in that stack. If you aren’t using a stack and are just deploying Ghost as one of the pre-configured applications that portainer shows, I strongly recommend using a stack/docker-compose for a public-facing production system so you have better control over your configuration.
- Ah, that makes sense. Are you using a stack? If so, please see item 1 above.
- Lots of containers! I know you said your box has 16GB RAM and shouldn’t be running out of RAM, but can you run
free or top to confirm? That is enough containers that I’d consider running out of RAM to be a viable concern.
- If you haven’t made changes, that’s just fine. The big issue appears to be that your logs aren’t persisting after a crash, and I’d focus on finding out why that is the case. Maybe set a script to copy all of your logs to a new file in your mounted volume so it persists?
As a side note of unsolicited advice that you can take or leave: if you are using some of those containers for personal/private use (e.g. nextcloud), I recommend separating out your hardware and segmenting your network to permit a DMZ for hosting your public-facing assets/containers. You could migrate all of your containers stacks/docker-compose files and volumes, install Proxmox PVE on your box, and then separate out your internal/private assets and containers from your external/public assets and containers into two separate VMs that have different IP addresses and are placed on different segments of your network. Just my 2 cents. Take it or leave it.
It’s A, there is zero more information to give you. Not data I think would be irrelevant, I mean there is nothing further. There is the standard startup information when you load the container, and then when it crashes it just shows the same information again. I am not holding back anything.
- I am using a stack for the redeployment while keeping the original temporarily. I have not migrated fully over to it in that I haven’t shut down the old one until I’m sure that it’s actually the fix. The output is identical regardless beyond it restarting several times before giving up trying to upload as opposed to only once in the single container deployment.
- See 1.
- https://pastebin.blairtech.org/?361247206b3c8420#HHbEBJqaytMhGboK3aKs14nxnGLpW6abkSpDpieWUC6Q
- You know what, I have a solution. Here. This is a realtime feed of my logs that even lets you search. This is the new stack: https://logs.blairtech.org/container/3962d56465d7 and this is the old single container: https://logs.blairtech.org/container/f4cf8dc3972d Now you can see that there really isn’t anything I’m leaving out.
Your advice is very much appreciated and I actually do intend on doing all of that, but the hardware as it stands currently would not be up to the task of using Proxmox to its full potential. When I have a custom built server as opposed to a makeshift one, I will definitely be doing that.
- So you’ve got ~.75GB free, which should be ample. Thanks for sharing that data.
- Dozzle is neato.
Hmm… so, this is the salient portion of the logs from your stack that has my interest:
last Saturday at 12:36:05 PM [2021-10-02 16:36:05] INFO Ghost is running in production...
last Saturday at 12:36:05 PM [2021-10-02 16:36:05] INFO Your site is now available on http://localhost:2368/
last Saturday at 12:36:05 PM [2021-10-02 16:36:05] INFO Ctrl+C to shut down
last Saturday at 12:36:05 PM [2021-10-02 16:36:05] INFO Ghost server started in 1.185s
last Saturday at 12:36:06 PM [2021-10-02 16:36:06] INFO Database is in a ready state.
last Saturday at 12:36:06 PM [2021-10-02 16:36:06] INFO Ghost database ready in 2.064s
last Saturday at 12:36:17 PM [2021-10-02 16:36:17] INFO Ghost booted in 12.423s
last Saturday at 12:36:17 PM [2021-10-02 16:36:17] INFO Adding offloaded job to the queue
last Saturday at 12:36:17 PM [2021-10-02 16:36:17] INFO Scheduling job update-check at 51 14 21 * * *. Next run on: Sat Oct 02 2021 21:14:51 GMT+0000 (Coordinated Universal Time)
**==last Saturday at 12:36:26 PM [2021-10-02 16:36:26] WARN Ghost is shutting down==**
**==last Saturday at 12:36:26 PM [2021-10-02 16:36:26] WARN Ghost has shut down==**
**==last Saturday at 12:36:26 PM [2021-10-02 16:36:26] WARN Your site is now offline==**
**==last Saturday at 12:36:26 PM [2021-10-02 16:36:26] WARN Ghost was running for a few seconds==**
last Saturday at 12:38:09 PM [2021-10-02 16:38:09] INFO Ghost is running in production...
last Saturday at 12:38:09 PM [2021-10-02 16:38:09] INFO Your site is now available on http://localhost:2368/
last Saturday at 12:38:09 PM [2021-10-02 16:38:09] INFO Ctrl+C to shut down
last Saturday at 12:38:10 PM [2021-10-02 16:38:09] INFO Ghost server started in 13.482s
last Saturday at 12:38:16 PM [2021-10-02 16:38:16] INFO Database is in a ready state.
last Saturday at 12:38:16 PM [2021-10-02 16:38:16] INFO Ghost database ready in 20.455s
last Saturday at 12:38:54 PM [2021-10-02 16:38:54] INFO Ghost booted in 57.948s
last Saturday at 12:38:54 PM [2021-10-02 16:38:54] INFO Adding offloaded job to the queue
last Saturday at 12:38:54 PM [2021-10-02 16:38:54] INFO Scheduling job update-check at 16 20 15 * * *. Next run on: Sun Oct 03 2021 15:20:16 GMT+0000 (Coordinated Universal Time)
I assume that those WARN errors occurred when you uploaded an image? Can you try uploading an teeny tiny image (like a 1x1 tracking pixel kind of thing) and see what happens? Can you try uploading a very large image (like something from natgeo or a high-res NASA photo or something) and see what happens?
In your stack I’m also eyeing those no space left on device, write errors. You are getting those on both of your Ghost deployments. You also have some [ERROR] InnoDB: Could not set the file size of './ibtmp1'. Probably out of disk <mark>space</mark> alerts on your Matomo container.
If your image upload issue is actually related to some space issue, then you should be able to upload a very small file, while the large upload should definitely fail.
I feel your pain, I was fooling around with nextcloud on a little 4GB optiplex for a while before I bought a server! You’re likely already aware of labgopher, but I’ll bring it up in case you were not.
I should have mentioned the space issues. They’re a red herring to the problem, unfortunately. There was indeed a point a day or two ago where I ran out of space due to some logging tool I was using somehow taking up every last byte of drive space, but I solved the problem by getting rid of all of it. I currently have about 15 GB of root space left.
I also did try what you mentioned about uploading different size images. It has the same issue even with a 32x32 3.1 KB image.
I was actually not aware of Labgopher. I’ll give a look when I have some time.
That is interesting that it had the same problem even with a very small image. That leads me to believe that the specific issue is the image upload. Not a processing/space/RAM issue. What happens when you embed an image using the Unsplash integration?
Specifically, can you (from your ghost instance) pull an image down without error? If so, then it is something with the file upload from your box. If you get the same issue, then it seems to be something with how images are being handled/embedded.
I’ve found that JSON File logging driver | Docker Documentation is critical configuration step. Otherwise logs will often balloon to something obscenely large and eat all drive space.
Yes, that is exactly how I have been (mostly) getting around it. I will upload an image to my nextcloud container, get a share link, then use that link to embed a picture into an article that way. Unsplash images also work just fine. It is only when I specifically upload an image into the Ghost container that I get problems.
Can you post your docker-compose file (the text you put into your stack in portainer)?
All I did was remove bits related to certificates since I use a reverse proxy that handles that, change the ports on the outside, and of course changed the passwords set for the database. docker-compose-ghost-quickstart/docker-compose.yml at master · robincher/docker-compose-ghost-quickstart · GitHub
Ok, thanks, can you post your $PWD/ghost/volumes/config.production.json file?