I used to have all my images on GitHub pages to externalise some of the bandwidth and complexity, and give more control over the SEO of the image URLs. But Ghost doesn’t allow setting post images to a URL anymore, so now I just use Ghost, which is less hassle.
I tried to install ghost-storage-cloudinary but after restarting my app I get this error:
chown: cannot dereference ‘/var/lib/ghost/content/adapters/storage/ghost-storage-cloudinary/node_modules/.bin/image-size’: No such file or directory
and my app crashed and needs volume restore from snapshot. Maybe this is because ghost is in docker image, I’m not sure. I follow all steps but have this error.
In my site I give author privileges to my paid members to post content, and I want images to be stored anywhere else, but not sure how to do it.
@curiositry I have some issues with loading time of my site. Sometimes is really slow. If I restart my apps It’s faster, but after some time slows down again. Do you know why is that and how to fix it?
Does is fix when you redeploy the MySQL server? If so, try scaling to 512MB RAM on your MySQL instance and let me know if that fixes it.
I sometimes noticed intermittent performance degradation on my DB. I assumed that it was related either to higher load days, or a bug somewhere in my software stack or configuration. It usually was fine for ~5-7 days after the last deploy. It seemed like it was maybe correlated with intensive activity (saving a post a over and over again, creating large numbers of bookmark cards in a short time) in the admin panel. But it never became much of an issue because a new version of Ghost is released so often I was redeploying quite a bit anyway. (And when the instance actually runs out of memory, it restarts and sends you an alert.)
Hello! Your “ekniga-mysql” application hosted on Fly.io crashed because it ran out of memory. Adding more RAM to your application might help!
Here’s the raw log message:
Out of memory: Killed process 528 (mysqld) total-vm:1066484kB, anon-rss:196536kB, file-rss:0kB, shmem-rss:0kB, UID:999 pgtables:824kB oom_score_adj:0
It’s working now, I guess It just restarts itself. I publish a lot of content right now, so I guess this is the problem. If slows down again, I’ll try to redeploy database app. Do I need to keep backup before deployment?
Is that with 256MB RAM or 512MB? Is it the default config from my tutorial? (I think I’ve had my Ghost app OOM and restart once or twice, and I’ve had my MySQL degrade, but my MySQL instance hasn’t OOMed and restarted. Odd.)
Thanks but I already migrate my site from fly.io to oracle cloud since I can’t find a way to install cloudinary storage adapter but I need more space for images. In oracle cloud I use aapanel and still deploy my ghost blog as docker image but at least I have much more storage on free tier and automatic backups of my site to google drive with aapanel. Also in oracle cloud I don’t need to create two separate instances for the site and database.
Good point @paulrudy. It looks like I missed that part, and never noticed because that’s the default production DB, so that line isn’t technically necessary.
Nice. FYI, I reached out to Fly with some questions, and they told me that there’s a managed MySQL in the works. Info with link to private beta signup here. As I understand it this will not only be managed, but also allow for Tigris automatic backups, database clusters with fallbacks, and more (like their current Postgres, but also managed?)
Until that option is available, they warned against having a single MySQL instance in production:
I wouldn’t recommend deploying your own MySQL instance on Fly for production use cases unless you’ve also implemented a replication system that lets you spin up additional Machines that automatically sync with each other and coordinate primary/replica roles. Generally speaking, you can backup data to Tigris by running a script that dumps data from MySQL and uploads the resulting dump file to Tigris.
But I honestly have no idea how to accomplish that at this point. Waiting for the managed option.
Thanks @paulrudy. I have signed up for the waitlist.
When Fly released the new app platform, a little banner appeared warning against running an app on a single machine.
Redundancy would certainly be advisable for anything critical; running on a single machine risks data loss if Fly makes a mistake.
That said, I’ve been running my MySQL on a single machine for some time without issue. I haven’t bothered to scale it up to two machines simply because the hassle (and cost) isn’t worth it, and I keep my infrequently-updated blog backed up well enough for my use case using rsync.
As far as I can tell, the content folder includes everything relevant (including JSON exports of the posts, and members) — though I’m sure it’s not 100% fresh, and don’t take my word on it being complete — so if Fly accidentally deletes my database it would be a pain, but not a disaster.
(I did once delete my production database about ten years ago, when I was running Postgres on Heroku. It was squarely my fault, and I never got it back.)
Let me know what you find in the way to replicate the MySQL, or set up automated backups, though. That would be preferable!
An interim step would be to set up an additional cron job that dumps the MySQL DB; that alone would probably be enough for a lot of people.
Are you saying that Ghost creates periodic JSON exports of everything of value that would be in the database, to the content folder, so that an rsync backup is good enough without a database dump in case of catastrophe? (New to Ghost, just learning about how it all works)
Officially, no. The only official way to backup is with the ghost backup command from the Ghost CLI (which isn’t available on Fly, because hosting on Fly uses the unofficial Ghost Docker image.)
Unofficially, I have seen JSON exports in my rsync backups that were fresh enough for me. I also have regular manual exports of members and posts, as well as fly snapshots. So the only thing I’m strictly relying on my rsync backups for are images/files/themes etc.
But there’s a lot of other stuff in there, and you could take a look in the Ghost source code to see how often it gets exported automatically, and decide if its complete and fresh enough for you…
Hmm, looking at it, I’m noticing something. I was seeing up-to-the-minute JSON files. But I’m also obsessive about exporting from the admin panel – usually at the same time as I’m doing the rsync backup.
My hunch is that when you export from the admin panel, it also gets written to the data folder?
I could be way off, though. In either case, I wouldn’t rely on it :)
Thanks for these insights. If you’re working on customizing a theme, how do you manage that in relation to your Fly instance? i.e. how do you push the new/updated theme to the deployed instance? Based on my limited knowledge, I imagine you could have a Dockerfile that copies the content folder on build, but maybe there’s another way.
I was planning to use GitHub actions, but a lot of my code is on SourceHut and GitLab, so I wrote a bash script that uploads and activates themes via the Ghost API:
Does it still work in 2024?
I was trying this code, and it was somewhat successful to deploy, for the first minutes.
After a while, the ghost app get automatically "suspended " (while the database was still running without shutting down.)
I pasted the log here so let me know if somebody has a clue what I should have done to solve this problem.
2024-09-09T14:11:38Z app[e286700dc75e38] ams [info][2024-09-09 14:11:38] INFO Ghost booted in 6.211s
2024-09-09T14:11:38Z app[e286700dc75e38] ams [info][2024-09-09 14:11:38] INFO Adding offloaded job to the queue
2024-09-09T14:11:38Z app[e286700dc75e38] ams [info][2024-09-09 14:11:38] INFO Scheduling job update-check at 42 42 15 * * *. Next run on: Mon Sep 09 2024 15:42:42 GMT+0000 (Coordinated Universal Time)
2024-09-09T14:11:38Z app[e286700dc75e38] ams [info][2024-09-09 14:11:38] INFO Running milestone emails job on Mon Sep 09 2024 14:11:38 GMT+0000 (Coordinated Universal Time)
2024-09-09T14:11:39Z health[e286700dc75e38] ams [error]Health check on port 2368 has failed. Your app is not responding properly. Services exposed on ports [80, 443] will have intermittent failures until the health check passes.
2024-09-09T14:11:42Z app[e286700dc75e38] ams [info][2024-09-09 14:11:42] INFO "GET /favicon.ico" 200 5ms
2024-09-09T14:11:42Z app[e286700dc75e38] ams [info][2024-09-09 14:11:42] INFO "GET /" 200 497ms
2024-09-09T14:11:43Z app[e286700dc75e38] ams [info][2024-09-09 14:11:43] INFO "GET /assets/js/global.js?v=0ad62bcf81" 200 8ms
2024-09-09T14:11:43Z app[e286700dc75e38] ams [info][2024-09-09 14:11:43] INFO "GET /assets/css/style.css?v=0ad62bcf81" 200 18ms
2024-09-09T14:11:43Z app[e286700dc75e38] ams [info][2024-09-09 14:11:43] INFO "GET /assets/js/index.js?v=0ad62bcf81" 200 19ms
2024-09-09T14:11:43Z app[e286700dc75e38] ams [info][2024-09-09 14:11:43] INFO "GET /public/cards.min.css?v=0ad62bcf81" 200 23ms
2024-09-09T14:11:43Z app[e286700dc75e38] ams [info][2024-09-09 14:11:43] INFO "GET /public/member-attribution.min.js?v=0ad62bcf81" 200 24ms
2024-09-09T14:11:43Z app[e286700dc75e38] ams [info][2024-09-09 14:11:43] INFO "GET /public/cards.min.js?v=0ad62bcf81" 200 5ms
2024-09-09T14:11:43Z app[e286700dc75e38] ams [info][2024-09-09 14:11:43] INFO "GET /members/api/member/" 204 33ms
2024-09-09T14:11:44Z app[e286700dc75e38] ams [info][2024-09-09 14:11:44] INFO "GET /ghost/api/content/settings/?key=9806ef18016873948e077b47ca&limit=all" 304 1160ms
2024-09-09T14:11:44Z app[e286700dc75e38] ams [info][2024-09-09 14:11:44] INFO "GET /ghost/api/content/tiers/?key=9806ef18016873948e077b47ca&limit=all&include=monthly_price,yearly_price,benefits" 304 27ms
2024-09-09T14:11:44Z app[e286700dc75e38] ams [info][2024-09-09 14:11:44] INFO "GET /ghost/api/content/newsletters/?key=9806ef18016873948e077b47ca&limit=all" 304 31ms
2024-09-09T14:12:08Z app[e286700dc75e38] ams [info][2024-09-09 14:12:08] INFO "GET /" 301 4ms
2024-09-09T14:12:38Z app[e286700dc75e38] ams [info][2024-09-09 14:12:38] INFO "GET /" 301 1ms
2024-09-09T14:13:08Z app[e286700dc75e38] ams [info][2024-09-09 14:13:08] INFO "GET /" 301 1ms
2024-09-09T14:13:38Z app[e286700dc75e38] ams [info][2024-09-09 14:13:38] INFO "GET /" 301 1ms
2024-09-09T14:13:42Z app[e286700dc75e38] ams [info][2024-09-09 14:13:42] INFO [Recommendations] Updating recommendations metadata
2024-09-09T14:13:59Z proxy[e286700dc75e38] fra [error][PR03] could not find a good candidate within 91 attempts at load balancing. last error: [PR01] no known healthy instances found for route tcp/443. (hint: is your app shut down? is there an ongoing deployment with a volume or are you using the 'immediate' strategy? have your app's instances all reached their hard limit?)
2024-09-09T14:14:08Z app[e286700dc75e38] ams [info][2024-09-09 14:14:08] INFO "GET /" 301 1ms
FYI and off-topic - the app region was Amsterdam because my fly.io instance constantly gives this error below while deploying. so i thought this service really wanted to build an app in ams region (despite I specified “bos” everywhere.) My guess is that it’s GDPR or EU data thing associated with my account, but I personally don’t know why
vol_450pyj1wy5g96wqr" is in the wrong region ("bos" != "ams") (Request ID: 01J7B6GWB0CVWWXNC5XJV8XXQG-fra) (Trace ID: 6e7c6dfd4886ad2663ca63211b6d09fd)
Sounds like fly.io introduced automatic starting/stopping of instances in 2023, which probably takes down your Ghost instance when there’s no activity:
i was digging that error log which says it’s a port problem, but there’s nothing i could do. I even made the ghost app instance memory to 512MB. But also didn’t change anything.