Hey guys, I noticed that there is no post about the current outage with MagicPages, so I figured I’d share a post about it here.
I contacted Jannis and he got back to me quick and instructed me to watch https://health.magicpages.co/
Currently:
Earlier today, two server nodes went offline. I have restored them in due time, however, an hour later, the issue has popped back up. Right now, it looks like Hetzner, the infrastructure provider, does not have enough servers available.
I am shifting to different server nodes in the meantime, hoping that this will eliviate pressure. Unfortunately, a small set of websites is affected by this and is currently offline.
Update 15:11 UTC: I have provisioned the new servers, however, these are in a different architecture (AMD64) compared to the old ones (ARM). I am rebuilding Docker images at the moment to get things up and running again.
Update 17:41 UTC: Getting an AMD64 image ready is turning out to be a lot more difficult than anticipated, since the Kubernetes cluster is running on two architectures now (some leftover ARM nodes, and new AMD64 nodes). Each image build takes about 30 minutes, and I am a few tries in. Will keep things updated here.
I’m sure many of you were worried or confused as I was. But @Jannis is on the case!