Today’s maintenance was only supposed to be a storage upgrade, and supposedly take less than an hour of downtime. kbin.earth’s storage was running low, and I knew between the ever growing database and the massive docker image/volume/etc. cache, we would need an increase in storage capacity. Today, it ended up being down for about 8 hours!

In the past, I would just upgrade to Hetzner’s next best VPS offering, as it would provide at least double the amount of storage, and even provide a boost in RAM and CPU cores. I knew that wouldn’t be an option though, as Hetzner’s prices have skyrocketed, and every single one of it’s VPSs that are advertised as “cost savings” are marked as unavailable. If I wanted to stay on Hetzner, I’d have to pay triple the amount of the current monthly cost in order to upgrade my server specs.

The first alternative I tried was to move the Postgres database on to a Hetzner volume. Hetzner lets you create a volume of an arbitrary size, which is then attached and mounted to your VPS. It sounded like the perfect solution! I could create a 130GB volume (for an extra $8.19/month) just for the database, and then use the existing SSD (which is 160GB) for the OS, Docker, and the remaining Mbin data. I spent about 2 hours getting that set up, and upgrading to Postgres 18 while I was at it.

When I started the Docker containers back up, it took about 20 seconds to load any page at all. After a few minutes, the whole server had frozen up as well. It turns out, Hetzner volumes just don’t work for a high load database 🤦. Somewhere in all the mess, it seemed like something in the operating system (or potentially the hardware) had become corrupt, as even after I moved the database back to the SSD, the server was freezing within just a few minutes of the server boot up.

I knew I’d have to migrate away from Hetzner anyway; their currently available offerings (that match the specs kbin.earth requires) were well over $100/month, and obviously a Hetzner volume is a no-go. So, I went ahead and ordered one of OVHcloud’s ECO dedicated servers. This new server has 32 GB of ECC RAM, 512 GB of SSD storage (with RAID), and an AMD Ryzen 5 3600X (much heftier than our previous server’s ARM processor)!

This new hardware, unfortunately, still comes with a 45% increase in the monthly cost, but that was going to happen sooner or later with all the increases in chip prices. I still have some more setup work I need to do with the server tomorrow, but it is operational, and the hardest work is done!

Anyway, I thought I’d give you guys a bit of an explanation as to why kbin.earth was down so long today; I hope you found the story interesting. As always, let me know if you encounter any issues with kbin.earth, or even if you have any questions regarding the migration. I’ll respond to any comments in the morning.