Today’s maintenance was only supposed to be a storage upgrade, and supposedly take less than an hour of downtime. kbin.earth’s storage was running low, and I knew between the ever growing database and the massive docker image/volume/etc. cache, we would need an increase in storage capacity. Today, it ended up being down for about 8 hours!
In the past, I would just upgrade to Hetzner’s next best VPS offering, as it would provide at least double the amount of storage, and even provide a boost in RAM and CPU cores. I knew that wouldn’t be an option though, as Hetzner’s prices have skyrocketed, and every single one of it’s VPSs that are advertised as “cost savings” are marked as unavailable. If I wanted to stay on Hetzner, I’d have to pay triple the amount of the current monthly cost in order to upgrade my server specs.
The first alternative I tried was to move the Postgres database on to a Hetzner volume. Hetzner lets you create a volume of an arbitrary size, which is then attached and mounted to your VPS. It sounded like the perfect solution! I could create a 130GB volume (for an extra $8.19/month) just for the database, and then use the existing SSD (which is 160GB) for the OS, Docker, and the remaining Mbin data. I spent about 2 hours getting that set up, and upgrading to Postgres 18 while I was at it.
When I started the Docker containers back up, it took about 20 seconds to load any page at all. After a few minutes, the whole server had frozen up as well. It turns out, Hetzner volumes just don’t work for a high load database 🤦. Somewhere in all the mess, it seemed like something in the operating system (or potentially the hardware) had become corrupt, as even after I moved the database back to the SSD, the server was freezing within just a few minutes of the server boot up.
I knew I’d have to migrate away from Hetzner anyway; their currently available offerings (that match the specs kbin.earth requires) were well over $100/month, and obviously a Hetzner volume is a no-go. So, I went ahead and ordered one of OVHcloud’s ECO dedicated servers. This new server has 32 GB of ECC RAM, 512 GB of SSD storage (with RAID), and an AMD Ryzen 5 3600X (much heftier than our previous server’s ARM processor)!
This new hardware, unfortunately, still comes with a 45% increase in the monthly cost, but that was going to happen sooner or later with all the increases in chip prices. I still have some more setup work I need to do with the server tomorrow, but it is operational, and the hardest work is done!
Anyway, I thought I’d give you guys a bit of an explanation as to why kbin.earth was down so long today; I hope you found the story interesting. As always, let me know if you encounter any issues with kbin.earth, or even if you have any questions regarding the migration. I’ll respond to any comments in the morning.
Thank for all the time you spend on this instance.
Regarding the increased costs: do you have an overview of the costs and how much is covered by donations?
Alright, I made a post about it here: https://kbin.earth/m/kbinEarth/t/3390220/kbin-earth-and-Interstellar-expense-donation-breakdown
Sure, I’ll see if I can write a breakdown today. I am also curious how much is covered by donations now.
Uff, that a simple storage attachment breaks down a server is more than unfortunate. Hope that at least the setup and migration to a new server went without a hassle.
Thanks for sharing and for making kbin.earth better for all of us!
Lot of these low budget hosting companies (not Hetzner) get hovered up nowadays, hosting costs upped betting on that people to scared (too much hastle to migrate to another host. Maybe this is also one of the factors hosting is getting more expensive…apart from the ai infestation.
Thanks for taking the effort keeping kbin.earth going, jwr1.
Thank you so much for this space and the work and expense to provide it. 🫂






