Do not ever push to production without testing things first. I went and moved us to the beta branch because this commit caught my eye and I wanted us to have the emoji picker fixed: https://github.com/LemmyNet/lemmy-ui/commit/ae4c37ed4450b194719859413983d6ec651e9609

The beta branch was on dockerhub so I thought that it was at least minimally tested. I was sorely mistaken. Lemmy (the backend) itself would load, but lemmy-ui (the actual website that renders everything) would keep crashing when attempting to load. I couldn’t roll back because the database was migrated to this beta version and it couldn’t migrate back to the old version when attempting to launch said old version.

I had no choice but to restore from backup. We’ve lost a whole day’s worth of posts (anything after 3AM CST.) I’m really really sorry… blobcatsadpleading

I was just so excited to be able to unveil this, I didn’t take my time with actually testing it.

  • nickwitha_k (he/him)
    3•9 months ago

    Yeah. Makes sense - best to have something that you absolutely know works. Having the dedi will be really nice - having control of the hypervisor should let you avoid a lot of issues and make testing new updates easier (clone prod, update the clone, test on the clone, swap LB backend to point to the clone and drain old backend, hold old prod VM for a bit to make rollback quick, if needed).