Today, like the past few days, we have had some downtime. Apparently some script kids are enjoying themselves by targeting our server (and others). Sorry for the inconvenience.
Most of these ‘attacks’ are targeted at the database, but some are more ddos-like and can be mitigated by using a CDN. Some other Lemmy servers are using Cloudflare, so we know that works. Therefore we have chosen Cloudflare as CDN / DDOS protection platform for now. We will look into other options, but we needed something to be implemented asap.
For the other attacks, we are using them to investigate and implement measures like rate limiting etc.
I’m not one of these 2 arguing. But in general the app servers don’t do caching or state handling.
You cache things in a third external cache such as redis or memcached. So if a user connects to app server 1 and then to app server 2 they will both grab cachee info from redis. No extra db calls required. This has been the basic way of doing things even with old school WordPress sites forever. You also store session cookies in there or in the db.
And even if you weren’t caching externally like this, databases use up a lot of memory to cache tons of data. So even if the same query hits the db the second hit would probably still be hot in memory and return super fast. It’s not double the load. At least with postgres this is the case and it’s what Lemmy uses.
Definitely this. I use PostgreSQL (which Lemmy uses on the backend) for an enterprise-grade system that has anywhere from 700-1k users at any given point in time, and it also takes in several million messages from external systems throughout the day. PostgreSQL is excellent at caching data in memory. I’ve got the code for that system up in another window while I write this.
At this point in time, it doesn’t look like Lemmy is using any form of an L2 cache like Redis or Memched. The only single point of failure (that’s not horizontally scalable) looks like the pic-rs server that Lemmy is using for image hosting. If anything, that could easily be swapped over to use something S3 compatible and easily hosted using something like Minio locally, or even directly off of B2 or Linode cloud storage (doesn’t charge for requests).