Hi All,
Tldr:
- We are monitoring publically available information (posts, comments and DMs).
- Monitoring is turned off as soon as the admin team are satisfied the user is legitimate.
- will only temporary remove, no bans or deletion without a human being involved in the decision.
- No user data is sent off to 3rd party tools, all processing is done on the instance server. No LLMs are involved.
Due to some ongoing issues with harassment campaigns, we’ve had to setup a rudimentary monitoring system for all new users.
- When a user’s signup is accepted, they will be automatically enrolled into the monitoring system. The admins team may also add accounts manually if they have been given a strike.
- The system will monitor all posts, comments and DMs sent by new users, and bring them to the attention of the admin team if it appears suspicious. In egregious cases, it will auto-remove posts and comments if required, but a human admin will always review and reverse any false positives as soon as required.
- Once we have validated that the user is not a harasser, they will be removed from the system.
We don’t want to go into too much detail on how it all works to prevent bad actors from bypassing it, but we can say that all the processing is being done locally on the instance server. For most of you, this wont have any impact, but some of you have been impacted by the systems false positives. It is also a good time to point out that DM messages are not private, and should not be used for anything that requires strong privacy.
There will likely be teething problems, but we are actively working on improving the bot to minimize impact and we are always open to feedback.
For what it’s worth, some other fediverse platforms have built in functionality like this to catch spam accounts. It’s a big problem on Pixelfed, and so there’s an admin feature that runs locally to try and catch that. We check for false positives about once a day, and it’s supposed to get better over time at catching harmful content.
I see the value in keeping the exact implementation a secret so that hostile people can’t game the system. Maybe you can automate a way to release a regular report on which accounts were flagged? That might resolve the transparency concerns people have.
Thanks for continuing to put in the volunteer hours to keep this little corner of the internet running and keeping it from being used for harm where possible.
I get that in the world we live in, many people are going to be primed to react harshly to “monitoring” wrapped in language about how it will help us. But considering the source, the public context, and the motivations involved, nah.
I’m mostly ok with it, paradox of tolerance and all that, plus that’s just a common moderation and since I’m not using DMs, I don’t care about that either, but seeing the reaction of others - have you cosidere not scanning DMs in the first iteration at least? Even if DMs are public thanks to how I presume activitypub works, it’s not something you can easily get to or understand as an average user, and reading the sentece “we will scan your DMs” will upset a lot of people, regardless of context.
From the PR standpoint, I don’t think it’s worth it, and it’d be better to just leave it on reports.
You will get headlines “Programmers.dev scans your DMs”, and people will not care or know how do they actually work on Fedi.
From the PR standpoint, I don’t think it’s worth it, and it’d be better to just leave it on reports.
The problem is that we can’t rely on reports of DMs since lemmy doesn’t federate them to us.
E.g.
-> troll@programming.dev makes a new account and sends harassment to victim@lemmy.instance
-> victim@lemmy.instance reports the DMs from troll@programming.dev
-> We, the admins of programming.dev, do not see this report because lemmy does not federate the DM report and troll@programming.dev can continue harassing others because we never find out about it.This isn’t just a theoretical, it happened just last month that one of our users (1 day old account) sent rape and death threats (which were reported), and we found out about it by pure chance when talking to admins from the other instance.
And just to clarify, the tool only automatically monitors new accounts, i.e. accounts that are being registered today. If you account is more than a few weeks old, the tool doesn’t monitor any of your activity.
I see, that sucks. Is it being discussed/adressed on the protocol level? This sounds like something that should be adressed in general, federation of reports, because it is a serious issue.
I can imagine a solution in upstream Lemmy repo, where report button also sends the report to the lemmy instance of the account instance outside of ActivityPub.
Yeah, not federating reports is a bug-level missing feature.
But having some kind of probation period also seems like an obvious feature to me (that should be optional, not mandatory). It was not uncommon in the past (on BBSes and old forums) to have new members’ posts require approval before anyone could see them. I suspect it partially went away because big platforms needed to “growth hack” and anything that slows down the influx of new users is bad.
It’s an open issue on github from 2024, it doesn’t seem to be a priority. This tool allows us to react faster than reports though, and hopefully remove some problematic content before it’s seen by others.
I’ve bumped the issue, since it already had an PR ready for review, and there seem to be some activity going on, so hopefully it will be merged soon.
It sounds like a really important thing to me.
Best of luck! I always see your work when a mass commenter gets sent loose, those get shot real quick. Is there a possibility of having victim-side measures? Like a DM whitelist, for example.
I do think having options like letting people disable dms, letting people choose to need to approve dms to get notifications about them and then having a whitelist makes sense. I can make tickets for those in the lemmy repo later if they don’t exist already
It would be nice to have a link to some list of expected etiquette. For instance, is it rude to DM people who don’t know you? Is there an acceptable way to go about this? Are there guidelines for what deserves a downvote? Some “Welcome to our server” post to get the lay of the land on consensus proper behavior…
There is a code of conduct here:
https://legal.programming.dev/docs/code-of-conduct/
If you’re within those guidelines, you’re probably fine.
@weskersucks@lemmy.world did you see this?
I think I’m mostly ignorant here. I of course know that abuse on these platforms is always an issue but I didn’t know we were having any particularly bad instances of it today. Can you point me to a reference so I can get up to date?
Fortunately, most of the visible stuff has been caught and purged quickly, so you may have just missed it.
The DM abuse is basically invisible except to the targets.
Thank you for putting in the effort to help curb this problem. I know it’s unpopular but automated moderation is unfortunately needed on Lemmy and in the fediverse or people will abuse the system.
deleted by creator
Do explain…
deleted by creator
We’ll take the fact that you haven’t noticed much spam as a compliment.
The most recent spam that we have seen is harassment/doxxing campaigns that target specific users. It’s not something the average user would notice as the harassment is often DMs or pings on random posts. That, and a ban evader with a recognisable writing style.
As you pointed out, there is no monetary interest on our end. We are just volunteers looking to fix the blindpots in the moderation tools. We have no interest in needlessly censoring speech, and if you ever feel we moderate the instance too harshly, we are open to taking feedback. You can always check the modlog to see our activity. The guidelines for our moderation can be seen here.
You just joined in February so maybe you are taking it personally? In either case, let me share that I have been on this instance for few years and I trust the admins motives as they expressed.
Your claim is very much and accusation without evidence. There’s no such track record for the humans involved. On the contrary, they are trying to build a community.
I rather not argue what is good or bad censorship. I kind of agree with you.
How about giving the admins the benefit of the doubt?
Time will tell if they are censoring a certain voice, and which one that is. Only then it make sense to judge their actions…
deleted by creator
what loaded gun? admins were already dealing with spammers, this just allows them to do it faster. lemmy already has a built in profanity filter and yet you’re using it just fine.
We got him, admins! This one right here!
Seriously though, if you want total freedom, no verification and no monitoring, setup your own instance. Tell is how it works out for you.
Or better yet, go back to Truth Social or 4chan
deleted by creator
but some of you have been impacted by the systems false positives.
Found the Palentir programmers!
If you’re open to feedback: this is truly horrifying to see entering lemmy on a day that isn’t April 1st.
ETA: this comment was directed to the OP, and the current version is a huge improvement over what was written then, hence the flood of downvotes on this now. The post is also not having an Edit: at the bottom to make clear such a significant edit was made.
We don’t want to do this kind of thing, we are being forced to, because bad actors keep using our server to send harassment.
If you know a better way to handle that, we are all ears
If your OP and your response to me could literally be word for word what Stephen Miller says about the need for current US immigration policy, you bet there has to be a better way.
That’s probably more than a quick back and forth but would include some questions like:
- is this just your instance or does it happen wider?
- have other instances had to deal with similar and how did they resolve it?
- how are is this being evaluated? OP seems to not really suggest transparency?
- is there anything on the instance or its communities that is causing a disproportionate amount of account abuse to come to your instance?
- is there merit to simply suspending new membership applications for a month before doing anything privacy invading.
Also: you do what you want on your instance but as a new user I hope this is very clearly advertised to new users signing up, as this is very yuck and privacy invading, not to mention tone deaf for the world we live in.
Then again I’m probably not in your target audience.
- We only monitor new users on our instance, the tool is written by the admin team and no personal data is sent out of the server. The tool operates on the server communicating directly with the lemmy database on the server.
- Other instances have their methods I’m sure, we haven’t cooperated with any other instances with this tool. I’m sure there’s a spectrum of 100% pure manual moderation to a highly automated process.
- The tool scans for typical patterns we see in the trolls. Some of the patterns I’m sure you can guess, such as using slurs or uploading images. The tool temporarily removes the content and pings the admin team so that we can manually verify the content and restore it.
- No it’s a problem across the fediverse, it seems to be people being banned from various instances that creates new accounts to evade bans. We have seen harassment campaigns where the user creates new accounts on multiple instances and attacks the victim via comments and DMs before moving on to the next instance to repeat the harassment.
- That would only fix the issue for the duration of new applications being closed.
There isn’t anything new in regards to privacy with this move. We are just adding a tool to help flag content quicker, and we want to be transparent about out moderation. That’s why we made our admin guidelines public.
From what I’m gathering is that OP possibly should not be your spokesperson, as your wording sounds far less dystopian and controlling.
I still don’t like it, but if you are disclosing it properly and you’re not offshoring the data that is monitored, that will earn you some goodwill.
It really sounds like Lemmy software is trailing behind on keeping up with the bots.
We will have a review of how we write posts and follow up in the comments later today to make sure we are clearer in our communication, and that we don’t end up with 3 different admins chiming in on separate comments.
I still don’t like it, but if you are disclosing it properly and you’re not offshoring the data that is monitored, that will earn you some goodwill.
We try to always make pinned announcements when making notable changes to our operation to keep the community informed. Any changes, minor or major, that we believe our users would want to know, we will always post about here.
It really sounds like Lemmy software is trailing behind on keeping up with the bots.
Lemmy is trailing behind when it comes to moderation tools just in general, which is what that is forcing us to make our own tools to help with moderation.
deleted by creator
for a couple of those
The things that are checked are public (posts, comments, and dms that are also sent to other servers we dont control) and can be moderated on regardless of this. There isnt anything that breaches privacy
The way its being moderated isnt disclosed so that people cant just bypass it. The processing is local though like said in the original message (so for example were not sending this to some LLM/Ai to then use)
deleted by creator
Everything is sent for human review so false positives can be reversed quickly and the worst action that happens is removal of content temporarily (before thats reversed)
This just means someone cant spin up a new account and spam things on the site or on federated sites and have that linger until its able to be moderated
even before the edit it was fine… you just need to read more carefully.
