Last night, between 23.00 on 23 June and 1.00 AM on 24, the Poliverse instance “exploded” a thousand messages around the fediverse!
** We have to apologize to a lot of people: ** • we apologize to anyone who was ** woken up by our notifications **; • we apologize to those who have cisted their ** federated timeline invaded by messages ** coming from Poliverso; • We apologize to the ** administrators and instance moderators who know us ** who probably wondered who hacked our server; • We apologize to the ** administrators who didn’t know us ** and who noticed us in the worst way • We apologize most of all ** to some of our users ** who, through our fault, have ** UNKnowingly ** behaved as a spammer …
Unfortunately, we could not foresee this collapse and we did everything possible to limit the damage: unfortunately we are sure that our instance or some of our users have ended up in some blacklist.
** If so, we ask you to read this message, in which we explained what happened. **
PS: if any Friendica expert should realize that some considerations are wrong or should be better defined, please let us have your own considerations in the comments!
What happened to the Poliverso server?
The update
The new Friendica release, the 2022.06 called “Giant Rhubarb” is a seemingly useless release, because nothing changes for users! Yet it has greatly improved compatibility with other software in the fediverse: there has in fact been immense work by the developers and there have been many changes; in such cases it is possible that disservices arise and, although some instances have already installed an exploratory version, it is clear that some problems can only emerge during the production phase.
Precisely for this reason we decided to proceed with the installation only several days after the release, to understand what the problems would be for users and notify them in time; and yesterday 23 June at 22.00, the Poliverso server was updated.
The problem
At first it didn’t seem like there were any problems and, in fact, after half an hour we stopped monitoring. But at 11.00 something happened …
Admin 1 22.56 (the developers of friendica) have changed something: some users get a lot of notifications about old things … then everything is fixed 22.56 (I’m seeing a lot of queue in processes)
Admin 2 23.00 So far I am not having this type of problem. Actually, I am not having any problems 23.01 To be more precise, I still can’t see anything different than before … 😅 23.04 Right now, after a few tens of minutes of pause, he’s posting stuff on Twitter 23.07 Holy shit! my accounts are flooding the universe with messages
Admin 1 23.14 in what sense? : D
Admin 2 23.15 in what sense? The system is shitting out all the messages posted by the RSS Feed. It is a disaster. I’m emptying the ocean with a spoon 🤣 23.15 This time we are banned from all the galaxies of the fediverse
Admin 1 23.16 I noticed a lot of RSS stuff, but I said to myself “boh”
Admin 2 23.16 They are the queues of those messages
In short, yesterday there was a real disaster for which the Poliverse instance will pay the consequences for a long time as in reputational terms. However, the management of the problem was absolutely timely both from the point of view of resolution (at 1:00 AM almost all the messages had been deleted) and from the point of view of communication.
https://poliverso.org/display/0477a01e-4062-b4e8-2a73-375308279474
The reasons for the problem
However, the problem was not systemic or due to problems of administration of the instance; the problem was due to a series of factors that multiplied its impact:
- Friendica upgrade bugs exist: the “system” upgrade all together, some integration problems with other activitypub instances; but all this causes a “re-reading” of the notifications from the RSS FEED; in this way, users who also repost some posts from the feed get** a new repost (😱!)** of the feed.
- this system (republishing from feed) is not very common among Friendica users; so the problem was not noticed too much on the forums
- this system, however, is very powerful and we Poliverse administrators have often sponsored it with users: therefore some of our users use it; and some “service” and information users that we manage directly on a personal level or as administrators also use it: [@ macfranc @ poliverso.org [(https://poliverso.org/profile/macfranc), [@ informapirata @ poliverso .org [(https://poliverso.org/profile/informapirata), [@ news @ poliverso.org [(https://poliverso.org/profile/notizie), [@ cybersecurity @ poliverso.org [(https: //poliverso.org/profile/cybersecurity), [@ privacypost @ poliverso.org [(https://poliverso.org/profile/privacypost), [@ piratepost @ poliverso.org [(https://poliverso.org/ profile / piratepost), [@ instruction @ poliverso.org] (https://poliverso.org/profile/ Formazione)
- inside Friendica there are no valid administration or moderation tools (an ancestral and well-known Friendica problem: https://github.com/friendica/friendica/issues/8724), which would allow us to delete messages in bulk or to immediately stop the republishing service from feed or to temporarily suspend the “unwittingly spammy” users. So we either manually deleted and deactivated everything or had users manually delete their messages - this was an incredible waste of time!
The reputational damage
All of this prevented us from predicting the problem and then amplified it. However, we can assure you that the management of the application was professional and within a couple of hours from the alert, we have erased all traces of the problem!
The fact remains that some users who followed the accounts in question probably wondered “what’s going on?”; others have probably defollowed, silenced or banned the accounts of innocent and unaware users …
The same thing will have happened for administrators: some of them, to protect their users from spam, will have silenced or banned users; or they will have silenced or banned the instance!
Fortunately, those in the Central European Time Zone were probably asleep at the time and were not online on Friendica, Mastodon, Misskey, Pleroma or Hubzilla
Final thoughts
Of course during those hours we felt [a furious rage] (https://poliverso.org/display/0477a01e-1062-b4db-9d28-f53436523782) towards the Friendica developers who caused us such a problem…
https://poliverso.org/display/0477a01e-1062-b4db-9d28-f53436523782
…but we clearly don’t get mad at them. Moreover, it is thanks to them that Friendica exists; it is thanks to them that there is what we believe is the software of the fediverse that allows the most complete social experience today.
Next time, however, we recommend that you report the problem well and above all, express at least a little regret to the Friendica instance administrators and other administrators who had to manage what seemed to all intents and purposes a massive spam activity!
I mean, users can decide to move to an instance with less moderation.
@Ephera People keep on saying that…same with those who keep on saying: if you do not like google, use bing. Don’t like Facebook? Use Twitter. And they are blind to the point that’s being made.
I really don’t see those as the same thing. You can’t talk to the same people on Twitter that you know from Facebook. With the Fediverse, you generally can.
A user on https://mastodon.technology can migrate to https://fosstodon.org and will generally find the same people + whoever may have been silenced by more aggressive moderation.
Same thing the other way around. Most users don’t care about seeing everything, and may just find it more convenient when trolls and assholes are already filtered out for them, so may migrate from Fosstodon to mastodon.technology.
No, this is not a perfect system on an individual user basis, since most users won’t really know what they’re not seeing.
But I do think, in the grand scheme of things, people will bundle up around instances that offer a good balance between interesting content and trolls/assholes.
@Ephera But my point is that this practice is harmful for the fediverse as a whole. Non transparent, decided for thousands of users, shallow decisions in many instances. To move from one federated platform to another is not as easy as you make it sound. I know, I have done that and I also manage a few instances. So this is a non-solution.
The solution is to, for one, make all instance-level blocking of users or other instances transparent and notify all users when any future such decision is taken. And second, try not to panic. It is fine. Words, no swords, we are dealing with. Users have a lot of control over that they see. Teach them to exercise that. This is a far better approach.
@tio @Ephera A very nice example of how one can be transparent is the public moderation register in Lemmy.
The measures of the administrators of the instance and of the moderators of each community are made evident. This is a very useful thing to understand how an instance is administered and understand if the moderators practice abuse or support a balanced moderation aimed at maintaining a plural and peaceful environment.
lemmy.ml/modlog
@admin @tio @Ephera Thats about as transparent as you can be 😆
@tio @Ephera Well when inatances block other instances it is quite transparent. For exmaple, if you go to https://mastodon.social/about/more you can see all the instances that are filtered/blocked. Im not exactly sure if this is automatic, but if it isnt, it should be and for all instances (on an opt out basis).
We must make a premise: there are different types of instances in the fediverse and some of these, especially mastodon, are of a strongly generalist nature, apolitical and dedicated to an audience that does not want to see disturbing content (Not Safe for Work NSFW).
These instances are forced to silence and sometimes block users who post content of this type without the content warning tag.
This can be considered excessive, but it is the only way to keep an environment free of disturbing elements such as overly fiery political speech, explicit content, foul language.
All this concerns above all the generalist instances.
As for the others, it becomes more rare to practice BAN and silencing.
Unfortunately, however, there are two types of drastic measures: those towards users of your own instance and those towards users of other instances.
In my opinion, the latter should undergo a periodic review, a sort of amnesty that occasionally cleans up the sentences imposed or at least reconsiders some users who at the beginning of their life in the fediverse behaved badly because they were used to the toxic environment of Facebook or of Twitter.
In fact, the problem with these users is that they risk being invisible for life from some instances.
@BobIsMyManager @Ephera Yes this is a decent start and I wonder too if it is automated. It should. Plus, if there is a new block, should create a notification for all users by default I would say.
@tio @Ephera Maybe this can be achieved with an account which is followed by default which announces whenever changes occur to the instance filter/suspend list
@BobIsMyManager @tio @Ephera
Alternatively, there could be a public modlog with a RSS feed (granted, not all fedi software allow to follow RSS feed 😀 )
Lemmy uber alles… 😁 https://lemmy.ml/modlog
@BobIsMyManager @Ephera Probably should be better than this, but for sure an instance needs to be very open about this practice.