They deserve it

2 years ago by Blaze (he/him) to c/fedimemes

For the context: https://feddit.org/post/2374543

TL;DR: Aussie.zone has been experiencing a 7-days delay with LW since quite a long time due to the way Lemmy manages federation, and the distance between LW and aussie.zone.

Lemmy.world is the only one having such delay, even Lemm.ee which is the second most active instance is up-to-date (same for Lemmy.ml with a lot of active communities)

Graphs:

The issue has been solved in 0.19.6 (https://github.com/LemmyNet/lemmy/pull/4623) which hasn't been released yet.

Explanation by @Nothing4You@programming.dev

lemmy’s current federation implementation works with a sending queue, so it stores a list of activities to be sent in its database. there is a worker running for each linked instance checking if an activity should be sent to that instance, and if it should, then send it. due to how this is currently implemented, this is always only sending a single activity at a time, waiting for this activity to be successfully sent (or rejected), then sending the next one.

an activity is any federation message when an instance informs another instance about something happening. this includes posts, comments, votes, reports, private messages, moderation actions, and a few others.

let’s assume an activity is generated on lemmy.world every second. now every second this worker will send this activity from helsinki to sydney and wait for the response, then wait for the next activity to be available. to simplify things, i’ll skip processing time in this example and just work with raw latency, based on the number you provided. now lemmy.world has to send an activity to sydney. this takes approximately 160ms. aussie.zone immediately responds, which takes 160ms for the response to get back to helsinki. in sum this means the entire process took 320ms. as long as only one activity is generated per second, this is easy to keep up with. still assuming there is no other time needed for any processing, this means about 3.125 activities can be transmitted from lemmy.world to aussie.zone on average.

the real activity generation rate on lemmy.world is quite a bit higher than 3.125 activities per second, and in reality there are also other things that take up some time during this process. over the last 7 days, lemmy.world had an average activity generation rate of about 5.45 activities per second. it is important to note here that not all activities generated on an instance will be sent to all other linked instance, so this isn’t a reliable number of how many activities are actually supposed to be sent to aussie.zone every second, rather an upper limit. for example, for content in a community, lemmy will only send these activities to other instances that have at least one subscriber on the remote instance. although only a fraction of the activities, private messages are another example of an activity that is only sent to a single linked instance.

to answer the original question: the week of delay is simply built up over time, as the amount of lag just keeps growing.

AwesomeLowlander 14 points 2 years ago

That seems like a huge oopsie in the design...

path: 0 12751445, hotness: undefined, score: 14, children: 0
Ringmasterincestuous 13 points 2 years ago

Slowly drifting further away with each moment… as we like it

path: 0 12752595, hotness: undefined, score: 13, children: 1
Blaze 5 points 2 years ago

We try to keep you guys with us!

path: 0 12752595 12752635, hotness: undefined, score: 5, children: 0
Emperor 11 points 2 years ago

Well there's another reason for not using l.w - it's now so big, you need to host your instance in Europe to reduce the lag.

path: 0 12752283, hotness: undefined, score: 11, children: 5
OpenStars 8 points 2 years ago

Until v0.19.6 (the git link says "federation: parallel sending per instance #4623"), although I'm surprised that they haven't switched to a package model yet - that must be absolute hell on someone's network bandwidth to operate a Lemmy instance. Imagine an accidental downvote, then press it again to cancel, then press upvote, altogether counts as 3 separate network traffic packages. In comparison, I would naively guess that e.g. sending a whole minute's worth of activity would be much more preferable, or at least bundle all the activities together that occurred within the prior second to send at once?

path: 0 12752283 12752466, hotness: undefined, score: 8, children: 4
Emperor 6 points 2 years ago

And that's with everything operating as expected, when things go sideways it can really hammer an instance's performance.

It's why having funded and dedicated developers is critical as they can respond to issues like this and fix it while we are still in the early days.

path: 0 12752283 12752466 12753472, hotness: undefined, score: 6, children: 3
OpenStars 5 points 2 years ago

It's a good thing that nothing of consequence is planned to happen anywhere in the world, especially the USA, like related to politics or anything. Y-y-yeah it's a good thing there's nothing like that coming! 😔

path: 0 12752283 12752466 12753472 12753576, hotness: undefined, score: 5, children: 2
Emperor 5 points 2 years ago

I'm glad we have spare capacity here as you never know what bugs, runaway processes, surprise enshittification or real-world mayhem could be around the corner.

path: 0 12752283 12752466 12753472 12753576 12753676, hotness: undefined, score: 5, children: 1
logan_hero 10 points 2 years ago

path: 0 12751635, hotness: undefined, score: 10, children: 0
Blaze 9 points 2 years ago path: 0 12751175, hotness: undefined, score: 9, children: 0
YourPrivatHater 6 points 2 years ago
path: 0 12751252, hotness: undefined, score: 6, children: 4
Spezi 12 points 2 years ago

They have to ship a thumb drive with all federated posts over there every day.

path: 0 12751252 12751522, hotness: undefined, score: 12, children: 0
Emperor 8 points 2 years ago

It's only 7 days because, apparently, it just deletes things older than that.

path: 0 12751252 12752272, hotness: undefined, score: 8, children: 0
Olap 5 points 2 years ago

There's the TCP sliding window to solve this. Just needs done, along with a whole host of other work. I'd donate, but are the original pair any better at sharing work yet?

path: 0 12751252 12751572, hotness: undefined, score: 5, children: 1
Blaze 9 points 2 years ago

The parallel sending has been fixed in 0.19.6: https://github.com/LemmyNet/lemmy/pull/4623

The issue might still be around until

  • 0.19.6 is released
  • LW updates to it (they tend to be slower to update due to their size)
path: 0 12751252 12751572 12751880, hotness: undefined, score: 9, children: 0
Nath 5 points 2 years ago

Oh hey! That's us!
We've known about this issue since March. OP's post is a very good explanation of the problem. Lemmy wasn't designed with one huge instance in mind like this and lemmy.world is the only instance I'm aware of being a problem. Our Kiwi sisters and brothers hacked around the problem five months ago: They set up a server in Finland, slurp up bulk lemmy.world content in batches and then insert that content into their local database. They invited us to share their code to do this, but:

  1. It required us to spend money on a VM in Finland specifically for lemmy.world content - a precedent that made us hesitate.
  2. This is a temporary problem, a new version of Lemmy was meant to come out some months to remedy it. It simply hasn't been released, yet.

Had we known in May that this would still be an issue in October, we might have chosen differently.

path: 0 12911649, hotness: undefined, score: 5, children: 2
Blaze 2 points 2 years ago

Hello, thank you for jumping in!

path: 0 12911649 12914536, hotness: undefined, score: 2, children: 0
BlazeAlt 2 points 2 years ago path: 0 12911649 12914643, hotness: undefined, score: 2, children: 0
onlinepersona 1 point 2 years ago path: 0 12774252, hotness: undefined, score: 1, children: 7
db0 3 points 2 years ago

Batching is not apub standard, so they would be not compatible with non lemmy services. The new approach should resolve it

path: 0 12774252 12808995, hotness: undefined, score: 3, children: 2
onlinepersona 3 points 2 years ago

It might be worth proposing as an addition. I think we all know that batching is better for the network and processing. Parallelization might work now, but the network grows (and it probably will), large servers might have more than ~6 activities/second to send.

Anti Commercial-AI license

path: 0 12774252 12808995 12809501, hotness: undefined, score: 3, children: 1
db0 4 points 2 years ago

I think if they make the amount of activities you can send adjustable (automatically or manually) it can also scale. The problem is that it will also cause a lot more load, but I'm not quite certain by how much. But I will agree that apub needs to extend itself to allow batching federation activities as it's bound to become an issue eventually.

path: 0 12774252 12808995 12809501 12809738, hotness: undefined, score: 4, children: 0
Blaze 2 points 2 years ago path: 0 12774252 12774632, hotness: undefined, score: 2, children: 3
onlinepersona 1 point 2 years ago path: 0 12774252 12774632 12774750, hotness: undefined, score: 1, children: 2
Blaze 2 points 2 years ago

Feel free to suggest this in a PR

path: 0 12774252 12774632 12774750 12774780, hotness: undefined, score: 2, children: 1
onlinepersona 1 point 2 years ago path: 0 12774252 12774632 12774750 12774780 12774848, hotness: undefined, score: 1, children: 0
fedimemes
fedimemes

@feddit.uk

login for more options
3529
639
749

Memes about the Fediverse.

Rules

General
  1. Be respectful
  2. Post on topic
  3. No bigotry or hate speech
  4. Memes should not be personal attacks towards other users
  5. We are not YPTB. If you have a problem with the way an instance or community is run, then take it up over at !yepowertrippinbastards@lemmy.dbzer0.com.
    • Addendum: Yes we know that you think ml/hexbear/grad are tankies and or .world are a bunch of liberals but it gets old quickly. Try and come up with new material.
  6. This is not the place to start flamewars between Lemmy, Mbin and Piefed.
  7. No generative AI content (instance rule)

Elsewhere in the Fediverse

Other relevant communities:

go to feed...