Fedia.io federation woes

2 years ago by jerry to c/fedia

Hi all. Several of you have reported problems with fedia.io not federating with other instances correctly.

The cause is that rabbitmq crashed, but not all the way. It crashed to the point where new connections would timeout, but the service was still running such that it wouldn't auto restart. I will be creating some automation to detect that proactively and restart rabbitmq if/when it happens again.

hitstun 5 points 2 years ago

Thank you! A thread I just posted made it to Lemmy instances within 15 minutes. The thread that didn't federate two days ago has found its way there too. Jerry is the best instance admin and Fedia.io is worth paying money for.

save

path: 0 12520673, hotness: undefined, score: 5, children: 1

jerry 5 points 2 years ago

Once it is more reliable, then I’ll agree, but thank you

save

path: 0 12520673 12523537, hotness: undefined, score: 5, children: 0

Chozo 3 points 2 years ago

Are we still having trouble? A thread I posted 45 minutes ago hasn't shown up on the instance I posted it to yet. The "Open original URL" menu on the post goes to the Fedia page instead of LemmyWorld. It seems like a comment I posted 2 hours ago is also not appearing on the original instance, either.

save

path: 0 12590845, hotness: undefined, score: 3, children: 14

jerry 5 points 2 years ago

ok - rabbitmq started having prroblems with the delivery queue again. I got it going again. Those messages should be delivered soon.

save

path: 0 12590845 12590890, hotness: undefined, score: 5, children: 12

dhhyfddehhfyy4673 2 points 2 years ago

save

path: 0 12590845 12590890 12600514, hotness: undefined, score: 2, children: 11

jerry 2 points 2 years ago

ok - the queues are processing again. I will work on a more permanent fix after dinner

save

path: 0 12590845 12590890 12600514 12600621, hotness: undefined, score: 2, children: 9

Chozo 2 points 2 years ago

Sorry to nag, but it looks like it might be acting up again lol. Doesn't look like any sync has happened for 5 hours.

save

path: 0 12590845 12590890 12600514 12600621 12618223, hotness: undefined, score: 2, children: 6

dhhyfddehhfyy4673 1 point 2 years ago

save

path: 0 12590845 12590890 12600514 12600621 12600622, hotness: undefined, score: 1, children: 1

jerry 2 points 2 years ago

Working on it

save

path: 0 12590845 12590890 12600514 12600617, hotness: undefined, score: 2, children: 0

jerry 3 points 2 years ago

I will check.

save

path: 0 12590845 12590877, hotness: undefined, score: 3, children: 0

testing 1 point 2 years ago

any guess why visibility on lemmy has been heavily affected, whereas from microfedi things looked normal?

save

path: 0 12512323, hotness: undefined, score: 1, children: 3

jerry 5 points 2 years ago

Not entirely. It looks like the rabbit issue was only impacting one of the queues (“deliver”), though I would have expected that to impact things like microblog too. All I can say with clarity is that the instance was operating in a very unhealthy state.

The queue appears like it’ll take several hours to flush, but it’s working.

save

path: 0 12512323 12512336, hotness: undefined, score: 5, children: 2

testing 2 points 2 years ago

i salute you!

save

path: 0 12512323 12512336 12512345, hotness: undefined, score: 2, children: 1

jerry 1 point 2 years ago

be aware that this is fixed now

save

path: 0 12512323 12512336 12512345 12733686, hotness: undefined, score: 1, children: 0