PSA: Lemmy.world is having some heavy federation issues lately.

3 years ago by simple to c/support

As per this post, it seems Lemmy World is having federation issues. I've had some of my posts and comments get suspiciously low response, so I've looked into it and it turns out that many of them aren't showing on other instances.

Here's an example: my Payday 3 post is only visible on Lemmy.world:

Meanwhile, in other instances, it doesn't show up at all:

It's not clear if this issue is exclusive to this instance but I thought I'd give people a heads up. I reposted and now it showed up in other instances. If you feel like you're getting no response from other communities, check on another instance if your content is visible, otherwise repost it.

thayer 90 points 3 years ago

It isn't just Lemmy.world, and isn't limited to v0.17 either. Lemmy.ca, which has been running v0.18 for days, is also not seeing a lot of federated content, nor accurate community subscription counts from other instances, including lemmy.ml and others running v0.18. I'm posting this from my .world alt because .ca is showing zero comments in this thread.

There have been several not-so-friendly comments made about the issues by someone in !lemmy_administration@lemmy.ml, calling out the site admins of .world and .ml for not sharing their server logs, and the overall silence on the issue. The user's concerns are genuine, but I don't agree with their behaviour.

I suspect the devs were/are completely overwhelmed by the mass influx of redditors these past two weeks. I assume someone is looking into it, but frankly I wouldn't blame anyone if they chose to step away for a few days. I think a lot of folks (particularly "the masses") forget that these are largely volunteer efforts, by people with day jobs.

path: 0 554284, hotness: undefined, score: 90, children: 10

ruud 106 points 3 years ago

Thanks for your reply. This site isn’t even a month old and the user growth is spectacular. Lemmy hasn’t ever been tested with these amounts of users.

I’m sure the devs are doing their best in getting it all more performant. Meanwhile the team at lemmy.world is trying to get a grip on what’s happening with federation. There’s not much to be found in logs, anyone who can help troubleshoot this let us know. The only real adjustable setting is the federation workers. We’ve apparently either set that too high or too low. I’m gonna set the log level back to info on the hopes we get some queue stats…

To be continued

path: 0 554284 554437, hotness: undefined, score: 106, children: 7

thayer 21 points 3 years ago

No worries, and I completely sympathize with the situation. My hat is off to you site admins, and the lemmy devs, for keeping things afloat these past couple of weeks, often at great expense of both time and money. I am confident that all of these kinks will be ironed out in the coming months, and in the meantime I am happy to have a few alts to hold me over.

path: 0 554284 554437 554631, hotness: undefined, score: 21, children: 0

sadreality 17 points 3 years ago

path: 0 554284 554437 554683, hotness: undefined, score: 17, children: 0

danc4498 12 points 3 years ago

Back in my day 1 month old sites had 0% downtime and no reported glitches. 1 million daily active users! There's no need for beta software either cause 2.0 was always the perfect release.

Those were the days!

path: 0 554284 554437 554833, hotness: undefined, score: 12, children: 2

setsneedtofeed 5 points 3 years ago

Looks like grandpa has gotten into the horseradish again.

path: 0 554284 554437 554833 575696, hotness: undefined, score: 5, children: 0

A_A 1 point 3 years ago

"fediverse" x "lemmy" - - proves to be too complicated to work reliably on available infrastructure ...while early websites were quite simple and so very efficient.

path: 0 554284 554437 554833 577406, hotness: undefined, score: 1, children: 0

deweydecibel 4 points 3 years ago

So to be clear, we're fairly certain this is a technical issue and not because of something other instances are doing?

path: 0 554284 554437 554581, hotness: undefined, score: 4, children: 1

thayer 12 points 3 years ago

It's almost certainly technical issues relating to scalability. The user I mentioned above has created a community for tracking several performance issues that are likely impacting the platform now that user and post counts have increased exponentially.

path: 0 554284 554437 554581 555011, hotness: undefined, score: 12, children: 0

cashews_win 9 points 3 years ago

for not sharing their server logs

WTF do users think they have a right to see these at all? Just because Lemmy is open source doesn't mean you get to run the damn thing.

Also my condolonces on you being a Canadian.

path: 0 554284 561169, hotness: undefined, score: 9, children: 0

Neato 5 points 3 years ago

I sympathize. I just hope it's not true. Friday and Saturday of this week are likely the second and bigger wave of redditors coming over as their 3rd party apps get disabled on/after 30th. I hope the admins are preparing for it.

path: 0 554284 556256, hotness: undefined, score: 5, children: 0

BobbyBandwidth 28 points 3 years ago

Correct me if I’m wrong but .18 lemmy doesn’t support captcha, so lemmy.world is waiting for a patch before updating so we don’t get flooded with bots.

Edit: fixed typo

path: 0 553411, hotness: undefined, score: 28, children: 1

Spliffman1 10 points 3 years ago

path: 0 553411 553560, hotness: undefined, score: 10, children: 0

dezmd 13 points 3 years ago

Just clicked on a Futurama related post on Lemmy.world front page, it opened wtih this MLB Game Thread as content instead of what I clicked on. **This is happened more than once. Previously I thought I derp clicked on the wrong item **

path: 0 561096, hotness: undefined, score: 13, children: 2

Sunforged 10 points 3 years ago

This has been happening to me for at least a day now. Reloading fixes it and brings you to the post you thought you were in.

Funny thing is sports communities are the ones I block as I come across them, and this glitch circumvents blocked communities.

path: 0 561096 561418, hotness: undefined, score: 10, children: 0

Couldbealeotard 2 points 3 years ago

I'm not kidding when I say I opened a meme post and the picture was from the "bulges" page. The comments were the correct meme post comments though. I went back and forward again in my browser and the image was correct.

path: 0 561096 561381, hotness: undefined, score: 2, children: 0

rcmaehl 12 points 3 years ago

I believe this is a side effect of the 0.17 vs 0.18 split which unfortunately wasn't backwards compatible. Once 0.18.1 rolls out with readded captcha everything should be hunky-dory again.

path: 0 553425, hotness: undefined, score: 12, children: 5

simple 8 points 3 years ago

Fingers crossed it doesn't take that long!

path: 0 553425 553440, hotness: undefined, score: 8, children: 1

chicane 3 points 3 years ago

Interesting that’s it’s been days without a minor beta at all. Where in the race for 0.18 the updates were 12 hourly at one point. Guess all the devs burnt the candle at both end. Much appreciated ❤️

path: 0 553425 553440 570035, hotness: undefined, score: 3, children: 0

danc4498 1 point 3 years ago

Was !games@sh.itjust.works the instance that was not showing the post?

path: 0 553425 554914, hotness: undefined, score: 1, children: 2

simple 2 points 3 years ago

Yes, I ended up reposting there and it shows up now.

path: 0 553425 554914 555538, hotness: undefined, score: 2, children: 1

danc4498 1 point 3 years ago

Maybe it's just the version difference, then. 🤞

path: 0 553425 554914 555538 557596, hotness: undefined, score: 1, children: 0

ericjmorey 12 points 3 years ago

Lemmy.world is definitely having federation issues. Lemmy in general is having a lot of federation issues, but it looks like lemmy.world is being effected more so due to the number of accounts and activity on lemmy.world in combination with different versions of lemmy handling messages a bit differently with different implicit assumptions.

path: 0 553480, hotness: undefined, score: 12, children: 0

static 8 points 3 years ago

Workaround, I had to do the same on kbin.

Editing your comment triggers another sync, often an edit will make the comment or post synchronise to all other instances.

path: 0 555615, hotness: undefined, score: 8, children: 0

dragontamer 7 points 3 years ago

To add some specificity to this:

I've been unable to post to !arduino@lemmy.ca or !avr@lemmy.ca successfully. And the comments seem to be failing to load at all.

Here's to hoping its the .18.x vs .17 issue. But I figured I need to post an official report of my problems somewhere, and this topic seems the most applicable location.

I've also been "Subscription pending" on !avr@lemmy.ca for... at least 20 minutes now. Suspiciously long times to get that resolved.

path: 0 554239, hotness: undefined, score: 7, children: 2

dragontamer 1 point 3 years ago

5 hours later, I'm still "Subscription pending". I dunno if this is related to the issues, but I figure I should document the problem here.

path: 0 554239 560846, hotness: undefined, score: 1, children: 1

dragontamer 2 points 3 years ago

Almost a day later, "Subscription pending". I'm guessing its this slow-federation issue that probably will be fixed with .18 upgrade (or maybe lemmy.ca needs to increase its federation workers?)

path: 0 554239 560846 575412, hotness: undefined, score: 2, children: 0

necropola 7 points 3 years ago

That's pretty much what I yesterday called the lemmy.world bubble.

path: 0 554564, hotness: undefined, score: 7, children: 0

SpezCanLigmaBalls 6 points 3 years ago

If it was reposted and worked that’s not a federation issue right? Could it be that world isn’t on .18 yet?

path: 0 553306, hotness: undefined, score: 6, children: 1

necropola 3 points 3 years ago

path: 0 553306 555884, hotness: undefined, score: 3, children: 0

snailwizard 6 points 3 years ago

There should be a fix in a day or two, there was an issue with captcha to my understanding.

path: 0 559843, hotness: undefined, score: 6, children: 1

CH3DD4R_G0BL1N 1 point 3 years ago

Ruud has said he doesn't know if any .18x update will fix these issues

path: 0 559843 576862, hotness: undefined, score: 1, children: 0

krayj 4 points 3 years ago

I'm very new to Lemmy and have already been experiencing this problem just testing how commenting to a federated community on another server works. It doesn't/hasn't worked for me yet in controlled testing. I've been assuming it was a problem with the other lemmy instance where the community is created/hosted but now I'm rethinking that assumption.

Usually, not assigning sufficient resources at a workload just causes the backlog queue to get longer, but in the case of lemmy it seems to result in some items never getting processed at all. That's an odd behavior for a backlog to have.

path: 0 556754, hotness: undefined, score: 4, children: 0

heili 3 points 3 years ago

I've had difficulties with responding or commenting on posts on federated servers. Comments work just fine if I'm on this instance, but threads on others it just never posts.

path: 0 575801, hotness: undefined, score: 3, children: 0

code 2 points 3 years ago

I got a report earlier that my community wasn't showing up on https://browse.feddit.de earlier as well, but it's back now. I'm not sure how it works but I assume it's related.

path: 0 553492, hotness: undefined, score: 2, children: 4

CAVOK 2 points 3 years ago

Any idea on that 0.18.1 release? I have no idea how much work is required to add re:capcha. Are we talking days, weeks or months off work?

path: 0 553492 554752, hotness: undefined, score: 2, children: 3

necropola 5 points 3 years ago

path: 0 553492 554752 555763, hotness: undefined, score: 5, children: 2

jjagaimo 4 points 3 years ago

It just got approved after some edits

path: 0 553492 554752 555763 571390, hotness: undefined, score: 4, children: 1

necropola 5 points 3 years ago

... and merged to LemmyNet/lemmy:main 🙂

path: 0 553492 554752 555763 571390 571545, hotness: undefined, score: 5, children: 0

possiblylinux127 1 point 3 years ago

This is why its a bad idea to have massive servers

path: 0 575972, hotness: undefined, score: 1, children: 0

LollerCorleone -12 points 3 years ago

Just wanted to let you all know that the federation with kbin.social is working fine.

path: 0 554738, hotness: undefined, score: -12, children: 2

atocci 2 points 3 years ago

I'm not sure, I can't see their original Payday 3 post, only the reposted one. Looking a little closer, they might have just deleted it since I can't see it from lemmy.world either.

path: 0 554738 555414, hotness: undefined, score: 2, children: 1

tal 1 point 3 years ago

It might be interesting to have one of the monitoring tools that people are banging check on post latency between various instances.

path: 0 554738 555414 556883, hotness: undefined, score: 1, children: 0

support

@lemmy.world

login for more options

Lemmy.world Support

Welcome to the official Lemmy.world Support community! Post your issues or questions about Lemmy.world here.

This community is for issues related to the Lemmy World instance only. For Lemmy software requests or bug reports, please go to the Lemmy github page.

This community is subject to the rules defined here for lemmy.world.

To open a support ticket

You can also DM https://lemmy.world/u/lwreport or email report@lemmy.world (PGP Supported) if you need to reach our directly to the admin team.

Follow us for server news 🐘

Outages 🔥

https://status.lemmy.world

go to feed...

support

@lemmy.world

login for more options

Lemmy.world Support

Welcome to the official Lemmy.world Support community! Post your issues or questions about Lemmy.world here.

This community is for issues related to the Lemmy World instance only. For Lemmy software requests or bug reports, please go to the Lemmy github page.

This community is subject to the rules defined here for lemmy.world.

To open a support ticket

You can also DM https://lemmy.world/u/lwreport or email report@lemmy.world (PGP Supported) if you need to reach our directly to the admin team.

Follow us for server news 🐘

Outages 🔥

https://status.lemmy.world

go to feed...