Outgoing federation may be fixed now (crossed fingers)

3 years ago by mrmanager to c/announcements

Please try to comment and post things now, and see if they federate again.

Hopefully you see your activity instantly federated. I have tried making comments both to instances running Lemmy 0.19.1 and 0.18.5 and they all federate like they should.

Hope you have the same experience! 🥳

mrmanager 6 points 3 years ago

Seems federation is still not fixed even though it certainly helped to federate some things, while leaving others... :/

On the positive side, the pull request that may fix federation properly in the software was approved by one of the Lemmy devs 2 hours ago so it should be merged with the main code soon:

https://github.com/LemmyNet/lemmy/pull/4330

path: 0 6352610, hotness: undefined, score: 6, children: 0

oleorun 6 points 3 years ago

I am reading this from lemmy.fan so you're definitely federating now!

path: 0 6334894, hotness: undefined, score: 6, children: 1

mrmanager 9 points 3 years ago

Thats great. :)

I know other instances on 0.19.1 had issues with federation so I can share what we did here to try and fix it.

There is a database table in postgres called instance and in our case, it had thousands of instances that were considered dead. Our instance didnt seem to try and talk to them anymore since the "updated" column would never update.

So we ran a simple database query to set the updated column to now() for all instances that we knew about:

update instance set updated = now()

This caused Lemmy to go out and try to talk to all of the instances again, and reconsider if they were dead or not.

According to this site we are now federated with 3000+ instances again:

https://phiresky.github.io/...

I hope we stay federated this time. Will keep an eye on it. :)

path: 0 6334894 6335077, hotness: undefined, score: 9, children: 0

Allero 4 points 3 years ago

Yay, thank you admin! Will test it out

path: 0 6338749, hotness: undefined, score: 4, children: 0

magnor 3 points 3 years ago

This fixed my federation issues as well. Thanks a bunch! I am however missing a lot of comments from this thread (Like the one from db0).

path: 0 6352198, hotness: undefined, score: 3, children: 10

mrmanager 4 points 3 years ago

Yeah its completely empty on https://lemmy.dbzer0.com, even after a restart of the server here.... :/

path: 0 6352198 6352509, hotness: undefined, score: 4, children: 9

magnor 2 points 3 years ago

Federation is really being weird since 0.19... But at least you managed to offer a bit of hope!

path: 0 6352198 6352509 6353740, hotness: undefined, score: 2, children: 8

mrmanager 2 points 3 years ago

Yeah I think it will be fixed soon. Just need to wait a bit longer for Lemmy devs to push some code. :)

path: 0 6352198 6352509 6353740 6355460, hotness: undefined, score: 2, children: 7

magnor 0 points 3 years ago

Turns out this only worked for a bit. I had to redo the fix today.

path: 0 6352198 6352509 6353740 6355460 6649088, hotness: undefined, score: 0, children: 6

MacNCheezus 3 points 3 years ago

Hmm, I made some recent posts to remote communities and so far they haven’t shown up on any other instances yet, but most of my recent comments at least seem to be visible.

Thanks for your hard work trying to figure this out and happy 2024.

path: 0 6352103, hotness: undefined, score: 3, children: 4

mrmanager 2 points 3 years ago

If you feel comfortable sharing a post that hasn't federated, please link to it. :)

path: 0 6352103 6352147, hotness: undefined, score: 2, children: 2

MacNCheezus 2 points 3 years ago

It’s not like they’re private anyways, but sure:

https://lemmy.today/post/4778593 https://lemmy.today/post/4786719

Neither of these seems to be visible on Lemmy.world

path: 0 6352103 6352147 6352166, hotness: undefined, score: 2, children: 1

mrmanager 1 point 3 years ago

Yes, it seems federation still is buggy. :/

Your comments are on lemmy.world now but they needed a Lemmy software restart to make them federate:

https://lemmy.today/post/4778593 => https://lemmy.world/post/10236722

https://lemmy.today/post/4786719 => https://lemmy.world/post/10236757

So still gremlins in the software...

path: 0 6352103 6352147 6352166 6352334, hotness: undefined, score: 1, children: 0

mrmanager 2 points 3 years ago

They should be there now - it seems a restart of Lemmy is still required for posts to federate....

For me Lemmy was working very well yesterday though, and I got responses to posts and comments right away. However, it seems like federation stops to work after a while again. The site I linked before:

https://phiresky.github.io/...

That one was showing that we had thousands of lagging instances and no updated ones just a few moments ago. And after a restart, they are all up to date again. So absolutely some bug remaining...

path: 0 6352103 6352424, hotness: undefined, score: 2, children: 0

db0 2 points 3 years ago

Interestingly, I see your OP, but I don't see your comment. However your instance's updated column is current in my instances table

path: 0 6352174, hotness: undefined, score: 2, children: 1

mrmanager 1 point 3 years ago

Yeah that post looks very empty on https://lemmy.dbzer0.com, and even after I restarted Lemmy here, none of it federated to the https://lemmy.dbzer0.com instance. But I can see that other posts and comments has federated to other instances after a restart, like these ones to Lemmy.world:

https://lemmy.today/post/4778593 => https://lemmy.world/post/10236722

https://lemmy.today/post/4786719 => https://lemmy.world/post/10236757

I get the feeling that there are many small bugs in the software....and getting a clear picture of why some things federate and some doesnt is difficult of course.

path: 0 6352174 6352502, hotness: undefined, score: 1, children: 0

Draconic_NEO 1 point 3 years ago

@mrmanager Tried following the community from Mastodon but it says pending. I'm not sure this entirely fixed it.

path: 0 6352164, hotness: undefined, score: 1, children: 4

mrmanager 1 point 3 years ago

Maybe it worked now after a restart of the server?

path: 0 6352164 6352524, hotness: undefined, score: 1, children: 3

Draconic_NEO 1 point 3 years ago

@mrmanager Doesn't seem to be fixed by a restart unfortunately.

path: 0 6352164 6352524 6356906, hotness: undefined, score: 1, children: 2

mrmanager 1 point 3 years ago

I guess we just wait for the software fix. Something must be wrong there...

path: 0 6352164 6352524 6356906 6356929, hotness: undefined, score: 1, children: 1

Draconic_NEO 2 points 3 years ago

@mrmanager Yup something is definately wrong here. Made a Github issue so hopefully it can be addressed soon.

path: 0 6352164 6352524 6356906 6356929 6356959, hotness: undefined, score: 2, children: 0

tal 1 point 3 years ago

I've been seeing federation issues again. For example:

https://lemmy.today/comment/4640480

That hasn't made it to sh.itjust.works, and I made the comment 4 hours ago.

Not sure if this is the same problem or something new.

path: 0 7219279, hotness: undefined, score: 1, children: 1

mrmanager 1 point 3 years ago

Yeah we need the new 0.19.2 version of Lemmy to hopefully fix things. Good news is that I saw it being worked on yesterday on github, so it should come very soon now. :)

path: 0 7219279 7219280, hotness: undefined, score: 1, children: 0

1984 1 point 3 years ago

Yep finally it works, getting responses to comments again!

path: 0 6336033, hotness: undefined, score: 1, children: 0

announcements

@lemmy.today

login for more options

Announcements about system updates or other things related to this instance.

go to feed...

announcements

@lemmy.today

login for more options

Announcements about system updates or other things related to this instance.

go to feed...