Potential workaround for federation issues?

3 years ago by doctortofu to c/community

I’ve found the following work-around works pretty well. If you host an instance that’s currently on 0.19.0 or 0.19.1, consider implementing this. There are two bugs that this helps with: - Outgoing federation stops after about 6 hours [https://github.com/LemmyNet/lemmy/issues/4288] - Another federation bug which is triggered if the server isn’t running at midnight every night [https://github.com/LemmyNet/lemmy/issues/4039] Work-around: Create cronjobs that restart the Lemmy container every 6 hours (but not at midnight). The following example is used for a Debian system running Lemmy in Docker. Type crontab -e into the terminal Add something like the following: 0 1 * * * docker container restart lemmy-lemmy-1 0 7 * * * docker container restart lemmy-lemmy-1 0 13 * * * docker container restart lemmy-lemmy-1 0 19 * * * docker container restart lemmy-lemmy-1 3 1-23/6 * * * docker container restart lemmy-postgres-1 && sleep 60 && docker container restart lemmy-lemmy-1 By restarting the container every 6 hours, outbound federation continues to work. There may still be some delays, but everything gets cleared up regularly. By telling it what time to restart (0100, 0700, 1300, and 1900 as opposed to “every 6 hours”), it avoids restarting at midnight. This avoids the second bug. My instance has been doing this for enough days where I’m confident that it’s working. You can check your federation status here [https://phiresky.github.io/lemmy-federation-state]. Note that it’s normal for there to be 0 up-to-date instances and a lot of lagging instances. As long as they sometimes turn “up to date”, then everything is getting caught up.

Just posting a link in case our awesome admin team missed it - Happy New Year guys, you're great!

example 5 points 3 years ago

thanks, we actually already have something similar in place since a couple days ago

save

path: 0 6332924, hotness: undefined, score: 5, children: 0

ticoombs 4 points 3 years ago

Why thankyou & Happy New Year to you too. As @example@reddthat.com has said, we've been doing this for what seems like a couple weeks but is probably only a week in reality. 😅

Unfortunately it's very inconsistent and we've noticed sometimes 3 hours before comments have synced between instances. My current theory is this is probably due to some instance admins having enabled the "fix" while others haven't.

I despise the turn it off n on again nature as it never solves the underlying issues and only causes temporary solutions. There is a possibile underlying solution https://github.com/LemmyNet/lemmy/pull/4330 but I have yet to find the time to start building our own containers.

If we start this route I would like to have the build systems incorporated with our git repository so that we can have repeatable builds and a way so I can take a week off and the other admins could still make a deployment if necessary.
This has a side effect to ensure I am compliant with any GPL requests, (or whatever the license is for Lemmy) as currently I can point people directly to the Lemmy git rather than my own.

This info was going to be part of the January announcement I've been putting off writing as well, due to the holidays. So we'll say it's a mini announcement and I'll pin this thread.

Cheers,
Tiff

save

path: 0 6339679, hotness: undefined, score: 4, children: 0

Reddthat Community & Support

Introductory Required Reading

Alternative Matrix Chatroom:

Alternative Support Forums:

Reddthat Community & Support

Introductory Required Reading

Alternative Matrix Chatroom:

Alternative Support Forums: