[Help] New self-hosted instance not crawling communities

Just got my own instance up and running finally, but it doesn't seem to be federating as expected.

I can search for and subscribe to communities, but unless I specifically search for the URL (e.g. selfhost@lemmy.ml) nothing shows up. I'm also making this post from my instance as well (to see if anything happens...)

Is this something I just need to wait a few hours to happen or is something incorrect.

Thanks in advance

Edit: Oh, and when I do subscribe to a community, I can only see new posts. It doesn't retrieve older posts

redcalcium 6 points 3 years ago

I can only see new posts. It doesn’t retrieve older posts

This is normal. You'll only receive new posts going forward from ActivityPub. Think of it like joining a mailing list, you'll only receive new mails after subscribing. If you want to pull older post, you'll have to do it manually by pasting the post (or comment) url in the search field.

save

path: 0 916463, hotness: undefined, score: 6, children: 1

NotNotMike 2 points 3 years ago

Yep you're right. And I've discovered that, in order to see posts from a community, at least one user in your instance needs to be subscribed to a community to view it.

Would be nice if someday the browse functionality provided by some sites was integrated into Lemmy but for now I had to go through and manually subscribe to a bunch of things.

I'd use a scraper but there's a lot of content I don't care to see right now, and I'm assuming subscribing comes with a small storage cost, so may as well avoid it for now

save

path: 0 916463 920247, hotness: undefined, score: 2, children: 0

fireshaper 5 points 3 years ago

You don't get federated automatically. On your own instance you will only see posts in communities that you subscribe to. What I did was set up a "bot" user that I used to subscribe to a bunch of communities that I might find interesting when just browsing the All feed but I didn't want in my personal Subscribed feed. There's a few tools that will automatically subscribe a bot user on your instance to all the top communities but I found just manually going through https://lemmyverse.net/communities and clicking subscribe as that bot user worked for me.

save

path: 0 918593, hotness: undefined, score: 5, children: 0

eric5949 4 points 3 years ago

I set up this: https://github.com/Fmstrat/lcs

It goes somewhat slowly but it's been pulling in communities for me.

save

path: 0 915515, hotness: undefined, score: 4, children: 0

datallboy 3 points 3 years ago

What version are you using? There were issues federating with v 0.18.0. I updated my instance to 0.18.0-rc.10 and that resolved the issues.

save

path: 0 915480, hotness: undefined, score: 3, children: 0

terribleplan 3 points 3 years ago

Things don’t get backfilled, so until a new action happens on an old post/comment/etc they won’t show up on your instance. New things should make their way in eventually though.

Taking the link of a specific post/comment from the community instance and searching for it from your instance should populate it on your instance, just like you probably had to do to get this community to show up so you could subscribe/post at all.

There are backfill tools/scripts, but unless you really want old posts I wouldn't use them. It unnecessarily increases the load on already struggling popular/overloaded instances like lemmy.world.

save

path: 0 917127, hotness: undefined, score: 3, children: 0

throwaway_OT05wZjv 2 points 3 years ago

save

path: 0 933153, hotness: undefined, score: 2, children: 2

NotNotMike 1 point 3 years ago

Do you know how much data is put onto the local device when subscribing to all these communities? Ballpark is it in the MBs? GBs?

Trying to determine whether discovering things I don't want is worth it

save

path: 0 933153 933284, hotness: undefined, score: 1, children: 1

throwaway_OT05wZjv 2 points 3 years ago

save