Introducing the Threadiverse Catalog

5 days ago by panos :catodon: to c/fediverse

Cross-posted from !fediverse@piefed.social

I'm the maintainer of Catodon, which offers both microblogging and forums (Lemmy-compatible communities), and we noticed that discoverability is not the threadiverse's strongest point: Our server only showed forums/communities that it "knew" because someone from the server already followed them. In Reddit it's easy to see if there is a community for a subject you care about; on the Fediverse it's not that easy.

So I looked at what indexes there are, and most were focused in Lemmy - lemmyverse.net has added MBin and PieFed explorers, but I was looking for something that covers more platforms, and is a unified directory, rather than per-platform only. Maybe there is something that I missed, but I didn't find what I was looking for: a directory for the threadiverse. So I decided to build it.

The Threadiverse Catalog lists forums/communities from Catodon, Discourse, Friendica, Lemmy, Mbin, Mobilizon, NodeBB and PieFed. It hasn't been online for many days, so it hasn't indexed everything, but we already list over 20.000 forums/communities. If there are any bugs or other issues, please let me know.

We built this primarily for Catodon, and on our latest release (26.10.0 "Bongo Cat") users have access to its results through the in-app Search and Explore features, but it's open for everyone to use. It's open source and written in Go. Other projects are also welcome to use it.

Cheers =)

#threadiverse #fediverse #catodon #discourse #friendica #lemmy #mbin #mobilizon #nodebb #piefed

Hamiller 12 points 5 days ago

There's also a directory of Friendica groups. Maybe we could adopt a few of them.

https://dir.friendica.social/group

save

path: 0 26208164, hotness: undefined, score: 12, children: 1

panos 4 points 5 days ago

Thanks, we added it to our sources!

save

path: 0 26208164 26217143, hotness: undefined, score: 4, children: 0

pinhead77 7 points 5 days ago

If you search for terms like "mastodon" you get a lot of results with no context to the search. So there should be an option to search within titles only.

save

path: 0 26210278, hotness: undefined, score: 7, children: 2

panos 7 points 5 days ago

Oh yes, that's a great suggestion, will add it sometime in the next 24 hours. Thanks!

save

path: 0 26210278 26210295, hotness: undefined, score: 7, children: 0

panos 2 points 5 days ago

Implemented =)

save

path: 0 26210278 26216473, hotness: undefined, score: 2, children: 0

lena 4 points 5 days ago

imo a better way to implement this kind of global discoverability would be to change the ActivityPub protocol itself so that instead of having to follow each actor/community on a server to get all of its content, users could just follow one community (or maybe the entire instance) to sync all of the instance's content.

save

path: 0 26208350, hotness: undefined, score: 4, children: 1

rglullis 2 points 5 days ago

There is no need to change activitypub for that. Relays already do exactly what you are asking.

save

path: 0 26208350 26214550, hotness: undefined, score: 2, children: 0

tal 4 points 5 days ago

lemmyverse.net is the closest thing I've seen to this, and my experience has been that it has problems with timeouts when used over a slow connection, so I think that this will be useful to many.

EDIT: Note that it does have some useful features that might be useful to replicate, like:

Showing post and comment count (not sure if this is applicable to all instance types)
A tri-state setting for NSFW content, so that someone could search only for NSFW content rather than only not for NSFW content.
Currently, lemmyverse.net does some processing of the Markdown text in the description to plain text to make it more readable. For example, for !technology@lemmy.world, this is lemmyverse.net's description:

This is a most excellent place for technology news and articles. Our Rules Follow the lemmy.world rules. Only tech related news or articles. Be excellent to each other! Mod approved content bots can post up to 10 articles per day. Threads asking for personal tech support may be

And on the Threadiverse Catalog:

This is a [most excellent](https://media.giphy.com/...) place for…

I'm not sure whether Catodon uses Markdown in its community descriptions or not.

I'm not familiar with Catodon, so I don't know if these would also be valuable for Catodon or not; might not be worth your effort if not.

save

path: 0 26212599, hotness: undefined, score: 4, children: 1

panos 2 points 5 days ago

All three implemented, thanks!

save

path: 0 26212599 26216469, hotness: undefined, score: 2, children: 0

dsilverz 3 points 5 days ago

It's excellent! Just two ideas I'd suggest:
- Ability to search/list all indexed forums/communities belonging to specific instances/servers indexed by the catalog; for example, having a Instance picker alongside the "Platforms" dropdown. It's not something that would fetch non-indexed instances, but rather something that would filter among the already-indexed ones.
- Speaking of "Platforms" dropdown, either some kind of general toggle/switch and/or some kind of "filter by this specific platform" so, for example, in order to search for a specific platform (e.g. Lemmy), one doesn't need to toggle off the switches for all the other platforms (currently the search form is default "all"; as more platforms and derivatives get eventually added, toggling 'em all off just to filter one platform will eventually become a problem)

Also, just to confirm, is lemmy.eco.br (and to a certain extent bolha.forum which only lists one community while they do have many) not in the list because of their usage of Cloudflare (and Anubis in case of bolha.forum)?

save

path: 0 26218731, hotness: undefined, score: 3, children: 2

panos 1 point 5 days ago

Good ones, will work on them in the next days!

About the two servers, I don't know - our crawler needs a few days to go through everything for the initial listing, so it might just not have gotten to them yet. But the best way is to use the form to suggest a community from each of these servers. which will introduce them to the catalog.

save

path: 0 26218731 26219064, hotness: undefined, score: 1, children: 0

panos 1 point 4 days ago

both implemented

save

path: 0 26218731 26224848, hotness: undefined, score: 1, children: 0

panos 2 points 5 days ago

Thanks to everyone who replied! Most of the suggestions have been implemented and are live on the site. It's great to see such positive feedback! :psi_happy:

save

path: 0 26218360, hotness: undefined, score: 2, children: 5

m3t00 1 point 5 days ago

shove a link in a menu somewhere. lemmy cross-post search could benefit from a good index. it only seems to know about a few instances.

save

path: 0 26218360 26218897, hotness: undefined, score: 1, children: 4

panos 1 point 5 days ago

Do you mean in Catodon? We don't have a link because you can already search the Catalog's index through Search and Explore in Catodon! It's the button saying "More from the Fediverse" when the server's search results end. But as we add more search options to the site, it might make sense. I'll think about it when the initial suggestions and changes have settled.

save

path: 0 26218360 26218897 26219123, hotness: undefined, score: 1, children: 3

m3t00 2 points 5 days ago

nah talking about lemmy search specifically. probably missing some configs. https://mastodon.social/@dansup has some indexing going on somewhere. prolific dev. i've been using https://fediverse.observer/ occasionally. good to know about search, thanks.

save

path: 0 26218360 26218897 26219123 26219252, hotness: undefined, score: 2, children: 2

panos 2 points 5 days ago

Right - I have no idea how I forgot the Fediverse Observer! I think it's because I have mostly used it for viewing microblogging software, so when I decided to build a threadiverse catalog, my mind only went to indexes specifically for the threadiverse. Adding it now, thanks!

save

path: 0 26218360 26218897 26219123 26219252 26219680, hotness: undefined, score: 2, children: 1

JupiterRowland 2 points a day ago

You may also want to support groups on Hubzilla, (streams) and Forte.

Fair warnings ahead: First of all, there is no central index for them whatsoever. You'll probably have to find a way to crawl them.

Also, crawling anything on (streams) is difficult. It officially and intentionally does not have a name (not even kidding here). It's the only Fediverse server application that doesn't even come with a default server identifier. A Lemmy server always identifies as "lemmy". A (streams) server rarely identifies as "streams", only when the admin explicitly and intentionally makes it identify as "streams". Otherwise, it has identifiers all over the place, usually automatically derived from the server name.

What makes crawling (streams) even harder is that almost all nodeinfo code has been removed from (streams) on purpose. It's also being kept away from all Fediverse stats and listing pages like The Federation, Fediverse Observer, FediDB and FediIndex on purpose.

Forte does have an official name and branding. However, AFAIK Forte still offers the possibility to change the server identifier to something else than "forte". Also, while it does have nodeinfo, Forte's nodeinfo doesn't send any stats AFAIK, only zeroes for everything.

save

path: 0 26276344, hotness: undefined, score: 2, children: 1

panos 2 points a day ago

Thank you! Yes, I have already investigated these options and rejected automatic detection for them for the reasons you mention. Anyone can still suggest specific communities manually, through the suggestion form.

save

path: 0 26276344 26277832, hotness: undefined, score: 2, children: 0

Rules

Rules