Is there a way to search through posts across all instances/communities?

Mon, 17 Jul 2023 10:59:38 +0200

MoonHawk@lemmy.world

Hi, I’m relatively new here. Like probably a lot of people, I appended the word “reddit” to most of my searches to get discussions around my interests/news or to find “curated” things. I noticed that it does not work as great with “Lemmy” for example, as the user base is more spread out and smaller in general.

I’ve looked for a search engine for the Lemmy or fediverse with no luck. There exists some to find communities but not to return posts it seems.
If it does not exists, is this even something that can be built? Would one need to build a crawler or use an API? I would be very interested to contribute to such a project.

show all 12 comments

Mon, 17 Jul 2023 11:05:13 +0200 zuletzt bearbeitet: Mon, 17 Jul 2023 11:06:55 +0200

Remi Dore

chimay@friendica.me

@MoonHawk

some useful links :

search-lemmy
lemmyverse

for the whole fediverse :

fedi-search

Mon, 17 Jul 2023 11:05:40 +0200 zuletzt bearbeitet: Mon, 17 Jul 2023 11:07:01 +0200

marsara9

marsara9@lemmy.world

Check out my post history.

But www.search-lemmy.com. It has a few bugs but it should work for you. Especially if you set your home instance to something large like Lemmy.world.

Edit: if you want to help contribute: www.github.com/marsara9/lemmy-search

Mon, 17 Jul 2023 13:01:49 +0200

MoonHawk

MoonHawk@lemmy.world

Oh wow thanks. That’s exactly what I was looking for. I hope this gets used by all the community!

Mon, 17 Jul 2023 17:11:27 +0200

Jupiter Rowland

jupiter_rowland@hub.netzgemeinde.eu

Searching the whole Fediverse, literally all of it, 100%, is technically impossible or at least very hard to implement, and if implemented, it'd eat up lots of CPU power and network bandwidth.

It's simply next to impossible for any instance of any Fediverse project, also for any centralised or decentralised dedicated search engine, to know all instances and all content on it without all instances actively pushing their existence, their status and all their content to the search engine in real-time.

A search engine that literally covers all of the Fediverse with no exception has to even know about brand-new instances that have just been started a split-second ago. An instance that's so new doesn't even have any connections into the Fediverse yet, probably no content and only one account, the admin account. (Replace "account" with "channel" on Hubzilla and (streams).)

So if someone spins up a new instance of whatever project, that search feature has to know about that instance immediately before the instance even connects with anything. That is, I'm not sure when that search feature is expected to know about a new Hubzilla hub since ActivityPub is optional per hub and per channel and AFAIK off by default for both: Shall the search feature already know when ActivityPub is still off, and nothing in the Fediverse that isn't Hubzilla or (streams) can connect to it anyway, or shall it only learn about the instance the second that the hub admin turns ActivityPub on?

And when the admin of a new instance puts out a test post to see if it runs as desired, and the instance still isn't connected to any other instance, the search feature would immediately know that test post so you can find it if it's that what you're looking for.

Mind you, Google doesn't know everything on the Internet either.

Mon, 17 Jul 2023 18:13:15 +0200

JayPalm

JayPalm@lemmy.world

Do we know if/that Lemmy posts are getting indexed by google? I haven’t had much luck throwing “lemmy” into my google searches but presumably if we do it should start getting more traffic and increase rank?

Mon, 17 Jul 2023 18:18:06 +0200

null

null@zerobytes.monster

They do getting indexed ;)

Mon, 17 Jul 2023 18:42:59 +0200

null

null@zerobytes.monster

www.google.com/search?q=site:lemmy.world

Mon, 17 Jul 2023 20:06:07 +0200

JayPalm

JayPalm@lemmy.world

Nice.

Mon, 17 Jul 2023 20:07:06 +0200

JayPalm

JayPalm@lemmy.world

This obviously isn’t very good though cause it will only index posts on lemmy.world, not other indexes, no?

Mon, 17 Jul 2023 20:15:30 +0200

DreamButt

tdawg@lemmy.world

Sure this is technically true, but it doesn’t really fix the human need to find things. It would be better if some grouping of Fediverse instances came together under a common banner and agreed to certain protocols that helped make things like mass-indexing easier. This would enable a better frontend experience for people trying to find good content. In fact I think building more protocols on top of the existing one would be exactly inline with the philosophical underpinnings of the Fediverse

Mon, 17 Jul 2023 20:15:49 +0200

DreamButt

tdawg@lemmy.world

Yoink

Mon, 17 Jul 2023 20:58:38 +0200

Spzi

Spzi@lemm.ee

A search engine that literally covers all of the Fediverse with no exception has to even know about brand-new instances that have just been started a split-second ago. An instance that’s so new doesn’t even have any connections into the Fediverse yet, probably no content and only one account, the admin account. (Replace “account” with “channel” on Hubzilla and (streams).)

So if someone spins up a new instance of whatever project, that search feature has to know about that instance immediately before the instance even connects with anything.

Yes, but who would want a search engine to specifically cover emtpy servers with half a nanosecond lifetime? For all practical intents and purposes, people search for content, which already excludes these theoretical edge cases. More realistically, people will search for quality content, which implies some engagement happened and some upvotes accumulated. There is no value in discovering servers before users discovered them, on the contrary.

If you really care about new and empty servers, you’re rather looking for a fediverse monitoring tool than a search engine. And even for those, it’s questionable what the value of those entries would be. I would prefer if they are filtered out to not bloat the numbers.