actually awesome and fast search engine (depending on which instance you use) with no trashy AI and ADs results also great for privacy, if you don’t know which instance to use go to https://searx.space/ and choose an instance closest to you

  • notfromhere@lemmy.ml
    link
    fedilink
    arrow-up
    2
    arrow-down
    1
    ·
    8 hours ago

    I seem to recall creative ways to index things without robots, e.g. browser addon that users opt into to send pages and such, essentially crowdsourcing the indexing. Anyways good to see you’re taking the high road!

    • Mojeek Search Engine@lemmy.ml
      link
      fedilink
      arrow-up
      2
      ·
      1 hour ago

      our preference is always to find out why the block is happening and try to convince people it should be otherwise; widespread abuse of robots.txt does no-one any good, having been crawling and indexing for so long it’s a standard that we understand and are quite fond of

      we can see some of the perils and pitfalls of it too, but web builders need to be given some tools and assurances that those tools will work for them

      • notfromhere@lemmy.ml
        link
        fedilink
        arrow-up
        1
        ·
        1 hour ago

        That makes sense. One thing I’ve noticed with Mojeek search results compared to Google is that I do not encounter the “old web” any more on Mojeek than on Google. Are you not crawling/indexing web 1.0 blogs and sites at all?