We have submitted a PR to the Open WebUI repo, which would enable Mojeek: https://github.com/open-webui/open-webui/discussions/6588
ColinHayhurst
joined 11 months ago
Where is your evidence for that? It used to be Bing and Yandex, but now it's just Bing. They use other non search engine APIs and do a small amount of crawling AFAIK. Details of who uses what here: https://seirdy.one/posts/2021/03/10/search-engines-with-own-indexes/
You should put these entries into your robots.txt file.
To block the Google search crawler use for all of your site:
User-agent: Googlebot
Disallow: /
To block the Google AI crawler use:
User-agent: Google-Advanced
Disallow: /
Yes, it was. Matt Wells closed it down just over one year ago.
yep, in footer "© 2024 Infospace Holdings LLC, A System1 Company"
We'd love to build a distributed search engine, but it would be too slow I think. When you send us a query we go and search 8 billion+ pages, and bring back the top 10, 20....up to 1,000 results. For a good service we need to do that in 200ms, and thus one needs to centralise the index. It took years, several iterations and our carefully designed algos & architecture to make something so fast. No doubt Google, Bing, Yandex & Baidu went through similar hoops. Maybe, I'm wrong and/or someone can make it work with our API.
view more: next ›
Are the "thousands of companies" hiding their identity?