This guy throws multiple reasons/conspiracies out there on why the website is really struggling to gain literally any sort of traction. Web is all bots, search engines not promoting competitors and being drowned out by SEO spam, yet he's failing to see the most obvious reason... the reason nearly all websites don't gain traction...
Because it's a bad website. It provides no value to the user. I put in a few search terms and had no relevant search results back. What use is a search engine that can't find what I'm searching for?
Hi, "this guy" here:-) If people come to a site but don't come back then it is reasonable to conclude that "it's a bad website", but as the blog entry put it "without any real users in the first place it is hard to gauge whether people like it or not".
Note also that it isn't intended to be a general purpose search engine, but a niche search engine to try and find some of the fun and interesting content, e.g. relating to hobbies and interests, which used to be at the core of the web but which can be difficult to find anywhere nowadays.
The general purpose search engines search the whole internet, and as a result claim that you can search for anything on the whole internet, even going beyond that to answer questions which aren't on the internet as such, e.g. "What is my IP?" and "What time is it?". However, niche search engines only search specific parts of the internet, and only claim to be able to deliver results relating to their specific topic, e.g. you wouldn't ask the search on a car forum what the weather is today.
I am a search guy and I would like you to succeed. But I don't get it. The name of the site is bland and makes me think you are a white label search service for websites. On the homepage it says "Open source search engine and search as a service for personal and independent websites." but it offers me to reason about why I (or anyone) would want to use it. The content it actually searches is random and of no real particular value as far as I can tell. Also, you are trying to avoid spam sites, but once you reach a certain size that's all you would see is people submitting spam sites. If you blocked people from submitting you would never get all the diamonds in the rough you are trying to expose.
You need to find an actual niche that solves a real problem people have and can understand and orient everything you do to tackling that. Then expand from there.
> general purpose search engines search the whole internet, and as a result claim that you can search for anything on the whole internet, even going beyond that to answer questions which aren't on the internet as such, e.g. "What is my IP?"
I think there are two distinct things here:
1) Searching the whole internet
2) Returning results that aren't necessarily from the Internet, but instead are convenience features of the engine
I understand that you're not trying to replicate things like "What's the weather today", but when I want results about <very specific classic car X>, how can you return meaningful results without searching the whole Internet?
Put another way, if you don't search the whole Internet, the results are going to be limited to only the curated list of sources you do search. This can be useful in its own way - i.e. if you are positioning this as "search this list of curated sources", but also means the site will only be as useful as the curation you provide.
For example, I dabble with Software Defined Radio. If I search your site for "rtlsdr", a very popular package, I get three results. Those results are somewhat interesting, but I know there's a whole world of content out there related to rtlsdr that I'm not seeing here.
So adding a bit to what the parent commenter was saying - if I'm using your site to look for my particular niche, and I only see three results when I know there are many more, I'm not likely to continue using your site to search for rtlsdr.
It then leads me to wonder what I can search for, or if there's much utility to searching at all.
Please take these comments in the spirit they are intended - I think a search engine that helps find things on the "old" web, or just helps me cut through all of the SEO optimized crap is a great idea. It's something I want to use. But I can also understand why someone might try a search and move on.
Just an idea, but maybe providing a way for independent creators to submit their site for indexing (or for an interested user like me to submit a site) would help increase your reach.
Ok, but answering questions like "what time is it?" doesn't subtract from the usefulness of a search engine. Seems like you're saying it makes your search engine better somehow because it can't do the above.
Google is demonstrating this nicely now. It's become almost useless, replacing the query I actually typed with something more popular.
And when that doesn't happen, the results are likely seo'd junk. (The latter is not purely googles fault, it's just that smaller search engines aren't targeted as much).
Try looking up a phone number (by number) in google for a great example of nothing but spam results.
Well, it's worse than that. The whole schtick is that it's only pure, real content by folksy people like us. The top reason to use it on the about page is:
Indexes only user-submitted sites with a moderation layer on top, for a community-based approach to content curation, rather than indexing the entire internet with all of its spam, "search engine optimisation" and "click-bait" content.
So I tried searching [kotlin] and got 123 results ...
Thanks for your feedback. It is just the home page which is moderated before indexing (and reviewed annually). When https://personalsit.es/ was listed it looked legitimate, but agreed the results for that site look infected with spam now. I've found at least one other site today where the home page and blog look genuinely legitimate, but which has a complete spam subdomain, quite possibly the victim of a subdomain takeover attack by spammers. I've delisted both. Unfortunately it isn't an easy task trying to defeat a vast army of well funded spammers in your spare time!
As someone that has a few sites that can get user generated content - I must say that it saddens me that spam stuffing would get the main domain and site delisted - and likely never re-listed.
A couple times a year I get hit with a bunch of spam blogs / user profiles and when I discover and clean them up, I assume that at least google/bing see that the spam-to-real ratio has been fixed and rank it higher again.. but I'm not sure really, especially since google took keywords out of click traffic.
What would be nice is something like the 'site has been hacked page' that I've unfortunately seen a few times for sites - that lets you clean it up and submit a re-check it's clean now button thing.
I've also suggested that google make it so you have to vouch for links which would expose people using the spam stuffing techniques.. kind of the opposite of the disavow tool - but they never read any of my disavow submissions.
Sucks to get spammed, fight spam, and then be penalized for it more ways than one.
One of my older buddypress/wpmu sites I recently turned off blog creation for users because it's just so tiring fighting the spammers - which are only doing what they do because google - meh.
Your problem is that SEO are under no obligation to be truthful with you, and will likely pull bait and switches as far as making accounts if it ever seems like your site will catch on.
Note, I nearly spit my food the first time I was at lunch and someone was talking about SEO a few tables away...oh a decade or so ago now. It's sad it's gotten this bad.
I second this. Don't get me wrong, I applaud the concept and the effort, but this implementation isn't quite there.
I searched for "document management system comparison" since I am currently in the process of selecting one for our legal team at work. Some on-the-ground reports from real users would be hugely valuable. But this is the classic example of where Google utterly fails; document management is a 100 billion industry and there are absolutely no search results which are not SEO, marketing copy, or astroturfed listicles with nearly zero value.
Unfortunately, this website returned even less relevant results. Not a single result pertained to document management at all; instead it returned random matches on words like "system" and "management."
Whoever solves this problem could definitely unseat Google as the go-to search engine for most people. So it's a big prize. But it's also a super hard socio-technical problem, requiring incredibly sophisticated and powerful tech in a highly adversarial environment. However, regrettably, it looks like this attempt hasn't even got the basic search tech down.
- The noncommercial filter gave a few more good results, but it seems like there is not much 'good' content written on this topic
I would definetely not call all Kagi results fantastic, but it does seem to be better than Google. We are trying hard to solve the problem of the nonsense on the web (Kagi founder here).
Is a comparison of document management systems something you expect actually find, as something written by humans? I wouldn't write such an article, I don't know who would.
The only people who seem to be writing these types of comparison articles are spammers.
I typed this reply without checking, but I checked now, and yeah -- if you google "document management system comparison", you get ads for document management systems, and search engine spam. That's hardly helpful.
I guess the use-case just isn't that popular. It's a good website if you want to learn what some devs are up to, but barely anyone cares about that.
Most people use search engines to find answers to their questions and Search My Site just doesn't work like that.
Searching for Astral Codex Ten, a popular, well-written, non-spammy blog which I would expect is indexed...
Returns only results in which _other_ bloggers are referencing ACX. Consider me as one of the datapoints that arrived from HN and likely won't be back, I'm afraid.
Thanks for your feedback. The idea was for people to submit sites they like, and search sites other people have liked. I've submitted Astral Codex Ten, and that site is now indexed for the benefit of others.
Ironically the Kagi search engine is not in the first few results in Google when you search Kagi (at least in Thailand)
And when I did make it to the site, it looks like I have to sign up to use it? I'm not sure putting a locked gate in front of a search engine in 2022 makes sense but okay
The whole concept of kagi is to be a paid service (is still in beta and for now it's free AFAIK), so you pay money instead of having ads or the search engine selling your data, use the service that suits best to your purposes and philosophy.
The concept in 2022 sounds doomed to fail on many fronts. A service that claims to offer privacy but requires identifying payment information. A required email signup so followup sales emails can happen when the service is ready.
Ddg was popular on here until they censored certain websites. Does this search service censor?
Sounds like they are trying to tackle privacy but in reality users of this service will have less privacy.
I searched "best dress shoes reddit" as a test, and just got a random list of websites that had the word "shoes" on the page somewhere, including a Dinosaur Comic from 2008.
So... yeah. Won't exactly be my first choice of search engine in the future.
Its difficult to gauge the quality of the engine itself at this point with so little content in it.
What I can say is that even remotely presenting the system as a general purpose internet search engine like the UI from https://searchmysite.net/ does is going to give people the wrong idea and make them think the system is bad. To start with I'd suggest adding the number of sites indexed to the main search page.
I also think that the https://searchmysite.net/ portal will likely never be a destination. I'd suggest trying to promote it differently, offer a service service for OG internet sites, they opt-in to the service because they want a search widget they can embed on their site that has filter to search just that site or all OG sites. Having website categories would also help so people could search across tech blogs, or aquarium, or bowling sites, etc. Basically the old web ring idea but powered by search instead of just browsing a list.
Since there is a chicken and egg scenario - What you really need are people that think Google sucks that are invested in a niche and want to build a search ring out. The "only sites submitted by verified site owners" restriction needs to go, you want good curation but this is just too restrictive. I also think "downranks results containing adverts" is too restrictive, switch that to "downranks results containing excessive adverts and SEO spam".
Whether or not his site is meeting his goals is his business.
I find this a really interesting post, because I'm also dealing with excessive bot traffic (it's generally about half of my overall), and specifically how to salvage analytics data when there's so much noise. Seeing what other people are doing to combat it helps me, regardless of whether you might think of them as successful or not.
I found a few pro-terrorism sites here. I don't think it's the OPs purpose, but he's being duped by the few users that do look for sites like this where they can add a "curated link" to their ISIS or Hezbollah or Hamas site with a slick facade.
Thanks for your feedback. If you can drop me a note I'll remove those sites - it is against the Terms of Use at https://searchmysite.net/pages/terms/ (not that spammers, terrorists, etc. care about complying with a Terms of Use). I think legitimate looking home pages as a front to other non-legitimate content is a genuine problem this model doesn't solve (also noting that some of those home pages may even be genuinely legitimate but have been hacked e.g. via a subdomain takeover).
Because it's a bad website. It provides no value to the user. I put in a few search terms and had no relevant search results back. What use is a search engine that can't find what I'm searching for?
Maybe if that was improved he may see traction.