It might be a philosophical thing. One APIs became popular, scraping started to be viewed as breaking the rules. I think this has discouraged many from writing search engines. Crawlers are just scrapers on a big scale.
I know I'd like to see more serious attempts at vertical search engines. I wonder if getting backlash, in the form of people complaining to hosting companies, would be the hardest part of writing a search engine for ruby blog posts that understands regular expressions. I could certainly use such a tool.
