In Search of the Ideal Search

Date:   Monday , March 02, 2009

Wouldn't it be nice if Google understood the meaning of your phrase rather than just the words that are in the phrase? We've made a lot of discoveries in that area that are going to roll out in the next little while," said Eric Schimdt, CEO of Google at the Earnings call last month. Saying this, he was actually hinting at the next big wave in search engine technology – Semantic Search, which brings more intelligence to search. This gives a small view on the continuous quest of search engines towards this query: What will constitute an ideal search engine? Undoubtedly the most powerful application of the internet, search engines today can not only return content results but image, news, and even video results. According to a recent report, Web users spend a total of 13 million hours per month interacting with Google alone.

A look at search engines that existed before the turn of this century, like Alta Vista, Hotbot, and Infoseek show that they were doing a pretty good job with their search, and people were relatively happy to use them. Then came Google, which figured out that all that the people cared about were the speed, simplicity, and the best possible way to search. By concentrating on the end result - better search - and pioneering new ways to get there, Google changed the whole landscape and created a brand name that's now both a noun and a verb.

Today, with multiple players in the market, it is an open secret that not only the large players but also small search engines are working towards improvements to core search technology and deeper advancements in different search areas to deliver a better search experience to the consumers and advertisers. Given that Web pages are changing constantly, search engines also need to update their index periodically in order to keep up with the ever-evolving Web. An obsolete index leads to irrelevant or 'broken' search results, wasting users’ time and causing frustration.

As Prabhakaran Raghavan, Head of Yahoo Research puts it, "We are redefining the notions of accuracy and relevance. Users are no longer satisfied with running searches and seeing a list of documents in response. Users want to complete tasks; our goal as a search engine is to enable them to complete their tasks, rather than read documents, assimilate information, and finally decide on actions." A user typing 'pizza san francisco' is not interested in a ranked list of documents containing these words. That user would best be served by a ranked list of pizza restaurants in San Francisco, each of which is accompanied by all germane information including location, open hours, menu, user ratings from around the Web, etc.

Know Your Customer
This clichéd statement has a great importance in the context of search engines. Due to rapid advance in technology and Web proliferation, the customer today is seeking more reliability, accuracy, and efficiency. Nearly 20 percent of the customer challenges came from the long tail of the Web, indicating the need for broader coverage to ensure that the right results can be returned for the highest percentage of queries. One key insight that really needs to be noticed is searches don’t occur in isolation – they are often part of a longer task. That is, searchers come to a search engine with something in mind, do an initial query, click on multiple results, perform multiple follow-up searches, and then frequently come back in the following days or weeks to return to the same topic.

That's not to say that there is no longer a straightforward, one-query, one-click activity. But the trend shows a large number of navigational searches seeking a specific site. While navigational queries are common and they account for a fair percentage of the search volume, they represent a relatively small amount of customer time spent on a task. The maximum time spent on search tasks is actually spent doing the kind of research that requires visiting multiple pages – either to find exactly the right one, or because the task itself requires going to several places.

New Ways of Search
However, even after trying to analyze the behavior of the customer with the best of intelligent tools and back end technologies, every search engine still finds more room for improvisation with regard to matching the query and the intent. Like Anurag Dod, cofounder of puts it, "Till now we have not cracked the best technology for addressing maximum customer satisfaction." But this has led to a lot of experimentation in terms of search. Now we have personalized search, localized search, vertical search, and even natural language search. All these specialized services are focused on one single task – a better search experience to the customer.

The search engines started focusing on personalized search in earnest around 2007. Any user willing to sign up for an account soon found that the search engine started to learn from his or her search behavior, identifying themes, topics and websites of particular interest. Relevant pages are given a boost in his or her search engine results.

Natural language search is yet another attempt in giving a better search experience. This type of search is the easiest to understand, but many databases don't offer it as a function. A natural language search is a search using regular spoken language, such as English. Using this type of search, you can ask the database a question or you can type in a sentence that describes the information you are looking for. The Internet search service offers natural language searching.

A lot of local search engines have also sprouted in countries across the world for providing localized information. Baidu, a local search engine in China, has more market share than Google in that country. Similarly, in India pioneered local search service, providing India based content. Dod says, "In India, local search is in trend. People are looking for India based content. is focused on providing better search results to Indian consumers, by leveraging proprietary algorithms and data in the Indian context."

"40 percent of searches in India fall into the categories of entertainment, shopping, health, and local information search," says Ravi Datanwala, Group Manager, Microsoft. He adds, "Unlike the popular notion that low bandwidth will affect the entertainment content, we find that Indian consumers are more patient when it comes to downloads and bufferings." Hence all the players have made real investments in the vertical search technology, which enables them to build rich vertical experiences that update on the fly.

Beyond the PageRank Technology
The search engines not only tried to improve search in terms of adding new features and bringing new methods, they worked on the core technology too, involving users and developers. The search engines had already found other ways of identifying high quality pages that went beyond the 'PageRank' methodology developed by Google in the 1990s.

Around 2006-2007 Yahoo! started its effort to make use of information gathered from its two bookmarking services websites, MyWeb and These sites, favored by users of these services, began to see a slight improvement in rankings. Google bought Magnolia and Simpy in 2007 (both were immediately merged with Google Bookmarks), and Ask acquired Bluedot in 2008. Now all the big three had efficient bookmarking services that could deliver input to search algorithms.

Semantic Web technology is yet another buzzword. However, it is both overused and poorly understood. It's safer to say that the plan is to transform the Web of pages (the traditional view of search engines) into a Web of objects, where an object can be a restaurant, a person, place, and so on. Along with this, they intend to model (as best as we can infer from the Web and other sources) the relationships between these objects with the goal of presenting to the user the objects most relevant to their task. This redefinition of relevance demands a new technology and science that surely gets at the hidden semantics of Web pages. This technology, when properly cracked could become a milestone in the history of search engines.

In order to encourage innovation in search technology, Yahoo! recently came up with a unique program. It opened up its search infrastructure for others to innovate on – this is their BOSS (Build your Own Search Service) program. Any startup (and Yahoo! also has academic researchers participating) can access layers deep in its search technology to deliver and market-test their own version of search experience. As an example, a search experience similar to that of Cuil was built on BOSS in a matter of hours. "Through BOSS we’re removing the burden of the hard engine work and letting others try their hand at search innovation," explains Raghavan.

As newer experiments and innovations improve the core search experience and infrastructure, search engines vie with each other to keep pace with customer demand. These new experiments are opening new and novel vistas for search like 'mobile search', which is evolving as a new branch with all the big and small players. An early example came from Microsoft that has come up with a mobile search client, a software plus service in the palm of the hand. Speech based search combines powerful speech recognition software on the mobile phone with the live search service over the Internet.

"There is plenty of room for innovation," says Datanwala, and as he puts it "The search to improve search will go on…" Each incremental piece of content, improved categorization and better organization will take the search service providers closer to their goal.