The Evolution of Search Engine Technology

According to Rajeev Motwani, professor of computer science and adviser to Google's cofounder Sergey Brin, searching the Internet in the mid-1990s was dicey, at best. Every year, a new search engine or two debuted: Lycos, WebCrawler, Magellan, Infoseek, Excite, HotBot. But when Motwani tested the new search engine Inktomi by typing in the engine's own name, it couldn't find itself. Simple searches performed in any search engine would typically produce page after page of results in no order whatsoever. The creators of the Yahoo! directory, for instance, spent long hours looking for relevant sites when they were compiling their directory, as did many of the earliest search engine pioneers. At the time, one New York City librarian noted that a search for 1950s beat poets, for one example, produced—in no particular order—pages on Allen Ginsberg, gingerbread, and metronomes.

The problems with sorting the information were many. Obtaining poor results was partly a problem of getting people to write their searches more specifically and partly a question of how results were returned. Since search results are often returned in order of relevancy—the number of times your search term appears in a document—being specific about your search could mean the difference between finding what you are searching for or not. And the problems of sorting the information became more and more difficult as the Internet and all of its content grew ever larger.

Yet another way to make your searches more closely match your requests is to use exact phrases in quotations. In this case, the search engine will narrow your query by defining the order of words as you expect them to appear in the returned documents since it knows to discard those results where the words don't appear next to each other in the correct order.