The Evolution of Search Engine Technology

Two approaches to searching the Web developed in the mid-1990s, and both continue to coexist. The first is the directory, sometimes called an Internet index. Human indexers who arrange Web addresses or URLs (uniform resource locators) compile directories by subject. The great advantage, however, is that a genuine human being—generally someone who knows about the subject he or she is compiling sites for—is actually reading and evaluating individual documents. A person, unlike a computer, can scrutinize a document before placing it in an appropriate category. He or she can consider its content and ask, What is this? What does it mean? Is it a site that is worthwhile? In fact, some of today's most popular guides to the Web, such as Yahoo!, began first as directories. Yahoo! was first called Jerry and David's Guide to the World Wide Web. It was compiled by Jerry Yang and David Filo and published on the Web in 1994 when both of them were working on doctorates at Stanford University. (They had to create their own crawler, a program to seek indexes of information on the Web, because there were no browsers. Mosaic, the first browser, didn't appear until they had been working on the collection for several years.) By 1994, Yahoo! had 100,000 unique visitors. It had become the Internet's first major directory.

The biggest problem of an index—any index—is that it cannot be kept up to date because the Web grows faster than anyone can index it and Web sites change too rapidly. Nevertheless, Web directories remain popular. At least 350 million people continue to rely upon Yahoo! as a Web directory and now use it as a portal to the Internet.