The Evolution of Search Engine Technology

The indexing process must also filter out duplicate information and, of course, spam, the term used to describe not only the unwanted junk mail that comes into your e-mail box but sites created or programmed with manipulated text in order to fool people into visiting them. Spammers who wanted to promote advertising or pornography on the Internet learned simple tricks such as including hidden text on their Web sites to increase visits, or hits. The text was invisible either because it was contained within the programming or because words or phrases were written over and over again in letters that were the same color as the screen.

In the early years of the Internet, search engines were not always equipped to block these manipulated sites. Hence, the Web pages that were returned when people searched “cars” were often pornographic or advertising Web sites that had the word “car” written over and over again in letters not visible to the human eye. The gap between what a software program recognizes and understands and what the human being sees and understands is vast. As it was in the early days and continues to be, the tension between people who create Web sites and the people who create search engines remains considerable.