Like rust, they never sleep. Tireless, purposeful, never pausing for a moment, thousands of spider robots creep silently through the computers of the world, hunting for information wherever it roosts. Surely this random search of the World Wide Web that goes on at every second of every hour, every day, is the most romantic of all the invisible processes that give life to the digital age. The spider robot may be the symbolic invention of the 1990s, what photography was to the 1830s or rock 'n' roll to the 1950s. The fact that it's a pure electronic metaphor, that no one has ever seen one or ever will, only heightens the mystique.
Last fall, an article in Newsweek said that a search engine is "simply a software program that crawls through the Web and makes an inventory of what's out there." Simply? The word has seldom been so grossly misapplied. The search engine has established a new era in that crucial human activity, the organization of knowledge.
The World Wide Web is a universe of data, and, just like the actual universe, steadily expands. In response, a small army of first-class brains has taken a decade to invent ways to navigate it, and the process is still underway. Nothing about search engines, neither their creation nor their results, is simple. Their mostly anonymous designers are the equivalents in our time of Melvil Dewey (1851-1931), the American librarian whose Dewey Decimal Classification and Relative Index, an attempt to tag and retrieve the contents of an otherwise unknowable ocean of books, was the great-grandfather of the search engine.
Of course, the spider robots themselves are simple-minded. If they come across Buckingham Palace, the Buckingham Diner in Memphis, and the Palace Theatre in Calcutta, they usually can't tell the difference; all three may end up on the same list. But the spiders are relentless. Encountering a Web page, they frisk it for information, tag it, and file its location. Then they tell us it's there and allow us, with a click, to learn all that it has to say.
Early encounters with search engines can leave you feeling both exhilarated and intimidated, like the first visit to a good library. AltaVista, Excite, Lycos, Northern Light and many others, with their armies of trained spiders, offer a startling array of data. And, since the mid-1990s, meta-search engines have appeared, rummaging through other search engines to find the best material.
The gaudiest is Google, which assembles more sites than any other, assembles them faster, and can't help bragging about its speed. If you enter the word "Matisse" and press the button, Google instantly pulls together 21,500 sites about him, smugly announcing that "Search took 0.04 seconds." Who will look at all 21,500 sites? No one, but Google has shrewdly arranged them according to how many other sites link to them.
Despite some earlier experiments, I didn't grasp the true value of search engines until three years ago, when I was writing about Son of Sam laws designed to keep criminals from profiting from crimes through books and movies. I needed to know about the origins and implications of that idea.
In the past I would have spent hours reading in a library, but in 1997 I went to www.dogpile.com, which connects you to 14 search engines. I didn't even know the term under which this subject would be filed, so, to start somewhere, I entered "Son of Sam Law." To my surprise, that was enough. Fifteen minutes later I was printing out two good legal papers on the subject, one favouring the laws and one opposed. The spiders had reduced half a day's research to a quarter of an hour.
My worry about citing the right term demonstrated my ignorance. Search engines are forgiving. Their designers recognize that humans are vague and imprecise, and so is language. Some people call it "oil," some say "gas" and some say "petroleum" -- in the trade this is recognized as "the verbal disagreement problem." The tags applied by the spider robots are usually broad enough to accommodate those differences.
But some people still find search engines frustrating, and Ask Jeeves has been invented to solve that problem. It deals in questions and answers. You ask a question of Jeeves (who, of course, is also an electronic metaphor) in ordinary language, and he races through a multitude of sites to find the answer. But Jeeves is the cyberspace equivalent of the consultant who can't admit he lacks an answer. Even if you ask, "What is the meaning of life?" he sends you to online philosophy courses and then informs you that some clown has set up a home page titled The Meaning of Life.
The other night I asked Jeeves: Who invented the search engine? I figured he ought to know, but the results were about as useful as the answers about life's meaning. Jeeves first pumped out a long list of inventions, none of them search engines (the word "invented" threw him right off track). Then he offered me career advice about becoming a builder of search engines. After that, he delivered a directory of 2,000 search engines organized into categories, such as music and pets. Jeeves was avoiding the question.
But on several other sites, notably SearchEngineWatch.com, I found the answer. The first search engine, named Archie, was created in 1990 by Alan Emtage at McGill University. It was slow, it dealt with only a limited number of mostly educational institutions and it had no World Wide Web to draw on; but it fetched sites, listed them and told others how to reach them. Within three years, search engines were springing up at universities all over the United States, imitating this strikingly original Montreal idea.
Search engines join isolated pockets of fact into something that begins distantly to resemble an ordered information network. They can never catch up with the growth of the Web and prepare a complete catalogue, but they deliver vast amounts of data to every office and house equipped with a computer and a modem. It would be foolish for anyone with a sense of curiosity to see this as less than a miracle.