Before Google: A history of search

When I was first on the Internet, we didn't have Google, or even the Web, but we still managed to search the Internet. Here are the tools we once used...and liked.

When I started using the Internet in the 1970s, it didn't look anything like it does today, and our search tools were primitive. But when all you have is stone knives and bear skins, you make do.

Before I ever turned my hand to writing for a living, I put myself through graduate school by doing research on the very first online database systems: NASA RECON Dialog, now ProQuest, and OCLC. These systems, which are still around, are part of what's called the Matrix—and, no, I don't mean the movies. The Matrix, as defined by Carl Malamud, is the superset of all interconnected networks. Today, you can get to ProQuest and OCLC over the Internet, but you'll find yourself blocked from getting very deep into them without permission.

As for the pre-Web Internet itself, at first it didn't have search tools. It wasn't until the late 1980s that the Internet became searchable. When I started using it, we had to go through FTP file directories screen by screen and hope that the file we wanted was in there somewhere.

Archie, Veronica, and other early search engines

The first major search advance was Archie, which beginning in 1990 made it possible to search through a site's file directories. Archie was painful to use, but compared with what we had been dealing with, it was wonderful. Archie was quickly followed by Veronica, a service from the University of Nevada System Computing Services that tried to provide Archie-style searches for plain text files.

Get the report from 451 Research.

Datacenter Modernization: Trends and Challenges

An even bigger advance was Gopher, which made it possible to search through online databases and text files. With Archie, you really had to have a clue that a file was somewhere on a given site. With Gopher, you could simply search and let the server worry about finding which site had the information you wanted.

While Gopher was being built in 1991, the Web was also being created. By 1993, just as Gopher reached its maturity, I thought the first real Internet search engine, WAIS (Wide Area Information System), was going to be more important than the Web! I was putting the cart before the horse. WAIS, like Archie, Veronica, and Gopher, exist now only as Internet historical trivia.

In a way, though, I was on to something. Yes, we use the Web for everything, but it's the search engines that make it useful. After all, we say, "We Google for information," not "We webbed for the answer."

The Web quickly knocked out those earlier Internet search programs as well as online services like Prodigy, CompuServe, and GEnie. Now, only bits and pieces of them remain (such as CompuServe forums), and for the most part, they're really only interesting to digital archaeologists.

Fortunately for early users, the first web designers set to work creating search engines. Indeed the first of them, the WWW Virtual Library, was created by the Web's creator, Tim Berners-Lee. The Virtual Library, and its most well-known clone, 1994's Yahoo, weren't really search engines. They were human-assembled catalogs of useful web links. You could (and did) submit your own webpages to Yahoo, for instance, and suggest in which category it should be included. (Why yes, search engine optimization was very different back then.)

These directories still exist. Indeed, sites that allow users to vote on which specific webpages are interesting, such as Reddit, are still popular. Most of the early ones, such as EINet Galaxy (1994) and Open Directory Project (1998), have either died off or are little used. Others, such as Yahoo, switched to a search engine model.

In search of search

And what is that? The very first search engines—JumpStation, the World Wide Web Worm, and the Repository-Based Software Engineering (RBSE) spider—used automatic programs, called robots or spiders, to request webpages and then report what they found to a database. Those first spiders weren't that useful. It took Archie-Like Indexing of the Web (ALIWEB—it's completely dead now) and Excite in 1993 before we had search engines that most of you would recognize as useful tools.

"Useful" is being kind, though. These first search engines were useful only if you knew the exact name of the website you were searching for. Searching inside webpages would have to wait for 1994's WebCrawler. WebCrawler is now largely forgotten, but in its first year it was so popular that it became a victim of its own success; the site was so slammed that it often couldn't be used.

That gave Lycos its chance. By pouring more servers into this project, Lycos was almost certainly the first search engine to have full page search for more than a million pages. That's nothing by today's standards, but it was remarkable in mid-1994. Lycos was also the first search engine to introduce proximity searching.

The mid-90s was a hotbed of new web search engines. InfoSeek, whose only long-lasting claim to fame was somehow convincing Netscape to make it the Netscape browser default search engine, came, went, and unlike the others, left no real trace behind. Inktomi, which appeared in 1996, briefly enjoyed a burst of popularity. Eventually, in 2003, it was bought out by Yahoo.

The most significant web search engines to arrive were AltaVista, my personal favorite in those days, and Ask Jeeves, now Ask.com. AltaVista was the first really fast search engine that also covered much of the Web. It also gave users the first successful Boolean search options. So, for example, you could search for "New York baseball history NOT Yankees" and get results about the New York Giants and Dodgers, but not the New York Yankees. Ask's claim to fame was its attempt to support search by natural language. So you could ask it, "Tell me about New York baseball history, but I don't want to hear about the Yankees." [Because, really, who does? –Ed.] Whether you'd get useful results is another matter, but Ask Jeeves did better with this question style than its competition.

How the 800-pound gorilla grew so big

So why haven't you heard of most of these? Well, each had its own problems. AltaVista, for example, shared in the woes of its parent companies, Digital Equipment Corp. (DEC) and then Compaq. Yahoo, which ended up with the ownership of AltaVista, finally killed off this one great web search engine in 2010.

But what really happened to all of them was Google. Google simply did a much better job than its rivals of making search easy and comprehensive. The key to Google's overwhelming success was PageRank. With PageRank, Google rates the relevancy of webpages to queries, based not only on whether the pages contain the search terms (the technique used by all search engines), but also by how many relevant pages link to it.

Sounds simple. Well, the basic concept is simple. It was a groundbreaking advance at the time. Perfecting Google Search and PageRank continues to this day.

Google has never rested on its laurels, either; no company has managed to give Google any serious competition in more than 10 years. Today, Google is used for more than 64 percent of all searches. In comparison, Bing has 23 percent of the market, while Yahoo continues to decline with only 12 percent of searches. Ask is barely hanging on with 1.4 percent of the market.

Had AltaVista managed to avoid its parent companies' troubles, it might have been a different story. None of the other BG (before Google) web engines had what it took to compete with Google's genius. Today, as a result, we really do live in AG (after Google) days. I doubt that will change anytime soon. Google certainly continues to work hard to make sure that will be the case. 

This article/content was written by the individual writer identified and does not necessarily reflect the view of Hewlett Packard Enterprise Company.