MacHouse's SEO and Web Hosting

History of Search Engines

(1) Search Database Service

There are 3 major players in the search-database service in the U.S as well as in Japan.  Yahoo! is an important player because it's one of the oldest directory service on the world wide web (since February 1994) and is still the most popular.  Google is not as old as Yahoo!.  It started in September 1998 and has now a directory of more than 8 billion websites (as of November, 2004).  MSN Search also started in September 1998.  Its relative importance comes from the facts that Microsoft has incorporated its Internet Explorer into the Windows operating system and that Internet Explorer's default home page is www.msn.com (for North America).  It is also true that MSN search is popular because of Microsoft's internet service.  MSN as an internet service provider has 9 million subscribers in the U.S. and is the 2nd largest although it is much smaller than Time Warner American Online, which has 26.5 million subscribers.  There is no surprise that msn.com is the 2nd most frequently visited site next to yahoo.com in the U.S.
     One of the most powerful and oldest searchable directory entities came into existence in 1994.  It was David File and Jerry Yang of Stanford University Ph.D. candidates in electrical engineering who created a searchable directory to organize a collection of their favorite web sites.   And they founded Yahoo!  As the number of search requests increased, they developed a searchable database with descriptions and categories.  For a long time, Yahoo! manually entered and categorized the sites that were submitted.  Therefore, the number of listings is way smaller than that of Google though they automate some of the tasks now

Yahoo!'s free submission service is over for commercial sites as well as even for personal sites in some countries including Japan.  There have been many changes for commercial sites after Yahoo's acquisition of Inktomi and Overture.  Overture has introduced to Yahoo! pay-per-click advertising service, which allows business owners to bid for the placement of their web site under the sponsored search listing.  Inktomi's Site Match, which now has changed its name to Search Submit Express, accepts submission of a web site for review with a one-time fee of $49 plus a cost-per-click fee.   Upon acceptance, it guarantees a revisit of Inktomi's Slurp (crawler) for every 48 hours.      Digital Equipment Corporation's Alta Vista, which went online in 1995, was the first search directory that used advanced search techniques like natural language inquiries and Boolean operators (AND, OR, NOT).  Alta Vista's natural language inquiry allows the user to enter a sentence to come up with right key words.  Alta Vista also allowed the user to access newsgroups to retrieve articles on the web.  Infoseek also began its directory service in 1995.  It was nothing more than Yahoo! at the beginning.  Then they provided additional services like UPS tracking and news to become a popular online site.

Once being an important player as a listing provider, Looksmart went online in October 1996.  They provided manually-edited listings to such major companies as AltaVista, HotBot and MSN in the late 1990s.  Then its importance as a mayor directory provider quickly diminished when its tie with MSN ended in late 1993.  Looksmart has also lost its status because they made numerous changes in submission service fees after the beginning of 2000 and then introduced pay-per-click rates.
     NorthernLight.com went online in 1997 with 30 employees.  They once had one of the largest set of search directories that offered full text documents covering magazines, journals, books and newswires.  Their diverse online business library is called the Special Collection.  Northernlight ended their free publicly accessible web search in January, 2002.
     NorthernLight, which is now part of Northern Light Technology, Inc., offers several members-only up-to-date categorized search services focusing on business and industry.
     There isn't much to say about MSN Search. It was originally launched in September 1998 for Microsoft's Internet Explorer.  It is used for the default home page of Internet Explorer.
     A public search directory service, Open Directory Project (ODP), also started in 1998. According to ODP, its human-edited directory is the largest of all.  According to ODP, as of July 2005, they have more than 4 million listings that are edited by about 70,000 people.  It's listings are distributed to hundreds of companies including AOL Search, Gigablast and Google.  ODP is actually owned by AOL/Time Warner.  Its original spirit is such that the submission of web sites and the use of the directory data are free of charge.

(2) Search Engines: Crawlers, Robots, Spiders 

In 1993, Matthew Gray, a student at MIT, wrote a perl script called the World Wide Web Wanderer to measure the size (the number of servers) and the growth of the world wide web.  Then it was used to capture URLs.  This index of URLs was called the Wandex.   Since it caused the network-wide degradation of performance (It sometimes accessed the same site hundreds of times a day.), Matthew's Gray's Wanderer was quite a controversial project.   Anyhow, the Wandex was the first robot on the web.      What also appeared in 1993 following the Wandex is Martijn Koster's ALIWEB.  ALIWEB also indexed web sites. It actually allowed webmasters of participating sites to post their own page description.  And this process didn't cause the network slowdown since it didn't require large bandwidth.   The problem of this indexing process is that the system required the submission of a special indexing file, which many people did not know of.

Furthermore, once a popular search engine Excite has its roots in the development of search software.  In early 1993 six undergraduate students at Stanford University started a project called Architext.  They used statistical analysis to effectively find world relationships for Internet searches.  And they released search software, Architext, for webmasters to use on their own websites.      In 1994, a University of Washington student Brian Pinkerton developed desktop software called WebCrawler.   While other bots stored titles, URLs and first 100 words or so, WebCrawler indexed entire pages.  And users could search the full text of each document.  WebCrawler became so popular that the network system at the University of Washington was devastated during the daytime hours.  An early success of Webcrawler brought Lycos such search engines as Lycos, Infoseek and OpenText within 1 year.      Lycos came out of a research project developed at Carnegie Mellon University in 1994.  Dr. Michael Mauldin's original research was to calculate the size of the web using a spider robot called 'wolf spider,' which is Lycosidae in Latin.  The wolf spider walked from site to site through page links.  And in no time Lycos' size of catalog went unimaginable at the time. By November 1996, Lycos has indexed more than 60 million documents.      In late 1996, Infoseek introduced its full indexing engine called Ultra, which collected 25 million urls.  And Infoseek started providing search results to Netscape in December 1995.      There was a problem by mid-1995 in pulling search results.  Different search engines came up with different sets of results.  So, in 1995, a Master's student Eric Selburg and associate professor of computer science at the University of Washington developed MetaCrawler.  This search engine accessed Lycos, AltaVista, Yahoo!, Excite, WebCrawler and Infoseek simultaneously to come up with the best results.  The way it worked is such that it gathered results from various search engines onto one page and then reformat them.  Savvy Search, which was developed at Colorado State University, is similar to MetaCrawler.  Savvy Search, which is faster but less reliable than MetaCrawler, accesses up to 20 search engines at a time.      In 1996, Inktomi introduced an important spider HotBot to the web.  HotBot was originally developed at UC Berkeley by assistant professor of computer science Eric Brewer and a Ph.D. candidate Paul Gauthier. It was a powerful search engine at the time because it was said to index 10 million documents per day.      Although it is a late comer, Google is now one of the most powerful and innovative search engines.   Two Ph.D. candidates in computer science at Stanford University Larry Page and Sergey Brin had a collaborative research project in the mid 90s.  And they developed a search engine called BackRub.  BackRub uses a unique idea of algorithm at the time that analyzes the back links pointing to the original website.  Today, a more advanced version of this inbound link calculation technique is called PageRank. In a sense, PageRank calculates the number of votes that are cast to the original website, taking into account, also, the quality of the casters.

(3) Timeline 

References 

A History of Search Engines by John Wiley & Sons, Inc.
History of Search Engines & Directories by CommerceFriends.com
Search Engine Players: A Brief History by Search Engine World