After the advent of the world wide web or what most people generally just call “The Internet”, it was soon realized that a system was required that was capable of accurately indexing and storing all the information available on the web. This was the dawn of the age of search engines. What search engines basically do, is send out hundreds of spiders or bots that crawl the web, repoting back all the information they find so that it can be catalogued and indexed by the search engine. The next time a user goes to use that particular search engine, it will do it’s best to deliver the most relevant results that match his query. The search engine achieves this by sorting through all the information and records it has based what its spiders and bots have returned to it. This is no mean feat and requires a tremendous amount of processing resources and as time has progressed, extremely intelligent algorithms that are able to weed out the nonsense and provide the search engine user the most accurate information based on his or her request.
In the beginning there were many search engines that people would use. Starting with “Archie”, the first ever search engine. Archie would store its finding on an FTP server in downloadable directory listings. Then in 1991 there came “Veronica and Jughead”, which searched filenames and titles stored in the Gopher index system. in 1993 “Excite” came into existence. It was a search engine developed by six Stanford University undergraduates. in 1994, “Altavista” changed the game completely by introducing unlimited bandwidth, allowing natural search engine queries, advanced searching techniques, the ability for webmasters to add or delete their websites to the index within twenty four hours, searching tips and new features. Also appearing on the scene in 1994 was “Webcrawler”, the first search engine to index entire webpages. The service was so popular that the servers where often overloaded making it difficult to use during the daytime.
In April 1994, Yahoo Directory was created by Jerry Yang and David Filo. Starting off as a collection of credible webpages, the rapid increase in size of this directory required them to introduce a feature that enabled users to search the directory. Yahoo actually only began development of their own search engine in 2002, prior to which they had been outsourcing their search services. Nineteen ninety four was a busy year in the history of the development of search engines. Lycos emerged in July of 1994, providing a catalogue of over searchable 54500 documents, ranking their relevance based on their retrieval frequency. By August that same year they had indexed 395000 documents and by January 1995 over 1.5 million documents had been indexed. In November 1996, Lycos reported having indexed over 60 million documents, more than any other search engine had yet achieved.
Nineteen ninety six was probably the most important year in the history of search engines, with the arrival of both Google, and Inktomi’s Hotbot. Sergey and Larry had begun working on “Backrub” in January 1996, a serach engine that included the value of backlinks or citations. Any mention of a website on another site would be credited as a vote of confidence in that site being linked to. A website’s credibility was based on the number of other websites linking to it and their own credibility. later on that year, in may 1996, Inktomi released their own search engine, dubbed “HotBot”. Inktomi was the first to pioneer the “paid inclusion” model, whereby websites could pay the search engine to have their websites appear first in the results.
Others followed, including MSN from Microsoft which went on to become “Bing”. There was also “Ask Jeeves”, an attempt at launching a natural language search engine where users could simply ask “Jeeves” a question and have answers returned to them. In this model, human editors actually tried to match the search queries. In 2005 AskJeeves was sold and re-branded as Ask.com.
Over time, Google has emerged as the most successful search engine of all time. The word “Google” has become something of a verb in the English language, where people simply say they are going to “Google” something, meaning search for it. Google has been extremely efficient and successful at maintaining its ability to provide users with the best experience by delivering the most relevant results to its users.