Crawler-Based Search Engines
Crawler-based search engines, such as Google, create their listings automatically. They "crawl" or "spider" the web, then people search through what they have found.
If you change your web pages, crawler-based search engines eventually find these changes, and that can affect how you are listed. Page titles, body copy and other elements all play a role.
Human-Powered Directories
A human-powered directory, such as the Open Directory, depends on humans for its listings. You submit a short description to the directory for your entire site, or editors write one for sites they review. A search looks for matches only in the descriptions submitted.
Changing your web pages has no effect on your listing. Things that are useful for improving a listing with a search engine have nothing to do with improving a listing in a directory. The only exception is that a good site, with good content, might be more likely to get reviewed for free than a poor site.
"Hybrid Search Engines" Or Mixed Results
In the web's early days, it used to be that a search engine either presented crawler-based results or human-powered listings. Today, it extremely common for both types of results to be presented. Usually, a hybrid search engine will favor one type of listings over another. For example, MSN Search is more likely to present human-powered listings from LookSmart. However, it does also present crawler-based results (as provided by Inktomi), especially for more obscure queries.
The Parts Of A Crawler-Based Search Engine
Crawler-based search engines have three major elements. First is the spider, also called the crawler. The spider visits a web page, reads it, and then follows links to other pages within the site. This is what it means when someone refers to a site being "spidered" or "crawled." The spider returns to the site on a regular basis, such as every month or two, to look for changes.
Everything the spider finds goes into the second part of the search engine, the index. The index, sometimes called the catalog, is like a giant book containing a copy of every web page that the spider finds. If a web page changes, then this book is updated with new information.
Sometimes it can take a while for new pages or changes that the spider finds to be added to the index. Thus, a web page may have been "spidered" but not yet "indexed." Until it is indexed -- added to the index -- it is not available to those searching with the search engine.
Search engine software is the third part of a search engine. This is the program that sifts through the millions of pages recorded in the index to find matches to a search and rank them in order of what it believes is most relevant. You can learn more about how search engine software ranks web pages on the aptly-named How Search Engines Rank Web Pages page.
Major Search Engines: The Same, But Different
All crawler-based search engines have the basic parts described above, but there are differences in how these parts are tuned. That is why the same search on different search engines often produces different results. Some of the significant differences between the major crawler-based search engines are summarized on the Search Engine Features Page. Information on this page has been drawn from the help pages of each search engine, along with knowledge gained from articles, reviews, books, independent research, tips from others and additional information received directly from the various search engines.
Now let's look more about how crawler-based search engine rank the listings that they gather.
5 search engines on the internet
Visit AltaVista
Actually don't bother. AltaVista is now an exact replica of Yahoo! search. Did Yahoo! take out AltaVista to minimize competition? Or because it was a better piece of programming? Seeing as AltaVista was once the exclusive search provider or the Yahoo! portal, it was probably the latter! Please note that I didn't realise that these were identical until time of writing!
Visit Lycos
I checked Lycos out, believe it or not it was actually my search engine of choice in 2001/2002. The site has been completely redesigned, and although a redesign was severely needed.... it looks a little like a premium Wordpress template; in fact even I have that translation widget. It certainly doesn't look like a website worth $36m, albeit the monthly traffic and revenues dictate that. I was impressed with the search results though, it actually brought me four times the number of results for my chosen search term than Yahoo!, it also provides a screen shot of the indexed page which really helps with navigation - a feature which Google have recently brought in. I actually enjoyed seeing a different set of results for once, check it out for yourself at Lycos.com.
Visit Excite.com
I decided to visit Excite.com and, frankly, it looks ghastly. In fact, it could be a throwback from 2001. I should be reserving my judgement to the search capabilities, but the default color scheme is off putting. I searched for the same term that I used for Lycos, and it brought me just 8000 results; Lycos found well over 1 million. The results themselves were uninspiring, several of the first page results led to domains which no longer exist. I think that I can safely assume that Excite.com is one which failed solely because it wasn't good enough to match Google and didn't have the right people managing the business. Unlike Lycos, which I felt was a great engine, this has nothing whatsoever on Yahoo! or Google. Visit Excite.com yourself to see whether or not you agree.
Visit ChaCha.com
This was the first time that I had visited ChaCha.com since it offered a human search, I once got banned from the service for pretending to be a lost child in a shopping mall. The less said about that the better. I searched for "How Many Elephants Left In The World" and was presented with an already answered question, as well as 6 questions which were very similar. The engine told me that a 2007 estimate suggested that there are in fact 470,000 to 690,000 elephants left in the world, that is actually far more than I had expected. Albeit, the question had previously been answered from Wikipedia. Another feature worth mentioning is that each search query also returns videos, in the left side bar I could watch any one of three videos about elephants! Now that is cool. I am a little torn with this one, I'm not convinced that this site is sustainable, and it brings you answers which you could simply find yourself in Google or Yahoo!; that said the design is pretty damn sweet and I can see how people could waste a couple of hours on ChaCha.
Visit Ask.com
I quite like Ask.com now that is has re-branded as a question and answer site. Ultimately though, it is still a search engine. Asking the question "How many elephants left in the world" just brings up a list of relevant search results, whilst searching for the same search term as used for Excite and Lycos brought me some decent search results. Ultimately though, I still think that the only search engine on this page which deserves to rival Google and Yahoo! is Lycos. If only it didn't look like an advanced Wordpress site.
