Search Engine Spider
What is Search Engine Spider?

Automatic Indexers

Search Engine Spider or Web Crawler or Web Ant or Web Robot or Automatic Indexers, Bots or Worms

A robot program used by the search engines for exploring the Internet in search for new pages to index (analyzing them with their specific algorithm) and also revisiting and indexing pages visited previously.

After visiting these pages (some or in some cases all the pages from a website, depending on some factors) they download specific information from these web pages to the search engine who sent the crawler. All the information that search engines collects from the Internet comes from these robot programs.

Since most spiders are programmed to follow up to three page links from each link in the page they entered in your site (hopefully the index page), the architecture of a site becomes very important. In order to have your site fully indexed, or at least get the most out of these search engine crawlers (depending on your site size) you should create a site map where you can place all the links to your most important pages or to pages that have many links to other pages. Try not to have over 100 links in a single page (Google recommendation). If you need to add over 100 links just link that page to another page in your site and add more links to it.

If your site have thousands of pages and you want all of them indexed (of course) just make the site map with the 100 links. Make sure that each of those links go to a page with 100 links each and so on up to 3 times, even more if you really need to but keep in mind that 3 is the maximum for most search engines. Just do the math and you will figure out how many pages your visiting spider can index. Be creative when doing this. You don’t want your site to be just search engine friendly but visitor friendly as well. Just doing a good linking job is not enough. You should also optimize each page in order to get maximum results.

In some cases you don’t want the spider to visit (and index) some of the pages in your site. Why? Simply because the information there is restricted to either some privileged or paying visitors and you don’t want other people to find and view them. It is fairly easy to find pages website owners don’t want you to see if they first don’t take some precautions including but not limited to keeping the spiders off of them. Also some pages like, although necessary, can not be made search engine friendly and they will only lower your overall site ranking (Frames, Flash, JavaScript and image text) . So how do you keep the spiders from indexing certain pages? Click here to find out different methods used to keep robots from following and indexing pages or parts of them..

Search Engine Spider What is Search Engine Spider?

Search Engine Spider
What is Search Engine Spider?