Blog

Posted by admin on January 20th, 2009

A Spider (also known as web crawlers, web bots, web scutters, ants, indexers, bots and worms) is a program or script that browses the web in a process called spidering or web crawling.

Spiders are mainly used by search engines and their job is to create and store a copy of the website for indexing by the search engine to provide the most relevant results quickly. It is also the spidering process that is responsible for gathering updated information and making it available in search engine results.

Crawlers can be used to schedule maintenance such as checking links are still active and for errors in html. One of the bad things about spiders is that they can be used for gathering information from websites such as email addresses which are used by spammers.

Web design is extremely important for when a crawler is visiting your site. A crawler has a list of url’s to visit called a crawl frontier. The crawler visits each url on the list looking for links to other pages and adds them to the list to visit the pages later. If the website design does not include links to other pages within the site the crawler will not know to visit the other pages on the site! urls on the list are crawled at regular points to check for new information which is used for search engine optimisation rankings.

Many people will have noticed that updates made on websites do not appear in search engines immediately, this is due to the volume of information on the world wide web. As spiders copy the information from a page they are limited to the amount of information they are able to process, this means that by the time information is added into the search engine there may be lots of new content on the site or pages that have been removed completely! Unfortunately there is nothing anyone can do about the speed information appears in the search engines and no matter how good the search engine promotion team are, the results won’t happen over night!

Also due to the amount of information and the speed it is being added, only a small section of pages are indexed. A study of search engines showed that surprisingly, no search engine indexes more than 16% of the information on the world wide web – this makes a search engine campaign all the more important to ensure your pages are included in those results! If you think of the last time you Googled something and the millions of results displayed, now just imagine how many results are missing that are included in the unindexed 84%! These sites are in the deep or invisible web and are unindexed because the regular crawlers can’t find them!

To make sure your website doesn’t suffer the same fate, why not give us a call on 0116 2758456 and find out how our SEMPO Platinum Accredited Search Engine Promotion team can help you make sure the spiders are happily crawling around your web home!

Dani
Studio 2 Online
Search Engine Promotion Leicester