Importance of robots.txt file

First some back ground on search engine robots

Web indexing robots are used by many search engines such as Google, Inktomi, AltaVista and others. These web indexing robots are also known as spiders. These spiders/robots are the tools used by engines to harvest data for their search engines. When you submit your website to the engines, you are effectively asking the search engines to send their web indexing robot to your website so that it can be crawled and added to their database.





So why do i need a robots.txt file?

Web-Indexing Robots can be controlled as to which part of your site they index by installing a simple text file called robots.txt in the root path of the server with explicit instructions on what the spider is and is not permitted to index on your website.
You can define which paths are off limits for spiders to visit an block off such . This is useful for such things as large directories of information, personal information, and parts of the website containing large amounts of recursive links, among others.
Now it is possible to include robots.txt indexing information directly in your meta tag and in some cases this is preferable if only one page needs to be controlled. You can use a meta tag like this meta name="robots" content="INDEX,FOLLOW> to tell the robot it is ok to index this page and follow links it finds on this page. However, if you have whole directories and multiple pages you want to control the indexing of then you need a robots.txt file to ease the burden of managing this task.




How accurate does my robots.txt tag have to be?

You need the correct path of the files or directories that reflect the web viewable path of the server.
Example: many servers use htdocs as the web root, but the ftp root will be different. Your robots.txt tag should not include the htdocs directory in front of the file/directory because the htdocs folder is not viewable on the web...the files in the htdocs are what need to be listed if you whish to control the spiders indexing of them.

Do I have to have a robots.txt file in order to have search engines index my site?

The short answer is no! A web indexing robot will crawl your site unless told not to. However lets go a little deeper than that. A good web indexing robot such as Googlebot or Slurp (Inktomi) are considered well behaved web spiders and will attempt to find your robots.txt file before it indexes your site. As well good robots will look at your meta tags file and check for the The advanced way in stopping malicious spiders that ignore or disobey your robots.txt file is to look at blocking users agents at the server level and even so far as blocking IP's etc where possible. A user agent is a signature that is attached to the robots (provided they added one) which can be used to identify the robot. When a page is requested from your web server, software such as IIS (windows server) or Apache (Linux/Unix) will store this user agent information in your log files which you can review and react accordingly.

What is SEO?

SEO Stands for search engine optimization basically it is a process or a technique through which our client find us very easily. It is an indirect marketing process to makes our business or website more popular so that we get more business. Over all motive of SEO to get more income. Through SEO we improve ranking of website in different search engine and page rank also.
Thus we can say
“More Visit More Traffic More Business More income “.

SEO - Link Building for Beginners

Search Engine Optimisation has two distinct area's the first being On-page optimisation and the second, off-page optimisation. On-page optimisation is what you can actually do to your website that will affect your ranking on the search engines. This includes changing your title tags, H1 Tags etc. Search Engine Optimisation (SEO) like anything else adheres to the 80/20 rule, whereby on-page optimisation accounts for 20% of search engine rankings.
The other 80% comes from link building, which is by far the hardest part when it comes to SEO. Link building is getting other sites to link back to your own website. Like everything else in the world, links have varying degrees of quality, you get really poor quality links which can actually harm your website or you can get excellent links which will help your rankings tremendously. Obviously, the best quality links are the hardest to obtain. For your information, the best kind of links to get, are links that come from Universities or Government websites. Search Engines love these links and if you do manage to get one your website will almost certainly help get your website on to the first page within a couple of months.