One of the challenges of search engine optimisation is that one must take in the information and decide what is right. This becomes more difficult when tips and techniques are contradictory. Robots.txt is an ideal example. Many SEOs recommend using robots.txt to control which sites spiders crawl; Google, on the other hand, recommends avoiding them. [...]
