A few months back, I blogged that Google was becoming more choosy on the pages it crawled and that having a few high-quality pages on a site might be a better strategy than a lot of low-quality pages.
Our suggested solution was to add unwanted URLs and directories to your robots.txt file. However, that’s an imperfect solution because search engines don’t always obey the directions in robots, nor do they act quickly.
Fortunately, there’s a new, improved solution….